'Critical' Outage for Blackberry Network: February 2008

The Blackberry e-mail network operated by Research In Motion (RIMM) was down four about four hours today, with service resuming for most users at approximately 7:30 pm Eastern time. Early reports indicated that RIM indicated its network had experienced a "critical severity outage," but the company later described the problem to media as "intermittent service delays" for BlackBerry subscribers in North America.

By 7 pm network admins in some cities were reporting signs of life on the Blackberry network, but it took a while longer for service to resume in earnest, perhaps due to a huge backlog of messages.

"This is an emergency notification regarding the current BlackBerry Infrastructure outage," RIM said in an e-mail about 4 pm Monday. The message said the outage affected enterprise clients and "users of the Americas network." CNBC is reporting that the outage affected e-mail on all of the estimated 12 million Blackberry users. Phone service remained functional throughout.

Research in Motion previously experienced a widespread network outage in April 2007, which left users without e-mail for more than 12 hours. RIM later attributed the outage to new code that was deployed without being thoroughly tested. The new software "triggered a compounding series of interaction errors between the system's operational database and the cache." When the network operations center sought to move to a backup configuration, the process "did not fully perform to RIM's expectations."

Early reports at Blackberry Forums indicate that problems with Blackberry connections started at about 3:30 pm, recovered briefly around 3:45 pm but then crashed again. The BB-outage mailing list has also been tracking the outage. See additional coverage at Engadget.

Research in Motion is known to have conducted maintenance over the past weekend (Feb. 9-10), including upgrades to hardware components, databases and administrative networking systems.

Research in Motion operates two data centers, one in Waterloo, Ontario and one in the United Kingdom. In the wake of the April 2007 outage, RIM was criticized for its reliance upon the Waterloo data center to handle its North American messaging traffic.