Blackberry Network Recovering After Major Outage

A major national network outage at Research in Motion (RIMM) left millions of Blackberry users without access to e-mail for at least eight hours, apparently due to "emergency maintenance" on the Blackberry network. The outage appears to have affected Blackberry service on all carriers in all regions, with downtime beginning in late afternoon Tuesday and extending into early Wednesday. As of 6 a.m. Wednesday morning, reports indicated that service was returning for many users.

The communications problems extended to RIM itself, which has been silent on company accounts on Twitter, which has evolved into one of the most effective channels of outage updates for many service providers. RIM appears to have confirmed the outage when contacted by major tech media, but not issued any direct updates for users. That silence left frustrated users scrambling for information about the service's second significant outage in a week, following extended downtime on Dec. 17.

UPDATE: RIM now says the problems are software-related. "(The) root cause is currently under review, but based on preliminary analysis, it currently appears that the issue stemmed from a flaw in two recently released versions of BlackBerry Messenger (versions 5.0.0.55 and 5.0.0.56) that caused an unanticipated database issue within the BlackBerry infrastructure," the company said. "RIM has taken corrective action to restore service." The full statment is online at Crackberry.com.

In the meantime, RIM is advising users who downloaded or upgraded BlackBerry Messenger since December 14th to upgrade to a new version (5.0.0.57) which resolves the issue.

The outages highlight RIM's ongoing effort to build additional data centers to add more capacity and redundancy to the Blackberry network, which has suffered a series of extended outages in recent years. Research in Motion said it added 4.4 million users to the Blackberry network in the third quarter of 2009, which helped the company post stronger than expected earnings.

But that kind of customer growth also creates the need for network capacity. Research in Motion is reportedly building a large data center in Atlanta to expand capacity for the e-mail network for its Blackberry mobile devices. The company has acquired 40 acres of land in Alpharetta, and is said to be building a 200,000 square foot data center at the site.

The new facility continues an expansion of RIM’s data center infrastructure beyond its home base of Waterloo, Ontario. The Blackberry e-mail network suffered a major outage in February 2008 that left its 12 million North American users offline for hours, prompting criticism that the company was overly reliant on data centers in Waterloo that handled all its North American messaging traffic.

In June of 2008, Research in Motion bought a 150,000 square foot data center in Plano, Texas to expand provide network diversity. Texas is a popular site for backup data centers because the state has its own power grid.