Data Center Outages Bring Headaches, Headlines

July saw a steady stream of data center outages due to equipment failures, several of which attracted media attention. The latest incident to make headlines was an outage July 30 at Seattle's Fisher Plaza, which is described by CRM Buyer today in a story titled Unsinkable Data Center Crashes in Seattle. The article digs into the cause of the downtime, which is in dispute, as Fisher Plaza cited an equipment problem after a Seattle City Light outage, but the power company says it was never offline.

The recent string of incidents provide a painful reminder that Murphy's Law has jurisdiction over even the most wired data centers. An AFCOM member survey from April predicted that within the next five years power failures and shortages will halt data center operations (at least briefly) at more than 90% of all companies.

The uptime industry is in the business of trying to anticipate everything that can go wrong, and engineering solutions for even the most improbable scenarios. Although SLAs promising 100% uptime are common nowadays, stuff happens. "Failure is inevitable. Fail small," said Richard Sawyer, Director of Data Center Technology for American Power Conversion, in discussing the AFCOM results. Outages are painful, but offer lessons as well. In that spirit, here's a recap of some of the recent incidents:

Which brings us to a potential learning opportunity in crisis communications. Providers housed at the Garland Building and Fisher Plaza have complained publicly about the lack of information during the outages, which made it more difficult for them to relay useful information to their customers. Nobody likes to talk about downtime, and this was a theme in the CRM Buyer article.

"Attempts to maintain secrecy and shift responsibility for the causes and consequences of the Fisher Plaza outage have made the events of July 30 as much about corporate culture and professional integrity in the face of adversity than about the technical issues surrounding the outage," writes CRM Buyer's Anthony Mitchell. (It's worth noting that Mitchell is not an entirely detached reporter/commentator, as his company uses one of the providers at Fisher Plaza affected by the outage, a wrinkle that isn't explicitly stated in the story but can be ascertained via his bio blurb).

At the other end of the spectrum is DreamHost, which published a lengthy blog item providing customers with a blow-by-blow of its downtime at the Garland Building. The level of disclosure was praised by marketing expert Seth Godin on his blog item titled Important Lessons From DreamHost.

"Lesson one: when things get messed up, being clear, self-critical and apologetic is really the only way to deal with customers if you expect them to give you another chance," Seth writes. "Lesson two: your story is all you've got. If you sell the 'up-time' story, better over-invest in whatever it takes to be sure your story is true."