Repairs Force Seattle Data Center Offline for 5 Days

fisher-busbar

The city of Seattle is shutting down some data center services to repair a faulty electrical bus, seeking to avoid the type of damage (seen above) that occurred during a 2009 bus failure at Fisher Plaza in Seattle.

The city of Seattle will turn off many public services for five days over Labor Day weekend so it can repair the power system supporting its primary data center, Seattle Mayor Michael McGinn said Thursday. The shutdown will allow the city to replace a faulty bus bar providing power to the city data center on the 26th floor of the Seattle Metro Tower.

During the shutdown, which begins Friday and runs through Tuesday, city residents will be unable to conduct many services they would normally access online, including paying bills and applying for licenses. Critical operations, including 911 and other public safety systems, will remain online.

"The city will certainly be open for business, but it won't be quite as convenient as it is normally," said Seattle Administrative Services Director Fred Podesta. "But this is a short-term event, and it's a holiday weekend."

Workarounds for Faulty Bus

The repairs were prompted by overheating of the primary bus bar, which McGinn described as "a big beam that brings electricity from the street up to the data center."

"The temperature of the bus had gone up because it wasn't conducting electricity properly at a joint where two pieces come together," said McGinn in a news conference. "The temperature was going up, and that's not good. It could have led to a failure, and some sort of arcing or short. Upon learning this, we immediately began shutting down servers and moving systems and reducing the load on the bus, and that brought the temperature back down to a safe and stable temperature. But we knew we would need to replace the bus and take care of that bad joint."

For a lesson in the hazards of a bus failure, McGinn won't have to look far. In July 2009 an electrical fire at Seattle's Fisher Plaza disrupted Internet commerce and caused $6.8 million in damages to the building, a major communications hub housing data centers. The fire was caused by a failure of two bus ducts – enclosures housing copper bars to conduct electricity – in a basement power room of Fisher Plaza. The incident caused extended downtime for the payment portal Authorize.net, which left thousands of merchants unable to process credit card payments through their web sites.

Crippled in Calgary

Last month the city of Calgary was hit by a data center fire that crippled city services and delayed hundreds of surgeries at local hospitals. The incident serves as a wake-up call for government agencies to ensure that the data centers that manage emergency services have recovery and failover systems that can survive as series of adversities.

In March the city launched an analysis of its data center operations, which will assess whether Seattle should build more data center space, use a third-party provider or move services to a public cloud.

"When we put the data center on the 26th floor (in 2000), we didn't anticipate the explosion of data and the number of servers we would be using," said McGinn. "We need greater redundancy in that system. We need to upgrade our center. This is not the most reliable way for us to support the city's operations."

Erin Devoto, the Deputy Director of Information Technology for Seattle, says a study of the city's data center options is underway and will take about a year.