Michigan County Offline After Data Center Fire

IT services in Macomb County, Michigan are offline after a fire damaged the building that houses the county's data center. Macomb County, which is just west of Detroit and has 850,000 resident, did not have a backup data center.

The fire Wednesday was in the basement of the Old County Building. The data center is on an upper floor, but the county is unable to assess damage to the equipment because the building has no electricity. County Executive Mark Hackel declared a state of emergency Thursday, saying the building could be closed for months and require millions of dollars’ worth of repairs and upgrades

The fire left county staff without Internet or phone service. "We ask the public to be patient with us while we assess the damage to our IT, internet and phone systems to determine the extent of damage," the county said on its web site.

Email is available due to a recent shift to Gmail, but with many other county computer systems unavailable, Macomb officials are resorting to pen and paper, carbon copies, and makeshift networks of laptops to try and maintain services. “The computers are down. What to do?" County Clerk Carmella Sabaugh told the Macomb Daily. "We have to go old-school and do everything on paper.”

Old Buildings, Inadequate DR Planning

The outage in Macomb County is the latest in a series of incidents that have underscored the vulnerability of local governments, who often have data centers in older buildings and maintain inadequate backup and disaster recovery plans.

Last year a data center fire in a Shaw Communications facility in Calgary, Alberta crippled city services and delayed hundreds of surgeries at local hospitals. The incident knocked out both the primary and backup systems that supported key public services, providing a wake-up call for government agencies to ensure that the data centers that manage emergency services have recovery and failover systems that can survive a series of adversities.

Macomb County was in the process of building a new data center, but never established a backup site for the existing facility at the Old County Building, a 13-story structure that was built in the 1930s and lacked a modern fire suppression system.

Hackel, the county administrator, said he had warned county officials about the need for a backup facility, but the county was unable to implement a plan before the fire.

The Macomb County Circuit Court is operating, but its case management system also is down, Court Administrator Jennifer Phillips said. “We’re up and running, but we’re asking people to be very patient," Phillips told the Detroit Free Press. "We’re reverting back to processes not as efficient and not used in a long time.”