Water Main Break Floods Dallas Data Center

IT systems in Dallas County were offline for more than three days last week after a water main break flooded the basement of the Dallas County Records Building, which houses the UPS systems and other electrical equipment supporting the data center on the fifth floor of the building. The county does not have a backup data center, despite warnings that it faced the risk of service disruption without one.

The 90-year-old water main ruptured at about 10:30 p.m. last Monday, flooding the basement-level mechanical room with six feet of water, forcing the shutdown of the data center. Power was not restored to the facility until early Friday, as it took several days for the county to pump out the basement. The water was mixed with oil and mud, and needed to be pumped into taker trucks rather than the sewer system, delaying the recovery.

"The county's criminal justice system nearly ground to a halt, as paper processing from another era led to lengthy delays," the Dallas News reported. Several courts have gone paperless in recent months, and the district clerk's office has been converting files to electronic form and destroying paper versions, the paper said.

The county had been warned in a 2008 consultant's report that it needed to address its lack of a backup data center, but never followed through on the recommendation. The county IT staff is now developing a failover plan and expects to submit an updated plan in two weeks.

A portable generator and UPS were used to restore power to the facility early Friday, and the county IT staff began checking to ensure servers were properly restarted. The Texas county said it believes no critical data was lost.