Power Woes Continue at LA's Garland Building

Tenants at the Garland Building (1200 W. 7th) in Los Angeles have been enduring a difficult week. On July 24 the building lost all power for the second time in less than a year, as an automatic transfer switch and UPS backup system failed. The resulting outage made headlines, as it knocked MySpace off the Web for an extended period. Then on July 28, the Garland building suffered *another* loss of power, which was attributed to a "dead short" that occurred while the building was on generator power.

The outages have been especially difficult for Dreamhost, a growing hosting provider which colocates servers in two facilities within Garland (Switch & Data and Alchemy). Dreamhost had been struggling with network problems when the July 24 power outage hit, and has had rolling performance problems ever since. The Dreamhost team maintains a weblog, and today it provided its frustrated customers with a lengthy chapter-and-verse timetable of the problems, titled Anatomy on a(n Ongoing) Disaster.

As Dreamhost's Josh Jones explains, the repeated outages at 1200 W. 7th are related to power capacity issues and the cost of redundancy.

Here's an excerpt from Jones' lengthy post:

In September 2005, much of LA lost grid power when a utility worker accidentally cut a key cable. The Garland building has five generators, but three malfunctioned, and the remaining two were unable to handle the load. In the July 24 outage this year, the generators worked, only to have the ATS and UPS systems fail.

Dreamhost is making the best of matters with a full explanation of the problems, and trying to use humor to dull the pain (interspersing photos of famous disasters throughout the timeline of incidents). While some customers are posting supportive comments, many others are critical of the impact on their business. Several noted that several other builduing tenants, such as MediaTemple and MySpace, had less persistent problems than Dreamhost.