Explosion at The Planet Causes Major Outage

An electrical explosion and fire Saturday at a Houston data center operated by The Planet has taken the entire facility offline. The explosion at 5 pm Saturday has affected 9,000 customer servers, and the company says it hopes to restore service by late Sunday afternoon. No servers or network equipment were damaged by the explosion, but the data center is without power. The Planet said it is working with the fire department and its facilities staff to restore power and get servers back online.

UPDATE: As of early Wednesday, power has been restored to the first floor (Phase 1) of the Houston data center. A generator failure early Tuesday slowed the recovery process at The Planet, leaving 1,500 servers offline and prompting the company to offer to physically migrate customer servers to a second Houston data center it operates. See our previous coverage of the damage to the facility and industry feedback on potential lessons learned from the incident.

"(Saturday) evening at 4:55pm CDT in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room," said Doug Erwin, CEO of The Planet, in a message on the company's forums (mirror). "Thankfully, no one was injured. In addition, no customer servers were damaged or lost."

Early indications are that the fire was caused by a short in a high-volume wire conduit. The fire department is not allowing the company to run backup generators, so the facility has been without power since the incident occurred. In the latest update, The Planet says the damage to the data center was more extensive than initially believed.

The explosion affected only the main Houston data center, with no impact on any of The Planet's other five data centers. The company hosts more than 50,000 servers and 22,000 customers in its six data centers, meaning that about a third of its customers and 20 percent of customer servers are currently offline.

This was the second time an explosion and fire has occurred at the Houston data center, which had a transformer explode in June, 2003 when the company was known as Rackshack.

Many of the customers affected are accounts from EV1Servers, the Houston-based dedicated hosting specialist that was acquired by The Planet in 2006 as part of a larger transaction in which private equity firm GI Partners bought both companies. The Houston data center houses the servers for ServerCommand, the management portal for former EV1 customers. "We are in the process of moving the ServerCommand servers to other Houston data centers so that we're able to loop them into communications," the company said.

The Planet has sought to move accounts to get customers back online, with limited success. "During the early stages of the H1 data center we opportunistically relocated some customers to another data center," wrote Urvish Vashi, the Director of Product Management for The Planet. "However, due to network and data center (power/cooling) constraints, this option is no longer available and requests for migration cannot be honored."

The Planet's main page was knocked offline briefly, according to monitoring from Netcraft, but was back online in less than an hour. The Planet's forums are also online, but are experiencing serious availability problems due to traffic, including a Slashdotting.

"This is a significant outage, impacting approximately 9,000 servers and 7,500 customers," said Erwin. "All members of our support team are in, and all vendors who supply us with data center equipment are on site. Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock." Center Networks has noted the strong customer communication during the outage.

The Planet said it intends to proactively credit accounts for downtime under terms of its service level agreements (SLAs). The Plant also operates data centers in Dallas, where it was headquartered until last year, when its main offices were shifted to a new facility in Houston.