Outage for Rackspace Customers

Many sites hosted at Rackspace Hosting were offline this afternoon when the company's data center in Grapevine, Texas lost power for about 45 minutes. Rackspace reported at 4:30 p.m. Eastern that it was "having an issue that is affecting part of our DFW data center." Power was restored shortly after 5 p.m., but the  company continued to manually bring up servers that did not restart properly when power was restored, a task that lasted well into the evening. Some sites that had support for SSL encryption were also experiencing delays in being returned to service, according to a status update at The Rackspace Cloud.

"These power interruptions were the result of a range of power infrastructure issues," Rackspace said in an update on its blog. "Right now, the Dallas data center is stable and running on utility power.  Our UPS units have been re-serviced and re-checked as of this evening, and we are in the process of doing the same with our generators."

"Although this outage only affected a portion of our customers in one of our nine global data centers, we consider any outage to be unacceptable," the company said in its update. "We sincerely apologize to our customers and those who were affected by this downtime. ... Now that we have the near-term situation stabilized in Dallas, we have some work to do to improve our reliability.  We will follow up with more information as we work through our root-cause analysis."

The DFW data center is the company's largest facility, with 144,000 square feet of space. The facilty in Grapevine figured into a 2007 power outage that interrupted service for many prominent web sites. In that incident, a vehicle struck a power transformer, and public safety officials turned off both the facility's power feeds during their emergency rescue operations.

The downtime was noted at TechCrunch and on Twitter. I'll admit to being surprised that Justin Timberlake was so knowledgable about the hosting industry.