365 Main Offers Apology, Final Report

365 Main has apologized to customers affected by last Tuesday's power outage at its San Francisco facility, and issued a final incident report. "As I reflect on the last week, I'd like to begin with an extension of our sincere apologies to our San Francisco customers who were impacted by the power incident on July 24th 2007," President and CEO Chris Dolan said. "Because we strive each day to deliver our customers the world's finest data centers, we are taking this event very seriously."

365 Main's final report confirmed preliminary findings that the outage was caused by flaws in a component known as a Detroit Diesel Electronic Controller (DDEC), a system which monitors a diesel engine's status, and can activate alarms or shut down a generator. Erroneous data from the DDEC caused 365 Main's diesel engines to malfunction, failing to start properly after a grid outage.

"In the days since the incident occurred, we identified and corrected the root source of the problem and are taking steps to prevent this type of problem from happening again," said Dolan. "We are also making our comprehensive findings available to other data centers to try to prevent the same problem from recurring elsewhere."

365 Main's VP of marketing, Miles Kelly, said the company is focused on communicating with its customers. "This is definitely a humbling experience and a painful experience," said Kelly. "We're following a very predictable emotional cycle after a traumatic event. Right now our customers are looking for assurance that the issues are fully addressed, and that something like this won't affect them again in the future people. This is about stabilizing the credibility we've established with our customers."

While there are reports that some customers may consider relocating their equipment - or at least establishing backup sites - Kelly said the company has benefited from its long-term performance. Kelly said that that over the last five years, 365 Main has maintained uptime of 99.9967% across its entire fiv-facility data center footprint.

After sharp criticism in the initial hours folliowing the outage, 365 Main has posted regular updates of its investigation on its public web site. The amount of information released was unusual compared with most previous industry outages. Kelly believes this has helped address customer concerns.

"We are an open book, frankly," said Kelly. "We hope we've now set the communications standard for dealing with power outages."