365 Main IDs Faulty Generator Parts

365 Main has identified faulty components in the systems that regulate the speed of its generators as they start, the company said Sunday in an update on its investigation into the power outage. In Tuesday's outage, which knocked many popular sites offline, four of the facility's 10 diesel generators failed to start properly.

"The digital controller for the diesel engine (know as a DDEC) has proven erratic and a spare DDEC is en route," 365 Main said, adding that its engineers have been able to reproduce failures in the controllers in testing. "While this component is the focus of the investigation, the team continues start/stop testing to rule out other potential contributors to failure." DDEC is an acronym for Detroit Diesel Electronic Controller, a system which monitors a diesel engine's status, and can activate alarms or shut down a generator.

It's clear that the outage is also becoming a priority for Hitec Power Protection, which makes the generators and flywheel UPS systems used by 365 Main. Staffers from Hitec's US operation have been on-site in San Francisco since shortly after the outage. On Saturday, Hitec flew in a senior engineer from its headquarters in the Netherlands to join the investigation, along with a member of the company's board of directors. 365 Main noted Friday that "once a root cause is discovered, we will be introducing a tested fix across all facilities that feature Hitec generators." 365 Main has seven data centers across the country.

Equipment failures often raise concerns about whether generators were adequately maintained and tested. 365 Main has actively publicized its generator maintenance program, but is taking additional steps to reassure customers.

"Five full years of preventative maintenance logs on the Hitec generators are currently available for customer review," the company said Friday. " All generators in San Francisco pass weekly start tests and monthly load tests where diesels are started and run at full load for 2 hours. Both of these tests simulate a loss of utility and the auto start function is accurately tested." 365 Main said it is also making available an Operational Risk Assessment of 365 Main's San Francisco data center by the California Data Center Design Group (CDCDG) that was completed in March.

365 Main also said it was reviewing Tuesday's grid conditions with PG&E, the local electric utility. "The event on 7/24 was unique in that it delivered 4-6 repetitive surges to the facility within a short period of time," 365 Main said in its Friday update. "We are working with PG&E and the investigation team to develop a test to mimic the 7/24 electrical event." It should be noted that other San Francisco data centers appeared to have managed the event with few major problems.