Insulation Failure Cited in Fisher Plaza Outage

fisher-busbar

A picture of the damage to two bus ducts at Fisher Plaza in Seattle from a July electrical incident that caused a major outage for its data center tenants.

A major power outage that disrupted data centers at Seattle's Fisher Plaza in July was probably caused by an insulation failure in an electrical bus duct that connected the building to the power grid, according to a report to tenants by the building's owner.

The July 2 incident knocked payment processor Authorize.net offline, disrupting e-commerce for thousands of web sites, while also causing lengthy downtime for Microosft's Bing Travel service, domain registrar Dotster, colocation company Internap and web hosting provider AdHost,

An investigation by Power Science Engineering Inc. of Shoreline, Wash. concluded that the failure of two bus ducts - enclosures housing copper bars to conduct electricity - in a basement power room of Fisher Plaza was likely caused by "progressive degradation" of insulation, which ultimately led the bus duct to fail. The incident report is available at the TechFlash web site.

No Single 'Stimulus' 
Investigators said such failures are usually prompted by a "stimulus" such as movement of the duct or sudden changes in large loads, but that none of those conditions existed at the time of the failure at 11:11 p.m. on July 2. Power Science concluded that the insulation wore down over time due to thermal and mechanical stresses at a bend in the duct.

The report rejected two other theories - that the bus duct was damaged by a falling light fixture or an accidental triggering of a sprinkler system. A flourescent light fixture was found laying on top of the bus duct, but investigators said the fixture collapse was caused by the "significant heat" from the electrical failure.

Water Damage Discounted
The report also found no evidence that the sprinkler system turned on prior to the electrical event, and noted that the bus duct was enclosed in a water-tight metal container. The initial report from the Seattle Fire Department said that "engineers suspect that water on the copper plates (of the bus) resulted in an arc and the subsequent fire damage."

The Power Science report noted that "routine maintenance and testing" such as load data and infrared scanning can help detect problems with aging power infrastructure, although this is more difficult for enclosed components like bus bars inside ducts. Fisher Plaza officials told TechFlash they conducted regular infrared and thermal scans and physical inspection of the equipment.

Power Science Engineering also said bus ducts can be separated to limit damage from a failure, or replaced by cables. "In our opinion, use of cable systems at critical locations can be a better option to bus ways because the cable systems use insulated cables instead of bus bars," the report said.

Building owner Fisher Communications has been powering Fisher Plaza using temporary generators while it rebuilds the core power infrastructure. The new design will use bus ducts, but will separate them to avoid simultaneous failures.