Temperatures Soar at Data Center Inside 111 8th Avenue

111-8th-Avenue

The exterior of 111 8th Avenue, one of the premier carrier hotels in Manhattan.

Temperatures inside a Zayo Group data center at 111 8th Avenue in New York City briefly soared past 100 degrees this afternoon as generator problems forced the company to power down its cooling systems. The company has since stabilized the temperature and brought the cooling system back online. Customers remained online throughout the incident, which was the latest in a series of challenges that have confronted New York's data center community since SuperStorm Sandy smashed the city earlier this week.

Like most of the buildings in lower Manhattan, 111 8th Avenue has been running on generator power since Monday evening, when utility outages spread across much of Con Edison's power grid. The problems began this morning when issues with a fuel pumping system left Zayo's generators unable to support both its networking equipment and its cooling systems in its data center on the 7th floor of 111 8th, a major data center hub in Manhattan. The company kept the networking gear online and powered down the cooling infrastructure.

"Temperatures jumped quickly after the initial problem," Zayo reported on its network updates. "We took action by opening windows on the upper levels and setting up multiple fans to circulate air. The general temperature has been stabilized at 95-100 degrees. This has bought us time to implement emergency plans for back-up power sources and to isolate the root cause of the fuel supply issue."

Customer equipment continued to operate within the data center, and Zayo said it expected to be able to solve the problems without service interruptions to customers.

UPDATE: "We have since resolved the issue with the generator and have brought cooling back to the 7th floor," reported Matthew Brouker, Zayo Group's Director of Marketing, in an email this evening. "As an additional precaution we will have a rolling 2 MW generator on site to supply power.  Also, we have a battery plant capable of supporting the facility for 5 hours.

Other data center tenants at 111 8th Avenue were unaffected by the cooling problems and continue to operate on generator power, including two other Zayo facilities on the third floor of the building. 111 Eighth Avenue is among the world’s most wired buildings, housing major data center operations for Digital Realty Trust, Equinix, Telx and many other providers and networks, as well as 500,000 square feet of office space for Google, which now owns the building.

Portable Generator En Route

The generator that experienced the fuel line problems was shared by Zayo and Equinix, which confirmed an outage earlier today but has not reported any problems with its cooling.

"Over the past two days our NY9 site, which is housed in a multi-tenant building with shared generator infrastructure and fuel service, has experienced three outages due to generator failures that impacted service to several customers," said Equinix. "Repairs have been made and service was restored. The site remains on generator power."

Zato said it has "elected not to attempt to bring cooling back on-line so as to avoid any potential collateral impact to generator operation," mentioning that it had coordinated its actions with Equinix.

Telx Takes Precautions on Heat, Power

Telx, which also has a data center at 111 8th Avenue, says it asked customers to power down non-essential equipment to conserve electricity and diesel fuel and "reduce heat generation."

"Our facilities are still maintaining a temperature that is within an acceptable range," Telx reported in a status update. "At this time we are not seeing temps increase at NYC2 (111 8th). Thermographic imaging to monitor electrical distribution systems was successful, at this point all emergency systems are normal."

111 Eighth Avenue is among the world’s most wired buildings. It was originally built as the Port Authority Commerce Building in 1932, and was redeveloped for telecom use by Taconic in the late 90s.The enormous building houses major data center operations for Digital Realty Trust, Equinix, Telx and many other providers and networks, as well as 500,000 square feet of office space for Google