How Sandy Has Altered Data Center Disaster Planning

Hurricane_Sandy_New_York_Bl

The Empire State Building stands out as a beacon of light in a darkened Manhattan landscape during the widespread power outages following Superstorm Sandy. (Photo by David Shankbone via Wikimedia Commons).

NEW YORK - Keep your diesel supplier close, and your employees closer. These were among the "lessons learned" from Superstorm Sandy, according to data center and emergency readiness experts at yesterday's Datacenter Dynamics Converged conference at the Marriott Marquis, which examined the epic storm's impact on the industry and the city.

The scope of Sandy has altered disaster planning for many data centers, which now must consider how to manage regional events in which travel may be limited across large areas due to fallen trees and gasoline shortages, restricting the movement of staff and supplies. Yesterday's panel also raised tough questions about New York's ability to improve its power infrastructure, as well as the role of city policies governing the placement of diesel fuel storage tanks and electrical switchgear.

A clear theme emerged: Data center operators must expand the scope of their disaster plans to adapt to larger and more intense storms, weighing contingencies that previously seemed unlikely. The power, size and unusual storm track for Sandy proved to be a deadly combination, bringing death and destruction  on an unparalleled scale.

Superstorm Sandy caused $19 billion of damage in New York City, leaving more than 900,000 employees out of work at least temporarily, according to Tokumbo Shobowale, the Chief Business Operations Officer for New York. Shobowale said the storm has led FEMA to redraw the storm surge maps and flood zones for the city.

"We have 200 million square feet of commercial space in the flood plain now,' said Shobowale, who said the city struggled to adapt to unprecedented flooding that damaged critical infrastructure for transit, telecommunications and power. "A lot of our response was figured out on the fly. Now that experience allows you to create standard operating procedures for next time."

Focus on Fuel and Personnel

The data center industry has begin that process in earnest. New York area facilities experienced both direct and indirect impacts from Sandy. A handful of data centers in the financial district were knocked offline as the storm surge flooded basements housing critical equipment. Nearly all of lower Manhattan was left without power when ConEd was forced to shut down key parts of the power grid, forcing major carrier hotels and data centers to operate on backup generators for three to seven days. Facilities in New Jersey also faced local power outages and road closures, as roads fell across streets and power lines.

Planning ahead is more important than ever, as data centers will need to consider padding their inventories to ride out longer periods in which they must operate independently.

"If you didn't have your service providers and employees on-site at the time of the storm, they weren't going to get there," said Paul Hines, VP of Operations and Engineering at Sentinel Data Centers, which has a data center in central New Jersey. "That's affected our planning." That includes keeping more spare parts at the facility, bringing more staff on-site and additional advance planning with maintenance contracts and fuel suppliers.

Several questions for the panel focused on the availability of diesel fuel for emergency backup generators, which was a key concern in the storm's aftermath. Data center providers typically arrange priority contracts with fuel suppliers. But what happens when a regional disaster tests supply and creates dueling priorities?

Providers in New Jersey reported no problems finding fuel, although some had to go outside the region to ensure a continuous supply. "We had 10 days of fuel, and contracts with two fuel suppliers" said Hines. "You also have to make sure your fuel suppliers can operate with no power, and have gravity-fed systems. We've now found an out-of-region supplier as well. But that doesn't solve the problem of access to facilities."

That was also a pressing problem in Manhattan, where flooding made some roads impassable. Building owners worked with city officials to ensure the availability of telecom services, for example. One of the city's largest data hubs, 111 8th Avenue, had a high priority because the building also houses a hospital.

The Role of the City

Audience members at DatacenterDynamics Converged also pressed Shobowale about the city's response to Sandy, especially the vulnerability of the utility grid. One questioner noted the failure of a major ConEd substation built alongside the East River.

"We've had a lot of conversations with ConEd," said Shobowale. "They are very aware of the issue. They've developed a capital plan to address this."

Shobowale, who was appointed last year by Michael Bloomberg, emphasized that while few concrete solutions have been put in place, the city is working hard on a process to address the broader impacts of climate change, including flooding from more powerful storms. that includes the location of mission-critical equipment, and the type of equipment being used.

"Let's get this equipment out of the basements," said Robert Bianco, VP and General Manager of TW Telecom. "We're putting this equipment on higher levels of buildings."

Sabey Data Centers, which is about to open its Intergate Manhattan facility, is housing its diesel fuel tanks in the basement, but is anchoring them and equipping them with submersible pumps, said John Sasser, VP of Operations. The company's building was not affected by flooding during Sandy, and is well above even the revised flood plains, he said. But the company is taking no chances.

Fuel pump failures in basement-level diesel tanks were key failure points for several buildings that were flooded during Sandy, including 33 Whitehall and 75 Broad, where staff of several providers formed a "bucket brigade" to relay 5-gallon buckets of diesel fuel up 17 flights of stairs to refuel a generator providing emergency power to a Peer 1 data center.

So why are these tanks in the basement? New York imposed restrictions on rooftop diesel storage after the Sept. 11 terrorist attacks, when an early engineering analysis from FEMA suggested that leaking fuel from diesel tanks contributed to the collapse of 7 World Trade Center. “We had recently applied to move the fuel tanks to the roof and were rejected,” Alex Reppen, the CEO and founder of Datagram, told DCK in November. “We’re hoping the city may change its view on fuel. We’ve talked to a lot of our competitors, and believe that this issue will be revisited.”

Shobowale said yesterday that New York's policies on the location of fuel tanks are being discussed, but said the city must also balance safety considerations associated with storing diesel on higher floors in buildings.

Looking Ahead: An Altered Readiness Landscape

New York's disaster readiness challenge is being examined on many levels, from the details of codes to fundamental questions about shoreline redevelopment and the best ways to protect the city's infrastructure, businesses and residents from the Sandys to come.

"We are a city of islands," said Shobowale. "We have more coastland than Los Angeles, San Francisco and Boston combined. We have a dense, urban environment which is much more difficult to (adapt) than just putting a building on stilts in Louisiana."

If you're weighing "lessons learned" and want to implement some of them in your data center or disaster recovery operation, you'd better get your budget requests submitted soon, panelists said, as the storm's grip on C-suite attention may have a shorter shelf life than you think.

"A year from now, Sandy will be a more distant event," said Hines. "Memories fade. You need to strike while the iron is hot. This is the time to do it. A year from now, it will be much harder."