Hurricane Sandy Post-Mortem: Will Your Critical IT Systems Survive Another Major Power Outage?

Vince Pelly works for Citihub, a global IT consultancy as an Associate Partner. He has more than 25 years of IT executive experience, having held senior posts in program management, infrastructure and data center design, where he performs data center and network assessments, designs, migrations and consolidations.

vince-pelly

The week of October 29th was a testament to the resiliency of data center infrastructure, critical systems and disaster recovery plans. Most notable were the numerous stories heard from the press and customers who ran out of diesel fuel, failed fuel pumps, flooded basements and failure to execute recovery plans in a timely manner. In lower Manhattan, New York’s Con Edison pre-emptively shut down power, forcing data centers on to generator backup.

One interesting note regarding backup generators and the use of diesel fuel to sustain critical systems operations is how The New York Times recently criticized these organizations for environmental issues due to diesel fuel usage during power outages.

Sustaining Critical Operations

This past event confirms that the use of diesel fuel to run backup generators, the location of the critical systems and primary data center as well as the geographic location of backup sites are significant to sustaining the ongoing critical operations for organizations.

The gaps and risks uncovered over the past two weeks demonstrate numerous vulnerabilities and design failures for IT mechanical and electrical systems, business continuity plans and operational procedures. In addition, the locations of these critical systems are within the primary buildings' basements or lower levels exposing design flaws that prevented these firms from continuing their operations. This doesn’t apply to all firms; some have implemented rigorous backup and fail-over plans, looked ahead and planned around risk accordingly. Others ignored executing backup plans which have resulted in long outages and exposed gaps within their disaster recovery designs.

Best Practices for Being Able to Survive and Execute a Recovery

The following highlights key areas necessary to minimize future outages:

Going forward, organizations should periodically review their fail-over architectures and incorporate redundancy models via active-active and/or multi-site fail-over designs using multiple data center sites that are located in and out of primary regional power grids.

Multiple site disaster recovery designs provide a higher level of zero data loss at long distances. As an example, implementing a remote asynchronous (out of region) replication model can provide recovery with low data loss when both the primary and local recovery site is impacted (as in the case of a regional outage such as Hurricane Sandy).

back-up-plan

Example of a multi-site design incorporating an out-of-region disaster recovery backup site. Click to enlarge.

Industry Perspectives is a content channel at Data Center Knowledge highlighting thought leadership in the data center arena. See our guidelines and submission process for information on participating. View previously published Industry Perspectives in our Knowledge Library.

Note: Reference architectures provided by the Uptime Institute and the Telecommunications Industry Association describe the standards and best practices concerning site selection and the design and location of Critical Equipment Rooms (CER). You should refer to the TIA specifications and best practices when planning and assessing their IT infrastructure and the Uptime Institute for data center certification and standards.