Harvey: Hurricane Preparation Tips for Data Center Managers

As Hurricane Harvey bears down on the Texas coast, expected to make landfall around Corpus Christi either tonight or Saturday morning as a dangerous Category 3 storm, the men and women who work in data centers in the area are undoubtedly earning overtime as they prepare for the storm’s onslaught. Keeping data centers operational during natural disasters can be critical to the health and safety of the affected area’s residents, as they supply the lines of communications for many first responders and provide access to valuable information about weather conditions and the state of the area’s infrastructure.

During pending disasters such as this, employees from Schneider Electric’s various data center divisions can often be found on the scene, offering their expertise to help data centers successfully get through the emergency. They’re good to have around, because as the old saying goes, they’ve been there and done that — countless times.

A month or so ago, Data Center Knowledge talked with David Gentry, Schneider’s VP of data center services, and Mark Rentzke, senior manager for its global data center services for Europe, the Middle East and Africa, on the subject of hurricane preparedness for data centers.

Both Gentry and Rentzke stressed that getting through an event such as a major hurricane unscathed depends on planning that begins well before the emergency arises.

“You have to make sure that all staff are trained on emergency preparedness for events that you are anticipating or not anticipating,” Rentzke said. “Make sure the staff understands the contingency plans, the escalation plans, the coordination from security. Make sure that if you need to load-shift any potential equipment, you know which equipment needs to be load-shifted, if you need to drop power at some point, or lower the power.”

“Once an event takes place, it’s obviously too late to acquire needed supplies,” Gentry added. “It’s also too late to assure that all the required equipment is available in top working condition — things that you might need to get through a big storm. You need to do some things upfront to assure that everything is fully charged and to train personnel on things such as proper communications and equipment operations. During an event like this, which is high-anxiety, it’s better if you can already have seen through and [given] those people on the site some points at which they know that they have to do a certain thing.”

Rentzke supplied us with a laundry list of things data center operators need to be prepared to handle:

One of the things we wondered about was preparation for the flooding that can be common during hurricanes. This might be particularly pertinent in the case of Harvey, which is expected to stall once it makes landfall and drop more than 35 inches of rain on the region.

“If a rack is flooded, none of the servers are going to be operating because you’re going to have some very significant electrical problems,” Gentry said. “But you should make sure the most critical servers are grouped together and are on a dedicated source, so that you can isolate them. Isolating those most important servers would definitely be an important thing to do so that you can maintain power, cooling, and whatever is necessary to those servers and that area of the data center for as long as possible.”

“A data center that is established and with a mature program in place will have a risk register,” Rentzke added. “Within the risk register they will identify any major risks. For example, if they have a low area within the campus, they might make note of that, and mitigation for that may be putting pumps in place. Preparedness for this would be to make sure those pumps are running.”

Rentzke pointed out that even if a data center survives a catastrophic storm seemingly unscathed, there still might be hidden equipment damage, caused perhaps by an electrical surge that went undetected during the storm.

“After the hurricane, we need to look at all the equipment,” he said. “In some cases, depending on the critical nature of the hurricane that came through, we need to make sure that the equipment is running at its optimum performance. We might even need to have certain equipment checked out and have a little bit of a deep-dive into the actual health of the equipment. Sometimes after a hurricane you can find stuff lurking within systems that you weren’t aware of that could be danger points later down the road. So there’s an increased risk of something going wrong after the storm as well.”

Ultimately, according to Rentzke, experiencing a major hurricane or any other catastrophic event turns into a learning experience for the data center operators and their emergency-preparedness teams to better prepare the facility for the next major event.

“The whole thing revolves around a continuous improvement, a lessons-learned program,” he explained. “If anything is identified as a corrective action that we can improve on for next time, we put that into our lessons-learned program and make sure there’s a continuous improvement in our methodology for operating a data center in hurricane conditions.”

As the storm approaches, however, the major concern is about well-being of the men and women who will potentially be putting themselves in harm’s way during the storm.

“The safety of our staff, the well-being of all the personnel on site, is our ultimate goal,” Gentry said.