Resilience in a Pandemic: Smaller Data Center Providers Hang On, Hold Fast

Day One for many of America’s regional and metropolitan data center providers may have been Monday, February 3, 2020. Just over two weeks earlier, the State of New York issued a warning to its clinical laboratories and healthcare facilities to prepare for a potential outbreak of a mutated animal virus, linked to a seafood and animal market in Wuhan, China. It had been reported to the World Health Organization on December 31, although China officials dismissed it that day as “viral pneumonia” — confined, they said, to some 27 people.

When Thailand reported its second case of the same virus on January 17, the State House in Albany, New York, took a gamble that it wasn’t pneumonia, and that it wasn’t confined. It issued its warning that same day.

About four blocks southeast of the State House complex is the headquarters of FirstLight, a colocation and data center services provider for the far northeast US.

“Probably 45 days before there was even a go-home order, we were having these discussions on how we’re going to deal with it,” said Robert Rivas, vice president of data center and cloud operations for FirstLight. “We said, let’s start planning this appropriately ahead of time. Let’s start telling our staff to start practicing social distancing on their own, and let’s not bring in five or ten vendors in a single day, just because we can, overlapping with everybody.” 

Related:Essential Workers: Keeping Data Centers Running During the Pandemic

Immedion’s data center complex and headquarters in Greenville, South Carolina

Some 710 miles southwest in Greenville, South Carolina, facilities managers from colocation provider Immedion had already begun digesting the signals coming out of New York. As its president and COO, Rob Moser, told us, they began comparing their emergency response plans against the growing likelihood of a global pandemic.

“I would say February was when we expanded, and brought in the larger management and executive team for communication and clarification,” Moser said. “Certainly by early March we were starting to implement work-from-home.” Since then, Immedion’s facilities teams throughout North and South Carolina, Ohio, and Columbus, Indiana, have been in twice-weekly communication about the health of their staff, the state of their services, and the resilience of their business.

Continuity

Immedion had already developed a pandemic response plan back in 2009, in response to the outbreak of H1N1 — a virus with a mortality rate of less than a tenth of one percent of infected patients. (COVID-19 currently has an estimated mortality rate of 1.4 percent.) The impetus for that plan, Moser told us, was an abundance of customers’ concerns. With facility locations amid hurricane paths, Immedion’s BC/DR (Business Continuity/Disaster Recovery) plans up to that point were focused on the typical “acts of God.” One of these typically lasts no more than a few days before it’s already time to clean up and regroup.

Related:How the Data Center Technology Market Is Faring in the Pandemic

“Typically in an emergency response plan we would stage team members either at or close to the potentially impacted area,” Moser said. “You could see the storm coming, and you could stage supplies and team members and be ready to respond. In this situation it’s really the opposite. You’re trying to protect the team members that have to be within the data center and send all non-essential staff home to protect their health. We have to minimize the team that’s within the data centers, as opposed to a more storm-related [plan], where we’re preparing staff to inundate the data centers.”

Most every BC/DR plan up until February was focused on maintaining contracted service levels. In a sustained event, however, one whose likelihood of having ended by this time next year isn’t clear, service levels cannot be sustained unless the staff’s wellbeing is considered first.

Cheyenne, Wyoming-based Green House Data reached this conclusion in 2019, during a company-wide role-playing exercise. Similar to what US national security teams attempted to stage for the incoming presidential administration in 2017 without success, as Green House CIO Cortney Thompson told us, his team simulated a scenario where a catastrophic event prevented them from accessing some of their facilities for as long as one year.

“We went through those exercises [with] our main services portfolio, our consulting portfolio, and our data centers as well,” related Thompson. “I think we were well armed in that aspect of it.”

“When this event happened, there were so many variables that we had to adjust on the fly and improvise,” FirstLight’s Rivas admitted. “Maybe meeting every day, twice a day, is not necessary. But we improvised on the side of caution.”

Green House Data’s primary facility and headquarters in Cheyenne, Wyoming

Plenty of guidance has thus far been offered to data centers of all sizes, though much of it has overlapped, quite a bit was conflicting, and one critically authoritative source has been missing in action.

“As you know, what the federal government says versus what the state government’s saying, versus county, and even down to the town level, [is required] of businesses as a whole can be very, very different,” FirstLight’s Williams said.

The most likely reason may have been lack of clarity at the federal level early in the crisis. The federal government did declare a national emergency on March 13, and New York Governor Andrew Cuomo’s official stay-at-home order came the following week. By then, however, two months had passed since New York’s initial warning raised concerns among businesses of all kinds.

As Alan Howard, Omdia’s principal analyst for cloud and data centers, told us, the absence of a kind of “mission control” on the federal level may have compelled tier-one through tier-three providers to look after one another.

“I would certainly expect any tier-two, tier-three colocation providers worth their salt — organizations that understand the business and what drives decision-making from enterprises – to have business continuity plans that take many of these things into consideration,” Howard said. “Whether or not they were all on the cusp of the bleeding edge, dealing with this, the first time they heard one of their peers talking about the things that they were doing, the rest of them would have fallen in line right away.”

Staffing Levels

The number of personnel allowed inside a facility was reduced, staffing arrangements were changed. Facilities of tier-two providers typically don’t have the luxuries one of the largest data center providers may have, such as cots and showers. Not all the providers with whom we spoke were in a position to tell us just what the minimum number of viable staff per building or within a given area of space actually was. These decisions appeare to have been made on a case-by-case basis. It may be easier for a hyperscale data center to provide seamless service migration to redundant servers with one or two commands; at a regional or a metro provider’s facility, it may take more of a plan.

Immedion’s Moser told us his facilities continue to be supported by network operations personnel round-the-clock, with a select facility manager onsite during regular business hours. Still, this limits its operations areas to a maximum of two people per shift. Exceptions to these limits require express permission from headquarters. General IT, network management, sales, and marketing teams are all working from home.

“Every company has a need to have their employees close, and historically we’ve tended to want people in the office,” remarked Green House Data’s Thompson. “We want them collaborating, we want them highly effective. And we’ve learned through those exercises that via additional efforts online we can really keep these guys close, productive, and collaborating even remotely. So, I think there may be less and less of a need over time for us to have people in that central office space.”

“I think the biggest impact we’re going to see is, we will have to and want to continue to offer greater flexibility in work-from-home,” Immedion’s Moser said. “That’s a philosophy that we had not embraced greatly coming into this, but now that we’ve seen that it is successful, that we’ve been able to maintain our standards and our communication levels and our customer support while having some of our key staff work from home — and now that some of them have also had a taste of what it’s like working from home — I think that genie is out of the bottle. We’re going to have to provide and offer greater flexibility in how we work from home going forward, coming out of this.”

The Albany, New York, building housing FirstLight’s data center and headquarters

For some providers this may be the permanent legacy of the pandemic: fewer staff on the data center floor. In an ironic way, this could also be the solution to the data center skills shortage.

But FirstLight’s Rivas believes there’s a limit to the extent automated processes can substitute for everything onsite staff have done pre-pandemic.

“I do not believe that a change in staff, based on the systems that we have in place, is warranted,” he told us. “There’s no data to show that now that we’re able to have somebody working remotely or getting monitoring or alerts remotely, it’s somehow an impetus to change our staffing levels in facilities or change the way that we support those facilities. A lot of the work that’s done in a data center is hands-on. So, if we had this type of scenario happen again, sending somebody home for a certain amount of time remotely still requires another individual to be physically onsite.”

Supply Chains

The early word from the nation’s largest data center operators was that supply chains and critical services were holding firm. Throughout the nation, data center service levels are largely reported to have held fast, at nominal or better levels.

For some mid-tier providers, however, maintaining that even temperament may have been more of a scramble.

“Not everything is continuously available,” Green House Data’s Thompson said. Vendors who were typically capable of happily fulfilling requests for products and services needed “yesterday,” he said, now often can’t meet those requests tomorrow or even the next day. Vendors for air-handling units and power supply systems, he noted, were warning that pending orders should perhaps be added to the system, because “they’ve got some uncertainty of what the supply chain’s going to look like over the next six to 12 months. So, there’s been efforts we’ve had to ramp up to try to get done sooner to make sure we can secure that equipment coming in.”

Some vendors have admitted that they’re looking at “creative ways” to meet equipment demands, Thompson admitted.

Immedion’s critical response plan, Moser told us, included making early contact with critical vendors “to understand any constraints they may have, to communicate to them the posture that we were taking and the staffing levels. We haven’t had any vendors push back and say they were unable to support us in this time; and the times we’ve had to call on them and rely on them, they’ve been just as prompt and reliable as they have been prior to COVID-19.”

FirstLight had similar early meetings with vendors after having ascertained how many days of inventory it had on hand for critical equipment. For example, in those meetings they learned ahead of time what additional shipping costs the vendors would incur for accelerating an order. Some vendors actually reached out to FirstLight first, including with warnings about the staffing challenges they faced. They, too, were reducing physical staffing, in some cases cutting warehouse staff from as many as 100 to as few as 10.

“We did a lot of the homework up front to understand what those time lines would look like and then adjusted accordingly,” FirstLight’s Rivas said. For instance, if a facility was installing new fiber trays every three days but its vendors were only capable of meeting demand every three weeks, it would place seven orders’ worth of trays in advance.

When every data center vendor thinks along the same lines, some may be driven to stockpiling. That was definitely the case for FirstLight.

“In the very beginning, as everybody was buying every piece of toilet paper in the world, in the [data center] supply chain people were buying as many servers as they could, or storage, or switches, or fiber cables,” Rivas told us. “So, there was a little bit of a drop in the inventory, but to me, it’s stabilized a lot now. There might be a lead time in delivery based on the number of staff they can have to get that equipment out the door of their warehouses, onto their trucks, and delivered. We understand that. But I don’t think there’s a situation where we feel like we can’t get inventory. We’re just managing it based on the current schedule and what the outlook looks like.”

Among countries that dealt with the pandemic earlier than the US and have since managed to flatten their respective curves, the danger still remains that a relaxing of social distancing policies may trigger a second or third wave of the virus. But there is a belief among the providers with whom we spoke that the increased attention to the supply chain will enable them to get by, if not entirely smoothly, even if a vaccine isn’t generally distributed until next year.

“I think there’s going to be some supply chain issues here and there,” Omdia’s Howard said. “But the momentum in the data center industry is quite heavy, and so far the feedback that we’ve seen from the large physical infrastructure providers is, they claim to be prepared and in a position where they don’t believe they’re going to have supply chain problems for things like power and cooling equipment. Like we keep saying, it’s early innings.”

If hospitals and emergency clinics are the front lines during this new and unfamiliar state of war, data centers are the warehouses and supply lines. When the history of our country during this bewildering period is written (and hopefully after it passes the school boards’ censors), it should record that for at least these few long months of 2020 data centers comprised one of the economy’s most resilient, stable industries. With all the safety factors in place and with demand from businesses either spiking or plummeting depending on their service profiles, as of the end of May, data center providers of all sizes remained on relatively solid ground.

But the crisis is far from over. Those who are just emerging from their first lockdown may yet be heading for a second. The true test of these data center providers’ resilience may yet lie ahead.