Cloud by the Megawatt: Inside IBM’s Cloud Data Center Strategy

Whenever Francisco Romero hears a client tell him they need so many thousand cloud VMs and an X amount of storage and connectivity across the sites that will host them, in his head, he converts the requirement into megawatts. They may be asking for 10,000 VMs of various kinds, but to him that means he has to deliver 1MW or 2MW of data center capacity, depending on the VMs. Little drives home the physical nature of cloud computing better than this.

As chief operating officer of IBM SoftLayer, Romero oversees the data center strategy that underpins IBM’s entire cloud business, as the company battles in the market dominated by Amazon Web Services and, to a much lesser extent, Microsoft Azure. All major US hardware vendors have tried and failed to become formidable rivals to Amazon in the cutthroat cloud infrastructure market; all except Oracle and IBM, which continues to expand both technological capabilities of its cloud and the global infrastructure that makes it all possible.

IBM’s Infrastructure-as-a-Service cloud, its Platform-as-a-Service cloud, called Bluemix, as well as the plethora of Software-as-a-Service offerings, including services enhanced by Watson, its Artificial Intelligence technology, run in SoftLayer data centers. “We are the main delivery mechanism for those,” Romero, who joined SoftLayer four years before IBM acquired the Dallas-based data center provider in 2013, said.

There are three basic flavors of infrastructure for nearly every cloud service IBM provides: public cloud, private cloud, and on-premise private cloud. SoftLayer data centers host infrastructure for the first two, he explained. This excludes cloud managed services, for things like SAP or Oracle applications, which are hosted in non-SoftLayer facilities IBM either builds or leases around the world.

Taking into account IaaS alone, IBM had the fourth-largest cloud market share in 2015, according to Structure Research. With an estimated $583 million in revenue that year, its IaaS business was slightly behind Rackspace’s, which turned over $646 million. For the sake of comparison, AWS raked in $7.88 billion, while Microsoft Azure is estimated to have made $1.209 billion, according to Structure.

Read more: Top Cloud Providers Made $11B on IaaS in 2015, But It's Only the Beginning

Since early 2014, IBM has been expanding its cloud data center footprint. That January, it announced a commitment of $1.2 billion to add 15 data centers around the world to the 13 SoftLayer and 12 IBM facilities that existed at the time. These expansion activities focused in part on adding new locations but also on expanding capacity in markets where the company already had cloud infrastructure.

SoftLayer’s website currently lists 31 data centers in 22 markets across North America, Europe, and Asia Pacific. There are also numerous network Points of Presence in places like Los Angeles, Chicago, New York, Stockholm, and Perth, among others.

Map of IBM SoftLayer data center locations, as of October 2016. (Image: IBM)

Growing in Place

Expansion in existing locations has been an important part of the strategy. As more enterprise customers deploy more critical workloads in the cloud, they are engineering those applications to take advantage of multiple availability zones in each region for load balancing and disaster recovery. In response, Romero’s team has been adding data centers in existing regions to increase the number of availability zones IBM’s cloud offers.

The European Commission’s annulment of Safe Harbor laws last year added another incentive to expand capacity in existing regions. When the set of rules that had for 15 years governed cross-border data flows between the US and all EU countries was in place, cloud customers had little concern about having a primary data center in Frankfurt, for example, and a disaster-recovery site in Amsterdam. Once Safe Harbor was struck down and each country was left to its own devices to regulate data flows, cloud users became a lot more interested in multi-site infrastructure within a single country, Romero said. This past August, Europe enacted a replacement for Safe Harbor, a set of regulations titled EU-US Privacy Shield, but companies have been slow to sign up, many weighing other compliance options.

Expanding the Empire

The newest markets IBM recently entered with SoftLayer data centers are Seoul and Oslo. The company weighs a number of factors before deciding to extend infrastructure into a new market. The most obvious one is demand: there has to be a critical mass of customers it knows will want cloud capacity in a new geography before it makes the investment. Another big consideration is network infrastructure. India, for example, has lots of demand for cloud services, but very limited and expensive connectivity options, Romero said. IBM announced the first SoftLayer data center in India, located in Chennai, one year ago.

Like other cloud providers, SoftLayer prefers to team up with a local partner when going into a new market. It partnered with local telcos in both Seoul and Oslo on launching data centers in those cities. Besides having a partner familiar with the local market, this strategy also helps in places with subpar infrastructure. Partnering with a telco, for example, means the partner will be incentivized to invest in new infrastructure to ensure the venture succeeds, Romero explained.

Cloud by the Megawatt

While emerging markets in Asia are growing fast, demand for cloud services in the US continues to grow faster than anywhere else in the world, according to Romero. “The US for sure is ahead of everybody else,” he said. This is because the US cloud market is the most mature one.

While even in Europe most cloud customers are still testing the waters with cloud services, US companies are now starting to shift really large critical workloads to the cloud, he said. IBM is now seeing cloud deals in the US that require commitments of 3MW to 5MW of data center capacity, keeping Romero and his team busy working in a whole new paradigm in capacity planning and deployment.

Unlike Amazon, Microsoft, and Google, who lease data centers and build their own, SoftLayer has so far relied exclusively on data center providers. Today, it uses around six providers, but its biggest one is Digital Realty Trust.

IBM is Digital Realty’s biggest tenant by annualized rent, occupying close to 1 million square feet in 23 locations and generating more than $115 million in annualized rent for the San Francisco-based data center REIT, according to investor documents available on Digital’s website. These include both leases with SoftLayer and with IBM, which has numerous non-SoftLayer data centers built for various purposes around the world.

One of the reasons Digital Realty gets so much SoftLayer business is that the cloud provider has a very specific uniform design for all data centers it deploys, and Digital has been delivering on that design for many years. It includes specific cooling and power density requirements, room layouts, strict PUE caps, tight SLAs for temperature and humidity ranges, and infrastructure redundancy, among other things.

Unlike other cloud giants, who use N+1 redundancy for UPS systems and other electrical infrastructure components in their mega-scale data centers, SoftLayer requires 2N UPS and treats this requirement as a major differentiator, Romero said. Put simply, N+1 means there are enough UPS units to handle the required load and one extra to compensate if one of the primary ones fails. 2N means there is full redundancy for the entire UPS plant.

SoftLayer’s design requires the ability to deliver variable power densities on the data center floor, anywhere between 3kW and 16kW per rack, for example. While power draw by physical hosts for cloud VMs is fairly predictable, power can fluctuate widely for bare-metal cloud servers, Romero said. Those rack-density ranges also change on a regular basis. The team revisits them about every six months, as Intel introduces new chipsets, to evaluate how much power the next generation of servers will need.

Changes Ahead

Romero doesn’t expect SoftLayer’s design requirements and its overall data center strategy to remain static. IBM’s cloud business has reached a scale where starting to invest in building its own data centers is an ongoing discussion for his team.

They are also evaluating new design innovations, such as modular in-row cooling, and becoming curious about data center providers with new ideas about structuring leases with their clients. Instead of pushing the client to reserve big chunks of capacity ahead of time, whether there is immediate need for it or not, these companies offer the option to increase the commitment incrementally, while still willing to build ahead to ensure capacity is available when the need comes. “Now we’re seeing providers that are willing to be much more creative with those commercial terms,” Romero said.