Rise of Direct Liquid Cooling in Data Centers Likely Inevitable

NEW ORLEANS - It's been a decade since cooling vendors began predicting that power densities would force servers to be cooled by liquid rather than cool air. Instead, the industry has seen major advances in the efficiency of air cooling, while liquid cooling has been largely confined to specialized computing niches.

Experts in high performance computing say that will begin to change over the next three to five years due to increased data-crunching requirements of scientific research, cloud computing, and big data. A key driver is the HPC community's bid to super-charge the processing power of supercomputers, creating exascale machines that can tackle massive datasets. The exascale effort is driven to a large extent by the U.S. government, which spends tens of millions annually on grants for research in exascale computing.

"In the HPC world, everything will move to liquid cooling," said Paul Arts, technical director Eurotech. "In our vision, this is the only way to get to exascale. We think this is the start of a new generation of HPC, with enormous power. We are just at the beginning of the revolution."

A recent report by 451 Group said liquid cooling was making somewhat of a comeback outside of the world of scientific computing too. More and more non-scientific workloads are approaching the level of compute power previously reserved for research.

Higher Density vs. Hydrophobia

The exascale initiative is one of several trends boosting adoption of liquid cooling. HPC researchers say liquid cooling offers clear benefits in managing compute density and may also extend the life of components. Although it can offer savings over the life of a project, liquid cooling often requires higher up-front costs, making it a tougher sell during procurement.

In addition, many IT and facility managers still experience "data center hydrophobia," the concern that introducing water into the IT environment boosts the potential for equipment damage.

Vendors of liquid cooling technology say they are seeing business gains, including traction from the rapid growth of the bitcoin sector. But HPC represents the front lines for liquid cooling, and will be the harbinger of any large-scale shift.

At a packed session on liquid cooling at the recent SC14 conference in New Orleans panelists outlines the benefits of liquid cooling and made a case for educating end users to overcome wariness about the technology.

"As leaders in HPC, we have to drive liquid cooling," said Nicolas Dube, distinguished technologist at HP who focuses on data center design. "By 2020, the density and heat we will be seeing will require liquid cooling. We’ve got to push it."

Light Adoption, But a Boost From Hyperscale

Despite widespread concern about rising heat loads in the data center, liquid cooling systems tend to be widely discussed and lightly implemented. A recent survey by the Uptime Institute found that just 11 percent of data center operators were using liquid cooling.

Frequent predictions of a huge spike in data center power densities have also proven premature, at least in the enterprise. Although many customers clamor for space that can support high power densities, Uptime said the median density in enterprise and colocation centers remains below 5kW per rack, a level that can be easily managed with air cooling.

coolit-sc14

At SC14, CoolIT Systems illustrated a design for a full rack system cooled by warm water pumped from an adjacent chilling distribution unit (CDU). The piping is visible overhead, running between the rack and CDU. (Photo: Rich Miller)

But the emergence of massive hyperscale computing facilities, along with the business world's embrace of cloud computing and big data, is beginning to change the playing field. The 451 Research paper noted that hyperscale operators like Facebook and Google have “legitimized a less conservative approach to the design and operation of facilities and paved the way for use of technologies such as (direct liquid cooling) by other operators.”

Methods of Liquid Cooling

Liquid cooling comes in a variety of flavors. "Anytime we talk about liquid cooling, we need to talk about what we mean," said Michael Patterson, a senior power and thermal architect at Intel. "There are very different approaches."

Cold water is often used to chill air in room-level and row-level systems, and these systems are widely adopted in data centers. The real shift is in designs that bring liquids into the server chassis to cool chips and components. This can be done through enclosed systems featuring pipes and plates, or by immersing servers in fluids. Some vendors integrate water cooling into the rear-door of a rack or cabinet.

Cool IT Systems, which makes direct-contact liquid cooling systems, is seeing strong growth this year, according to CEO and CTO Geoff Lyon. The company has seen revenue growth of 382 percent over the past five years, placing it on Deloitte's Fast 50 list of fast-growing Canadian technology companies. A growth area this year has been the bitcoin sector, where CoolIT is working with several hardware vendors on direct-contact systems for custom ASICs to mine cryptocurrencies.

Asetek has seen year-to-year revenue improvement of 53 percent in the data center sector, primarily boosted by customers in the defense sector. The company, which partners with Cray and other OEMs, is beefing up its sales staff to focus on opportunities in the HPC sector.

Opportunities in Immersion

On the immersion front, Green Revolution Cooling supports the world's most efficient supercomputer, the Tsubame-KFC system, which houses immersion tanks in a data center container. The company has several installations spanning multiple megawatts of power loads, including a seismic imaging application at CGG in Houston.

3M, which uses a slightly different approach known as open bath immersion, has seen traction in the bitcoin sector. The company's Novec fluid was used in a high-density bitcoin mine within a Hong Kong high-rise building that can support 100kW racks.

Immersion solutions usually come into play when an end user is building a new greenfield data center project, and is seen less frequently in expansions or redesigns of existing facilities. Direct-contact solutions are more likely candidates for existing facilities, but require bringing water to the rack requires piping (either below the raised-floor or overhead) that is not standard in most data centers.

liquidcook-sc14

LiquidCool Solutions shows off servers immersed in a tank of liquid coolant at the SC14 conference in New Orleans. (Photo: Rich Miller)

In the HPC sector, liquid cooling presents opportunities in two areas, according to Ingmar Meijer, a senior researcher at IBM Research.

"If you’re someplace where energy is expensive, it’s about cost," said Meijer, the lead architect of SuperMUC, an IBM petascale supercomputer in Munich. "If you’re in Oak Ridge (Tennessee), where its 3 or 4 cents (per kW hr), the benefit is clearly about density. Liquid cooling makes the best sense where energy costs are high."

Up-Front Cost vs. Long-Term TCO

Meijer says liquid-based solutions like the warm water cooling approach used in SuperMUC offer significant savings over the life of a project. But the up-front installation cost can be higher than those for air-cooled systems, which can be a barrier to adoption for some users.

"If we don’t get the cost down, the acceptance of liquid cooling will suffer," said Meijer. "The customer is not always good at calculating their TCO."

HP's Dube agreed that cost issues can complicate procurement, but said HPC professionals must press the issue and evangelize the benefits of liquid cooling.

"You have to be ready to climb that hill and carry the flag," said Dube. "There’s no liquid cooling system that will ever be as cheap as a Chinese-made air-cooled platform.”

Reliability as a Selling Point?

While some data center professionals are wary of introducing liquids into racks and servers, HPC veterans say liquid cooling can offer advantages in reliability.

"In almost every instance of liquid cooling, the processor will run much lower and be more reliable," said Intel's Patterson. "With liquid cooling, there will always be less temperature fluctuation than with air cooling, which will help reliability."

A number of solutions offer direct-contact cooling for server memory as well as processors. Panelists at SC14 were split on whether liquid cooling improved the reliability of memory, with some seeing improvement and others citing failure rates equivalent to air cooling.

The Future: The Absolute Zero Data Center?

The new frontier for liquid cooling may be network equipment, according to panelists. Extending liquid cooling to network gear - whether using direct-contact cold plates or immersion - could free data center designers from the need to cool the data hall to support network equipment. Networking advances such as silicon photonics could boost interest in new cooling technologies.

"For switches, there will be a transition in the next 5 years," said Dube. "As we go to photonics, that will need liquid cooling."

Some approaches to exascale computing could drive larger paradigm shifts in facility design. An example is quantum computing, an approach being explored by Google and several government agencies. Quantum computing pioneer D-Wave had a booth at SC14, and some see its technology as the best way to reach exascale within the target power envelope of 20 megawatts. The challenge: D-Wave's systems need to operate at a temperature near absolute zero, or about 273 degrees below zero fahrenheit. The systems operate inside refrigeration units using liquid helium as a coolant.

sc14-dwave

D-Wave Systems had a booth at SC14 to discuss its approach to quantum computing, in which processing occurs at temperatures nearing absolute zero. (Photo: Rich Miller)

Corrected: A previous version of this article erroneously stated that the SuperMUC supercomputer was located in Zurich. It is in fact located in Munich, and the article has been updated with the correct city.