Water-cooled Solutions for High Density Rack Cooling

Graham Whitmore is the president and CEO of the Motivair Corporation.

Server densities are steadily increasing for HPC applications in legacy, government and institutional data centers. As densities increase, traditional cooling systems are struggling to keep up. Water cooling is a concern for many data center operators. However, the heat transfer rate of water is about 3-4 thousand times that of air so the efficiency of water cooling is compelling.

A look into cooling solutions

Air cooled and water-cooled CRAC units have been the mainstay of data center cooling for many years.  Originally designed for mainframe cooling, CRAC units provide bulk cooling of data center air with underfloor air distribution. These are not suited for high density rack cooling because they cannot direct sufficient cooling capacity or airflow through high density racks. Adding fan-powered floor grilles in front of racks helps but the servers are still dependent on their own fans to draw air over the servers.

Water-cooled in-row coolers are suitable for low-medium densities up to around 15kW, but they require expensive aisle containment structures in order to function to their full potential, while increasing the total rack footprint by up to 50 percent. Situated between racks and containing a cooling water coil and fan(s), in-row coolers function by drawing air from the hot aisle and discharging cooler air to the cold aisle. The server fans draw this cooler air across the servers, which again is the major limiting factor.

Passive rear door coolers were the first to address directly removing server heat at its source, using water as the cooling medium. These are replacement rear doors with a chilled water coil mounted inside. Their limiting factor is they must rely on the server fans to push the hot server air across the resistance of the cooling coil. Maximum cooling capacity is limited to about 15kW by coil performance with airflow from the server fans. Server fan power is a critical factor in overall server energy consumption and adding coil resistance to the server fans increases their power draw.

Direct on-chip cooling is a recent water-cooled solution for high density rack cooling but is not widely used due to the high initial cost, invasive installation inside the servers, and highly specialized installation/service. The most attractive aspect is the ability to use warmer water for this system, due to the high contact temperature within the servers. This reduces the cooling cost of outdoor radiator fans or evaporative towers without the need for refrigeration.

Active rear doors are comprised of a chilled water coil, variable speed (EC) fans, water control valve and PLC controls. They are dynamic and able to provide the exact cooling capacity, airflow and water flow for any changing heat load up to their rated capacity. Active rear doors extend the front to rear dimension of the racks by about 9”, consuming far less space than in-row coolers. Slightly narrowing the aisles while maintaining rack width saves valuable floor space and allows for future expansion.

Remove server heat at its source

A recent independent test by the Lawrence Berkeley National Laboratory determined that in optimum operating conditions only 70-75 percent of the server heat is removed at source, leaving at least 25-30 percent of the heat to be removed by the data center AC system. Less than optimum operating conditions (rack loading, water temperature, etc.) reduce on-chip cooling accordingly.

Active rear door coolers are designed to remove a minimum of 100 percent server heat at its source.  Current cooling capacity ranges to 45kW. They can also be used to remove the data center building, lighting and personnel loads so no other AC is required in the white space of a rack-based system.

By removing a minimum of 100 percent server heat, active rear doors are able to deliver return air to the room at or below the server entering air temperature. This also allows the servers to operate at a lower temperature than other systems, improving their overall efficiency and life expectancy.

eBay recently installed active water-cooled doors in the Phoenix data center after a complete evaluation of alternative systems to remove 35kW per rack.

A few things to consider

Active rear door power cost is a combination of EC fan power and external mechanical cooling. They are designed to operate with 65-75F entering water, which can be from a chilled water system, or blended water from other sources including aquifer water, radiators, towers etc. to minimize energy usage.

The running cost of all chillers is reduced by >30 percent when operating with 65F leaving water, instead of industry standard 45F. This reduction is achieved when the difference between the chiller evaporating and condensing pressures is narrowed by higher water temperature in the evaporator, and the refrigeration compressors consume proportionally less power.

The growing use of free cooling and adiabatic chillers in Northern and Central states reduces chiller power ≥ 70 percent when delivering 65F or higher water temperature. This is highly advantageous when applied to water-cooled data center cooling systems which can operate at higher temperatures.

An Independent global user has confirmed the fan power in active doors is offset by the reduction in server fan power, due to reduced frictional resistance, effectively eliminating rear door fan power from the energy equation. EC fans in active doors are controlled thermostatically to operate at the minimum power relative to server load. An air pressure switch overrides in the event that server fan output ever exceeds that of the rear door fans.

Today, water cooling is a rapidly growing design consideration in high density rack cooling. Active water-cooled rear door coolers offer reduced occupied space, ease of installation, and increased efficiency with lower operating costs when combined with high efficiency chillers. Most importantly they deliver a minimum of 100 percent server heat removal at an attractive initial and operating cost.

Industry Perspectives is a content channel at Data Center Knowledge highlighting thought leadership in the data center arena. See our guidelines and submission process for information on participating. View previously published Industry Perspectives in our Knowledge Library.