Sandia Adopts Unique Cooling Approach

sandia-airflow-1

This diagram shows the airflow for the Red Sky supercomputer at Sandia National Laboratories, which combines cool air that flows through racks equipped with a rear-door Glacier refrigerant cooling system.

When David Martinez wanted to discuss his concept for cooling the Red Sky supercomputer at Sandia National Laboratories, he had trouble finding anyone who could offer him guidance.

“It turns out no one had ever done this (in a data center), so we had no one to talk to about how to do it,” said Martinez, a Distinguished Technologist for Computer Infrastructure & Operations at the Department of Energy lab in Albuquerque, New Mexico.

Raised Floor Meets the Glacier Door

His idea: develop a cooling system that combined the use of traditional raised-floor air cooling with refrigerant-cooled rear door heat exchangers. Martinez wanted to use an approach known as “laminar air flow,” in which cool air from a single perforated floor tile is drawn through a series of cabinets with rear-door cooling units, extending the effectiveness of both systems.

So Martinez went ahead and built it, and shared about his experience with peers at the recent AFCOM Data Center World conference in Orlando, Florida.

RedSky uses 36 racks of cabinets filled with hardware from Oracle (Sun Microsystems) providing 10 teraflops of computing power apiece. Each cabinet uses about 32 kilowatts of power. The system is currently ranked 16th in the Top 500 list of the most powerful supercomputers, after debuting in 10th position in 2010.

The design uses just a single row of perforated tiles in the raised floor, reducing the under-floor cooling requirements by 70 percent. After the air travels through the rows of racks and their cooling doors, the air is returned via a ceiling plenum.

Emerson, Sun Team on Door and Pumping Unit

Red Sky uses the Sun Cooling Door (also known as Project Glacier) designed jointly by Sun Microsystems and Emerson Network Power, which was demonstrated at the SC08 show. The Sun Cooling Door 5600 attaches to the back of cabinets and uses an inert refrigerant gas called R134. The unit is supported by a Liebert XD pumping unit. The passive design that doesn’t require additional fans to circulate air, saving on energy used to power the fans.

The XD pumping unit cools the refrigerant using a heat exchanger and a chilled water loop. In the warmer months, this 45-degree water loop is cooled by a chiller system. For the remainder of the year, Sandia uses a plate-and-frame heat exchanger, is a type of heat exchanger that uses metal plates to transfer heat between two fluids.

When the chiller is in use, the system has a Power Usage Effectiveness of 1.27, Martinez said. When the plate-and-frame heat exchanger s active, that PUE drops to 1.19.

Red Sky operates alongside Red Mesa, a cluster for the National Renewable Energy Laboratory (NREL). The combined Red Sky/Red Mesa supercomputing platform dramatically reduces the time required to simulate complex fuel models, from four months to just four weeks, allowing researchers to accelerate the pace at which they can address these complex problems. Its speed also reduces the need for laboratory and field testing, allowing for energy reduction far beyond its data center.

This video from Sandia provides an overview of Red Sky/Red Mesa, as well as a closer look at the data center design and cooling system, which begins at about the 6:30 mark.