Dunking for Density: New Projects Pursue 3M's Take on Immersion Cooling

3m-applied-cooling

This immersion cooling project in Hong Kong was created by Allied Control using a two-phase cooling technique called open bath immersion (OBI), using 3M's Novec fluid.

We're continuing to see new examples of immersion cooling at meaningful scale. In July we brought you an update on an immersion cooling system at CGG using technology from Green Revolution Cooling. We've also been tracking early projects using "open bath immersion" cooling based on technology developed by 3M.

Open bath immersion (OBI) is an example of passive two-phase cooling, which uses a boiling liquid to remove heat from a surface and then condenses the liquid for reuse, all without a pump. The servers are immersed in 3M’s Novec, a non-conductive chemical with a very low boiling point, which easily condenses from gas back to liquid. The OBI technique, which we first saw at last year's Data Center World show, is now in use in a handful of sites. Here's an overview of some of these projects.

Allied Control

Hong Kong-based Allied Control is specializing in developing high-density cooling solution using 3M's Novec and OBI. The company has recently deployed a 500kW high performance computing (HPC) production installation known as Immersion-2 for a client in Hong Kong. This design uses OBI in standard 19-inch racks, and was deployed in less than six months. Allied Control says the system operates at a Power Usage Effectiveness (PUE) of 1.02, which would make it one of the most efficient designs in the world, even though Hong Kong has a hot and humid climate. The facility is located in a high rise building and fits in the size of a standard shipping container.

One issue with a rack-mounted approach to immersion cooling is weight. "The weight is indeed a small challenge for standard sized racks, but actually more due to the increased system density, not really the fluid," said Alex Kampl, VP of Engineering for Allied Control. "You also remove a lot of weight by not using air cooling. We've been working with a rack manufacturer who has been very helpful." Here's a look at the facility:

applied-control-racks

These racks are filled with tanks containing high-density servers immersed in cooling fluid. The installation in Hong Kong was created by Allied Control using 3M's Novec fluid. (Photo: Allied Control)

applied-control-novec

Inside the rack-mounted tanks, the heat from dozens of servers causes the Novec fluid to boil. The vapor cools when it reaches the condenser at the top of the tank and is then reused. (Photo: Allied Control)

The company previously built a dedicated immersion-cooled facility called Immersion-1 to cool a unique supercomputer comprised of FPGAs (field programmable gate arrays), which are semiconductor devices that can be programmed after manufacturing.

Allied Control created Immersion-1, a system using 6,048 FPGA chips combining 890 million logic cells, which will encompass up to 24 tanks. The company says that a similar installation using traditional air cooling would require more than 8,500 Dual Xeon 1U servers in more than 200 racks.

"Immersion-1 has become a massive prototype and proof of concept for a whole new generation of computing," Allied Control says on its web site. "Since the special application tweaks the maximum performance out of each FPGA, they generate much more heat than in traditional FPGA applications. Often, FPGAs have to be throttled down or are not running at maximum performance due to cooling issues. In case of Immersion-1, the cluster would not be able to run on passive cooling and the FPGA chip temperature rises  above its maximum specifications within seconds. Only by using immersion cooling it was possible to build and run Immersion-1 with its very demanding cooling requirements."

The company has also developed a design concept to adapt Intel's Xeon Phi coprocessor for HPC workloads in immersion cooling, and is  interested in developing high-density designs using Intel's Dense Form Factor (DFF) cards in open bath immersion.

"OBI is in its early stages, but I am sure we'll see exciting progress very soon," said Kampl. "Unfortunately we are wasting a lot of time right now to literally remove unnecessary parts from hardware built for air-cooling, so my hope is that system designers start offering similar hardware like the DFF cards."

The Allied Control technology will be on display at the 3M booth at next week's SC13 conference in Denver.

Next: Projects by 3M, Lawrence Berkeley Lab

3M

The 3M team that developed open bath immersion is working on a supercomputer as a proof of concept. "We are building a small supercomputer based on state of the art hardware that is normally water cooled," the 3M team reports on its Facebook page.

The half-rack (21U) system can accommodate up to 144 Intel Sandy Bridge CPUs. 3M has built a custom tank to hold the servers and Novec fluid, as well as custom cooling tower to support the system:

3m-cooling-unit

Hand-Built Dry Tower that will Cool the 3M Supercomputer System. (Photo: 3m)

3M commenced building its own supercomputer to provide a larger showcase for the potential of Novec and the OBI concept, hoping to jump-start interest among both users and equipment vendors.

"It has been challenging for us as the fluid provider to move this forward," said Phil Tuma, an Application Specialist at 3M Electronic Markets who has championed OBI cooling. "When we did find an end user interested in the technology, we couldn't point them to a system vendor, only a pail of fluid. With one exception, efforts with the big OEMs were largely a waste of time. They all move slowly. We decided early this year to approach smaller, more agile HPC companies interested in differentiating themselves."

Tuma said 3M has been working with HPC server vendor Cirrascale and modular data enclosure specialist Elliptical Mobile Solutions on developing open bath immersion deployments.

Other Installations

Lawrence Berkeley National Laboratory is also developing an OBI/Novec project, working with technical computing hardware vendor SGI to create an immersion cooling system for the Naval Research Laboratory (NRL).  The initiative is being funded by the Environmental Security Technology Certification Program (ESTCP), the Department of Defense's environmental technology demonstration and validation program. ESTCP works to help innovative technologies to overcome barriers to deployment. The LBNL/NRL project is expected to house a half-rack of SGI IceX hardware with 144 Sandybridge sockets. Under Turbo 2.0 conditions, it will draw over 40kW. 

The Mayo Clinic is also test-driving a Novec-based OBI solution. The clinic's Special Purpose Processor Development Group (SPPDG) has built an 80kW tank to serve as a test platform for developing OBI technology.