NSA Exploring Use of Mineral Oil to Cool its Servers

Security secrets may soon be stored on swimming servers. The National Security Agency has been testing the use of immersion cooling for its massive data centers, dunking servers in tanks of a coolant fluid similar to mineral oil. The agency says the technology has the potential to slash cooling costs.

"The National Security Agency's Laboratory for Physical Sciences (LPS) acquired and installed an oil-immersion cooling system in 2012 and has evaluated its pros and cons," the agency said in a technology publication. “Cooling computer equipment by using oil immersion can substantially reduce cooling costs; in fact, this method has the potential to cut in half the construction costs of future data centers."

That's of interest to the NSA, which spent more than $1.5 billion to build a massive data center in Bluffdale, Utah, that spans more than 1 million square feet of facilities. That's why it has been testing immersion cooling technology from Austin-based Green Revolution Cooling. The initiative reflects the NSA's ongoing interest in adopting cutting-edge technology in its computing infrastructure.

Liquid cooling is used primarily in high-performance computing (HPC) requiring high-density deployments that are difficult to manage with air cooling. Interest in liquid cooling has been on the rise as more applications and services require high-density configurations, prompting data centers to consider infrastructure previously limited to HPC and supercomputing facilities.

Data centers at scale

The NSA says the massive Utah facility, along with a similar one under construction near Baltimore, will be used to protect national security networks and provide U.S. authorities with intelligence and warnings about cyber threats. But the agency data centers have become a flash point for controversy in the wake of public disclosures about the NSA’s covert data collection efforts.

The project will have a power capacity of 65 megawatts, making power a big component of its operations. The 1 million square-foot Camp Williams facility houses 100,000 square feet of data center space, while the remaining 900,000 square feet is used for technical support and administrative space.

eff-nsa-utah-data-center-47

An aerial view of the NSA data center in Utah. (Photo: Electronic Frontier Foundation and Wikimedia Commons)

Green Revolution says its liquid-filled enclosures can cool high-density server installations for a fraction of the cost of air cooling in traditional data centers. The company’s approach allows users to operate servers without a raised floor, computer room air conditioning (CRAC) units or chillers. Green Revolution’s CarnotJet cooling racks are filled with 250 gallons of dielectric fluid, with servers inserted vertically into slots in the enclosure. Fluid temperature is maintained by a pump with a heat exchanger using a standard water loop.

Intel recently concluded a year-long test with immersion cooling equipment from Green Revolution Cooling, and affirmed that the technology is highly efficient and safe for servers. Current GRC projects include Tsubame 2.0, the world's most energy efficient supercomputer, and energy exploration specialist CGG, which operates an entire data hall of submerged servers.

Advantage of mineral oil

Mineral oil has been used in immersion cooling because it is not hazardous and transfers heat almost as well as water but doesn’t conduct an electric charge.

"While mineral oil does not have the heat capacity of water, it still holds over 1,000 times more heat than air," wrote David Prucnal from the DoD's Advanced Computing Systems, in The Next Wave, a quarterly research publication from the NSA.

The primary advantage of liquid cooling is that it will support much higher power densities than air cooling. The NSA said that immersion cooling systems can support loads of 30kW per rack with no special engineering or operating considerations, compared to an upper range of 10kw to 15kW per rack with air cooling.

cgg-grc-quad

Four of the many tanks of servers submerged in liquid coolant at a CGG data center in Houston, Texas. (Photo: Rich Miller)

Immersion cooling also allows the removal of fans, which are standard on most commercial servers and maintain proper airflow through a server chassis, consuming about 10 percent of the total energy use for a server.

Prucnal found that immersion technology also has the potential to lead to fewer equipment failures by maintaining an even temperature across server components, and reducing exposure to dust and dirt from air blowing through the equipment.

“The final side benefit of immersion cooling is silence," Prucnal wrote. "Immersion cooling systems make virtually no noise,” Prucnal writes. “This is not an insignificant benefit, as many modern air-cooled data centers operate near or above the Occupational Safety and Health Administration's allowable limits for hearing protection."

Heavy racks

One potential challenge is the weight of immersion racks, Prucnal writes, noting that the a Green Revolution rack loaded with servers and cooling fluid can weigh 3,300 pounds, or about 1.6 tons.

But the opportunity for savings in future facilities is significant, Prucnal noted as he applied the economics to the scale of the agency's recent builds.

“For large data centers, where the technical load is in the neighborhood of 60 MW, construction costs can approach one billion dollars," he notes. "This means that about 500 million dollars is being spent on cooling infrastructure per data center. Since immersion-cooled systems do not require chillers, CRAC units, raised flooring, and temperature and humidity controls, etc., they offer a substantial reduction in capital expenditures over air-cooled systems. Immersion cooling can enable more computation using less energy and infrastructure, and in these times of fiscal uncertainty, the path to success is all about finding ways to do more with less."