Servers on Demand: Custom Water-Cooled Servers in One Hour

ovh-servers-BHS

A wall of servers inside the new OVH facility near Montreal, Quebec. The former aluminum factory doubles as a data center and server manufacturing facility. (Photo: OVH)

In 2006, word began to emerge that Google was building its own servers to make its infrastructure faster and cheaper. The search giant wasn't alone, though. By that time, French hosting company OVH had been building its own servers for a number of years, developing a design that use water-cooling.

"We've been doing our own servers from the beginnings of OVH in 1999," said Germain Masse, the Chief Operating Office of OVH. "The 1U pizza box servers available at the time were real expensive. We didn't think the way the components and drives were positioned was smart. We found it would be better if we could reduce the case to a simple sheet of metal. We could make them faster and more efficient."

The concept for these early OVH designs - vanity-free hardware, minimalist server trays, a limited component set - may sound familiar. These are the guiding principles behind the Open Compute Project and other initiatives to customize hardware for cloud server farms. OVH was an unusually early adopter of the build-your-own hardware model, embracing the economics of hyperscale environments in its fast-growing hosting business.

OVH now has 150,000 servers running in eight data centers. The company's newest campus in Quebec is a unique facility, housing a factory to assemble servers and the data center to house them. It is the engine for OVH's ambitions in North America, where it hopes to sell hundreds of thousands of servers. The facility in Beauharnois is a former Rio Tinto aluminum factory located just 100 yards from a hydroelectric power dam and substations, providing access to more than 100 megawatts of power capacity.

ovh-server-assembly

OVH employees assemble the company's custom servers inside the Beauharnois data center facility. (Photo: OVH)

Inside the factory, a team of 25 workers man an assembly line where they build servers from boxes of components. Masse says OVH sources components from five different vendors, including Intel, whose Xeon and Atom chips are found in the company's servers. Each server board takes about 15 minutes to build, and after being tested can be quickly deployed to racks housed in another part of the building.

OVH says that when a hosting customer places and order through its web site, the server can be built and installed in Quebec in less than an hour.

"We've been doing this for 10 years now, so we know how many server orders we typically get," said Masse, who said orders sometimes experience bottlenecks when the company introduces new server products or opens new data center facilities.

Masse said building its own servers gives OVH the freedom to innovate.

"We are free to choose the right manufacturer for our motherboard and drives, and free to change it when the market changes," he said. "It allows us to choose the right equipment at the right time."

Next: The Shift to Custom Water-Cooled Servers

ovh-servers-line

An example of that freedome to innovate emerged in 2003, when OVH began experiencing difficult cooling its servers with traditional air cooling techniques.

"Since we weren't focused on the chassis or provider, we were free to think of new types of systems," said Masse. "This (cooling challenge) gave us an incentive to change the rules and introduce water cooling for all our servers."

OVH uses piping to bring water inside the chassis to a dissipator (heat sink) adjacent to the CPU. Bringing water directly to the chip allows the use of hot water cooling, which reduces the need chill the supply water. This allows OVH to operate the water-cooled servers with pumps and fans, and no compressors.

The water-cooled systems were designed to provide targeted heat removal for servers that live in a traditional data center with room-level cooling. The water-cooling can handle about 70 percent of the heat produced by the system, with the remainder being managed through airflow through the server. In 2006, OVH began to research new designs that would replace its room-level air conditioning with air economization, using fresh air from the outside to cool servers.

That research led OVH to create "The Cube" - a Borg-like design in which the entire building functions as an air handler. The servers are housed in an exterior corridor built around an open center, allowing for easy airflow through the facility.

OVH's latest servers include a traditional 1U server, as well as a new high-density design that can pack 144 servers into a traditional 42U rack, creating a power density of up to 20 kW per rack.

One of the advantages of the build-your-own model is that it also allows OVH to recycle some chassis and some components. This provides the flexibility to refresh one hardware component - such as a faster CPU - while retaining other elements.

"We refurbish and recycle all our servers so each has a second and third life," said Masse.

TOMORROW: Part 2: Cloning The Cube and Going Modular