Microsoft Reveals its Specialty Servers, Racks

microsoft-rack-design-470

The design concept for the Microsoft servers that power its global cloud computing platform. Click for a larger version of the image.

REDMOND, Wash. - As it seeks to slash power usage across its global cloud computing platform, Microsoft has been refining its designs for energy-efficient data center hardware. The company is now sharing the details of those designs, which feature customized servers, a high-efficiency power distribution system and in-rack UPS units.

“When Microsoft saw its server counts and data center footprint growing, it became clear that we needed to improve efficiency,” said Dileep Bhandarkar, a Distinguished Engineer at Microsoft, who oversees the company’s server hardware architecture. Bhandarkar began working with server vendors (OEMs) to develop specialized designs based on specs developed by the Microsoft server team. He recently discussed these designs with Data Center Knowledge.

Over the past three years, Microsoft has openly shared information at industry conferences about its data center modules, known as IT-PACs. But it has offered fewer details about the racks and servers inside those containers, limiting disclosures to events for small groups of industry researchers.

Half-Width Server Design

Microsoft's latest designs for its cloud-scale services are based on a  half-width design that fits 2 servers into a 1U slot in a rack. Each server board houses two CPU sockets, with room for 4 DIMM memory slots in each socket. The hardware currently running in Microsoft data centers follow the broad design concepts, but includes customizations by individual vendors and some app-specific modifications. The servers for Bing, Hotmail and Windows Azure reflect different implementations of the design goals.

This design concept allows Microsoft to fit as many as 96 servers into an extra-tall 57U rack, pushing power densities as high as 16 kilowatts per rack. Each rack includes at least two sets of battery packs providing short-term backup power, allowing Microsoft to operate without a central UPS system.

Microsoft also saves energy in its power distribution system, which brings 480V, three-phase AC power directly to the rack, where power supplies convert it to 12V DC power output for the servers. This eliminates power losses from the multiple AC-to-DC conversions common in centralized UPS systems.

In the IT-PAC, the servers have no fans, using air pressure within the module to manage airflow through the servers. That's an extension of the data center team's focus on eliminating unnecessary components on each server . “We've been doing that for the last three years,” he said. “When I came to Microsoft, every server had a DVD drive.”

Optimizing Server Energy Usage

The Microsoft team optimized its servers to use 40 to 60 watts of power (depending upon whether it uses 4 cores or 6 cores). The design emphasizes high-efficiency components that can operate in expanded ranges of temperature (up to 90 degrees F) and humidity.

Microsoft developed two types of specifications for its server vendors:

Bhandarkar says Microsoft has worked with vendors to reduce power use when processors are idle.  "It used to be an idle server would be 50 percent  of the power (used when the server is active)," he said. "We've pushed that down to about 30 percent."

Another goal is to reduce the power draw from DIMM memory modules. "The DRAM industry is getting the green bug," said Bhandarkar. "The market is shifting from 1.5V to 1.35V, and heading to 1.2V."

'Lots of Opportunities' for Innovation at the Rack

"We're working with a lot of the processor vendors, and helping them understand when our server loads look like," said Bhandarkar. " There lots of opportunities to optimize rack level power and cooling."

Shifting the UPS and battery backup functions from the data center into the server cabinet reduces power losses from multiple conversions that occur between the utility power grid and the data center equipment in a standard UPS architecture. ""We did enough analysis to convince people this (rack-level UPS) could be done," said Bhandarkar.

On the power distribution front, Microsoft examined a number of options, but focused on the power supply. "The entire data center industry revolves around 480 volts and the server industry likes 208 or 415," said Bhandarkar. "This doesn't make any sense. The infrastructure for 480V is too big change, so were driving the server industry to change power supplies."

Part of Broader Industry Conversation

The use of rack-level UPS and streamlined power distribution are similar in concept (if not execution) to cloud-scale system refinements that have been publicly discussed by Google and the Facebook Open Compute project. Bhandarkar says the disclosures are helpful in advancing innovation and best practices in the data center.

“This is not rocket science," said Bhandarkar. "Smart people facing the same problems will come up with similar solutions. Driving the entire industry forward helps us in the long run."

While Microsoft has optimized its design, Bhandarkar avoids the word "custom" in describing its process. “I call us a leading-edge adapter," he said. "Our stuff is not custom. We don't own the IP (intellectual property). We encourage our vendors to sell it to others so they can recoup their investments. We would like them to sell it to the industry at large. "

Here's a look at Microsoft's design concept for racks and UPS systems. Click for a larger version of the image.

microsoft-rack-concept-470