Mega Data Center Design: What DuPont Fabros Learned from Building Its Biggest Facility Yet

About one year ago, DuPont Fabros Technology, whose business is building massive data centers and leasing wholesale space to companies on a long-term basis, brought online its biggest facility yet: ACC7 in Ashburn, Virginia.

It was the first facility to use the company’s newest data center design, whose core concepts it said it would replicate in future projects. ACC7, with its total power capacity of 41.6 megawatts, was an exercise in mega data center design driven to a great extent by technology and business trends, Scott Davis, executive VP of operations at DuPont, said. Davis talked about designing the facility and what his team learned as a result in a presentation at last week’s Data Center World conference in National Harbor, Maryland.

The four most important new design elements at ACC7 were medium-voltage power distribution, medium-voltage isolated-parallel UPS topology, a high-efficiency cooling plant, and a new approach to laying out the computing space.

Medium-Voltage Power Distribution

DuPont decided to use medium-voltage power distribution because of recent advances in medium-voltage circuit breakers, switchgear, and power distribution units, Davis said. The approach gives the data center design more reliability and flexibility but also costs a lot less than a more traditional low-voltage design, primarily because it requires a lot less copper, he said. It reduces the total length of feeder wire required by a factor of seven.

To get a duct bank to a 2500 kVA rating using 480V power, the design would need eight sets of four-wire ducts with about 2,700 feet of copper wire. The 4160V system at ACC7 achieves the same rating with only 180 feet of shielded cable in a single three-wire duct. The design also requires a lot less concrete and PVC piping.

Here’s the cost comparison in detail from Davis’s presentation:

The medium voltage system is more reliable because there are fewer terminations and because the duct banks run cooler, according to Davis. The layout is more flexible because you can have longer feeders without the huge cost.

According to Davis, the new medium voltage PDU could be a disruptive technology. The system separates transformers from distribution panels, replacing 10 conventional 300kVA PDUs with a pair of oil-filled PDU transformers. The system is 99.6 percent efficient and takes up a lot less space, which for a data center provider means more room for customer cabinets.

The choice to go with medium voltage is not without its drawbacks, however. One challenge is there are fewer quality products on the market. Medium voltage is also a much bigger electrocution hazard, and construction workers are less familiar with methods and best practices for working with medium voltage. Other drawbacks are labor intensity of cable handling and terminations, tighter area for termination work, and a more complex relay protection scheme.

Audits, Smaller Deals Call for More Physical Segregation

DuPont has changed the way it lays out computer rooms at ACC7. The facility has 28 rooms, yielding nearly 250,000 square feet of computing space, with 1.48MW per room. Each room has the dual-transformer PDU for 2N electrical redundancy and 18 to 22 CRAH units. There are no raised floors.

Another first at ACC7 is steel mesh fencing to separate IT racks within a single room. More customers demand such separation, which the mesh enables without taking up a lot of floor space and without modifications to electrical and mechanical infrastructure. Mesh walls can separate DuPont’s cooling and power equipment in a room from the customer’s IT racks or separate sections of rows of equipment or individual rows from each other.

There are two trends at play. One is more stringent audit requirements. Companies providing services to especially privacy-sensitive customers, such as healthcare organizations, increasingly need to have physical segregation of some of their clients’ equipment, Davis said.

The other trend is customers who would traditionally use retail colocation space taking space at DuPont facilities. It used to be that the smallest customer at the provider’s facilities would have 500kW. Today DuPont is making more deals with customers who want from 100kW to 200kW and who want their equipment physically isolated from neighbors.

Warmer Temperature Means More Efficient Cooling

Because servers can now run warmer, DuPont is able to use more free cooling. The company went with a water-side economization plant, assisted by a chiller in Ashburn. The heat exchangers provide 65F to 70F water, running year-round and using the mechanical chiller as needed.

Higher operating temperatures allow the operator to use much less fan power on CRAH units – 60 percent less than traditional units – and to use much smaller chillers with less pumping power.

The caveat is more involved management of hot and cold air and mandatory containment.

More details about the ACC7 data center design here.