First Look: Facebook's Oregon Cold Storage Facility

fb-hdds-470

Facebook's cold storage systems incorporate Western Digital hard drives. (Photo: Jordan Novet)

PRINEVILLE, Ore. - Last Thursday, Facebook began migrating data - primarily pictures - to its newly constructed cold storage facility, within walking distance from its two huge data halls containing thousands of servers. On Tuesday, it opened the facility to reporters, providing a look at a hyperscale implementation of custom-built long-term storage.

The new building can be likened to an attic for cost-effectively housing data that hasn't been accessed in a while - and might never be again - but still ought to be kept around.

"What's really different about this place ... (is that it's intended for the) storage of data over the long term," said site director Chuck Goolsbee, as he entered the new building on Facebook's campus in the middle of the Beaver State.

As Facebook users add hundreds of millions of new photos to the site every day, the company is responding by treating different kinds of data in different ways. Not every company can afford to create custom designs to meet its needs, but it could be that the tiered approach will become more popular as data proliferates, not least because Facebook has revealed many aspects of the cold storage through the Open Compute Project.

The racks that executives showed off were largely kept clustered together in a small portion of the available square footage inside the building. The few that were on site sat several feet away from one another. But this configuration won't last. Over time Facebook will add racks full of cold storage on either side of those in place currently. The new racks  will connect to the switches inside these initial racks. After all, the hard disks are connected to a couple of nodes in each rack, leaving plenty of space to fill up ports on the switches.

fb-coldstorage-racks-470

Specifications for Facebook's cold storage, shown here at the company's data center in Prineville, Ore., have been made public through the Open Compute Project. (Photo: Jordan Novet)

Each disk in the cold storage gear can hold 4 terabytes of data, and each 2U system contains two levels of 15 disks. This configuration allows for 4 petabytes of cold storage in a rack (each storage head has 2 PB attached and there are 2 heads per rack).

The disks are rarely spinning - perhaps one will at any given time. As a result, these use less power than Open Compute racks filled with servers, and there's "no need to put electrical infrastructure on each row," Goolsbee said. Instead, power can be distributed across multiple racks.

Goolsbee doesn't think Facebook is the only company with good reason to consider implementing highly efficient tiers of storage. Other webscale companies could be inclined to do so, if they haven't already. Web companies with smaller scale will follow suit, then enterprises and eventually governments, he said.

Then again, wholesale copying might not be the game plan.

"It's not about who builds this, but what can you build from these basic blocks. (Companies) can build exactly what they need," said Goolsbee, who previously ran the digital.forest colocation facility in Seattle.

fb-coldstorage-chuck-470

Chuck Goolsbee, site director of the Facebook Prineville data center campus, with one of the company's new cold storage racks. (Photo: Jordan Novet)

While adoption of Facebook's take on cold storage might not come overnight, companies can access the designs right now. Cold storage Specifications on the Open Vault storage gear are available through the Open Compute Project.

Facebook director of infrastructure Jason Taylor called for flash for cold storage earlier this year. Even so, there was no flash-based cold storage in the new building. "It's just not ready for cold storage, for obvious reasons," Goolsbee said. But given Taylor's recent comments, that could change in the years to come, as the price of flash per gigabyte gets closer and closer to that of hard disk storage.

Cold storage won't be installed at every Facebook data center, just at the Prineville site and the one in Forest City, N.C, a spokesman said. And that should be fine for now. Facebook is projecting that the cold-storage building will be filled to capacity by 2017, Goolsbee said.