Facebook Dials Up Infrastructure to Render "Look Back" Videos

fb-nc-racks-470

Facebook's armada of servers delivered up to 9 million "Look Back" videos per hour during its recent 10th anniversary celebration (Photo: Rich Miller)

Aiming to celebrate its 10th anniversary in style last month, Facebook (FB) developed "Look Back" videos and ended up challenging even its own massive infrastructure, rendering over 720 million videos. As a last minute idea within the company, the Look Back video tool would enable people to generate one-minute videos that highlighted memorable photos and posts from their time on Facebook.

The video project would prove to be a challenge for engineers to put it together in less than a month, and for the Facebook infrastructure and network to handle the additional processing, storage and network impact it would incur. With a mission to render faster and store better the team rose to the challenge, Facebook figured it would need around 25 petabytes of storage and an estimated 187 Gbps of additional bandwidth - based on a potential 25 million videos generated within a day of the anniversary, each shared to and viewed by five people.

Animate Photos for a Billion People

The compute power available in Facebook's global infrastructure would make any rendering project easy, but a key challenge for the Look Back project was to not disrupt regular Facebook operations. On top of every day infrastructure demands, Facebook was about to launch its Paper app,  and a week later the site would be inundated with the start of the 2014 Winter Olympics.

In a recent blog post, the Facebook infrastructure team explains that "for compute, although we had tens of thousands of servers available, our data center power efficiency plans were not designed for all of them to be running at full capacity at all times. We decided to track power usage while the jobs were running and added dials to the software to allow us to slow down if needed."

The team reviewed configurations on its Haystack media servers, especially the geographical location of the server, to help prevent disruptions to normal site operations. The Look Back videos would be isolated in their own servers across both U.S. coasts and they would treat them as different media types instead of regular uploads. The video rendering jobs would be distributed and scheduled, as users would edit their videos and then quickly render them on-demand.

9 Million Videos Rendered per Hour

After quickly prototyping and testing the tool internally, it was ready for launch. In the days and hours before launch the team watched the space used in Haystack grow from 0 bytes to 11 petabytes, rendering over 9 million videos per hour at peak.  The video feature went live on February 4th, and people quickly checked out their videos and shared with friends. Facebook estimated that about 10 percent of the people who saw their video would share it, and to their surprise, well over 40 percent of the viewers were sharing.

Without a single breaker tripping, Facebook's Look Back video project logged some impressive statistics:

Outgoing bandwidth of 450 Gbps is impressive - as the blog post states, "even by Facebook standards."  That amount of bandwidth is equivalent to almost every single person in Kansas City simultaneously using 1 Gbps. The success of the tool is a testament to the capacity of Facebook's global infrastructure and to the almost internally crowdsourced team of engineers and developers that put it together in such a short amount of time.

"The Look Back project represented the things we love about Facebook culture: coming up with bold ideas, moving fast to make them real, and helping hundreds of millions of people connect with those who are important to them," the company wrote.