At Facebook Data Centers, New Protocol Helps Add Servers Faster

Things move fast at Facebook. The company has been known to have new engineers write and deploy new features to the platform on their first day on the job.

Combined with the rate of growth of its user base, this kind of application-development speed requires a level of nimbleness on the part of the social network’s infrastructure team that’s rarely seen in enterprise data centers. Shortening the speed to deploy is a constantly moving target for both software and data center capacity at Facebook.

The company has developed a blocks-of-Legos approach to building out its data centers to shorten the time construction takes, and a lot of the hardware design work that’s now open source through the Open Compute Project has to do with accelerating the time it takes to mount a new server onto a rack.

One of the Facebook data center team’s latest projects had to do with shrinking the time it takes between the point a server is physically installed in a data center and the point it comes online and starts crunching numbers. The project was to switch from an old version of Dynamic Host Configuration Protocol to a new one.

When a new device is connected to a network, DHCP is used to assign it an IP address from a defined range configured for that network. It’s a small but important function in IT infrastructure management that has proven to make a big difference in the time it takes to deploy new servers in Facebook data centers.

The code for the previous version of DHCP, called ISC DHCP (ISC stands for Internet Systems Consortium) has been around for close to 20 years. The new alternative, called Kea, is more appropriate for today’s IT, which, especially in the case of a company like Facebook, bears little resemblance to IT of 20 years ago.

Facebook recently switched from ISC DHCP to Kea in its data centers and as a result has seen major improvements in the speed with which its sys admins are able to spin up new servers, according to a blog post by Angelo Failla, a Facebook production engineer.

The company uses the protocol to install the operating system on new servers and to assign IP addresses to out-of-band management interfaces. These are interfaces sys admins use to monitor and manage servers remotely, whether or not they are switched on or have an OS installed.

The old model was simply too slow for Facebook’s scale and pace of change. Techs working in its enormous data centers constantly add or replace things like network cards or entire servers, and every change could take up to three hours to propagate, slowing down repair times.

ISC DHCP has been one of the biggest reasons it’s been so slow. To add or replace a part, the techs would load a static configuration file into DHCP servers, but the servers would have to restart to pick up the changes. “The DHCP servers were spending more time restarting to pick up the changes than serving actual traffic,” Failla wrote.

With the help of Kea, the new set-up stores things like host allocation and subnets in a central inventory system. When a DCHP server needs to be deployed or changed, the system grabs the configuration information from the inventory. This means there’s no longer a need to generate static configuration files and reload DHCP servers for changes to take effect.

Facebook’s new DHCP application runs in Tupperware, which is the company’s own Linux container technology that’s similar to Google’s Borg, according to Failla.

The old model was also less resilient. The company used to have two redundant DHCP servers per cluster of servers, but if both of them went down, the cluster suffered.

The new approach is a virtual cluster of DHCP servers distributed across the network. They manage a common pool of IP addresses, and any virtual DHCP machine can assign an address to any other device on the network. This way, if local DHCP servers in a cluster fail, the system can recover faster.

With the new stateless design, it takes one or two minutes to propagate changes in the system, Failla wrote, which is more than an incremental improvement from three hours.