Data Center Power Outage Brings Down GitHub

GitHub, the most popular online repository for open source code and hosting services, went down for two hours Thursday due to a power outage in its primary data center.

“A brief power disruption at our primary data center caused a cascading failure that impacted several services critical to GitHub.com's operation,” Sam Lambert, GitHub’s director of systems, wrote in a status update on the company’s blog Friday morning. “While we worked to recover service, GitHub.com was unavailable for two hours and six minutes.”

Read more: Why Should Data Center Operators Care about Open Source?

The outage started around 4pm Pacific. The final procedure to fully restore the facility’s power infrastructure was completed Thursday evening.

Utility power outages do not bring down data centers in most cases, since these facilities are designed with UPS units, backup generators, and transfer systems that fail over to the generators automatically. When they do happen, power-related data center outages are caused by failure of those backup systems.

A recent study by the Ponemon Institute, sponsored by Emerson Network Power, found that UPS failure is the most common cause of data center outages, followed by cybercrime (namely DDoS attacks), human error, and cooling system failure, in that order.

Read more: Cybercrime Fastest-Growing Cause of Data Center Outages

It’s unclear where San Francisco-based GitHub’s primary data center is. Lambert told us in an earlier interview that the company does not disclose its data center locations.

GitHub’s Bare Metal Cloud

What we do know is that it has to expand its data center capacity rapidly to keep up with growing popularity of open source software. “We have a massive intake of new repositories and just new data, so we’re continually expanding our storage infrastructure,” Lambert said.

GitHub doesn’t use virtual machines. Instead, it has built a bare-metal cloud, which allows it to provision physical machines the same way cloud VMs are provisioned.

“We deploy onto physical machines,” he said. “We have a system internally that allows us to deploy physical machines as the cloud.”

The Horror!

Effects of the outage were widely felt across startup and enterprise developers around the world, many of whom use GitHub for coding day-to-day. Here is a collection of Tweets from Thursday that illustrate the carnage: