CloudFlare ‘Let the Internet Down,’ CEO Says

Gerrit De Vynck (Bloomberg) -- CloudFlare Inc., an internet service meant to protect websites from going down, faced its own network issues Tuesday, leading to several prominent sites -- like blogging platform Medium and video game chat provider Discord -- being unavailable for some time.

A bug in CloudFlare’s software caused one part of its network to suck up computing resources from the rest of the company, leading thousands of websites around the world that rely on CloudFlare to go offline for as long as 30 minutes, Chief Executive Officer Matthew Prince said in an interview. The problem wasn’t because of an outside attack, as was earlier speculated, Prince said.

“This was entirely our mistake,” Prince said. “We let the internet down today.”

The episode underscores how many services have come to rely on CloudFlare. The system works by effectively acting as a buffer between a website and the end user, making sure to block attacks that could bring a site down by overloading it. A bug in CloudFlare’s firewall made its own software think it was under attack. It pulled computing power from other company products to shore up defenses, as it was programmed to do. But the system took so many resources it starved other CloudFlare products that help other companies deliver their web pages to people around the world, hence the outages.

“We misdiagnosed this initially as an attack and that wasted precious minutes,” Prince said. Once they realized what the problem was, they sequestered the software bug and the system quickly returned to normal.

Websites in Europe and eastern North America were hit the worst because it was day time and more people were online. Other websites that were affected include small business commerce platform Shopify Inc., music service SoundCloud Ltd., Bitcoin trading platform Coinbase Inc. and online storage site Dropbox Inc.

Even the service that tracks online outages, Down Detector, crashed.