Routing Snafu Causes Downtime for Web Hosts

Was your web site down today? You can blame a sysadmin somewhere in the Czech Republic.

Several large web hosting companies experienced outages today when an ISP in the Czech Republic began broadcasting bogus commands, causing routers at other service providers to drop traffic. Media Temple and iWeb were among the web hosting companies that reported customer downtime due to network problems caused by the incident.

The problem involved settings in the Border Gateway Protocol (BGP), which helps direct traffic among the many networks that make up the Internet. Each network is known as an "autonomous system" (AS) and is assigned a number that identifies it to the rest of the Internet. When AS settings are botched, it can affect other providers.

Early today a Czech network with AS 47868 began broadcasting invalid BGP data containing unusually large amounts of data. The problems were quickly noticed by the North American Network Operators Group and Internet Storm Center.

But by then, the problem had caused routers to reboot at Media Temple, disrupting tens of thousands of web sites. "This invalid BGP data exploited a software bug in our routers," the company reported. "We have applied filters to prevent us from receiving this invalid data."

iWeb, a large Canadian web host, said the lengthy BGP transmission "caused our Cisco routers to reset the BGP sessions intermittently when they received the bogus routes." iWeb also solved the problem by filtering the problematic AS.

This isn't the first time that one provider's changes to the BGP table have caused havoc. Last February YouTube was knocked offline for two hours when Pakistan Telecom inadvertantly claimed its IP addresses. In the wake of that incident there was much discussion of the trusting nature of the BGP protocol.  Renesys has also noted  a 2004 incident involving Turkish ISP TTNet and a 2006 event involving Con Edison. In the 2004 event TTNet “pretended to be the entire Internet” on Christmas Eve, while Con Ed assumed routes belonging to New York ISP Panix.