Twitter Using BitTorrent to Speed Servers

Twitter has begun usin file distribution technology from BitTorrent to boost the performance of its back-end infrastructure, which has required constant attention to keep pace with the growth of the popular microblogging service, which has now reached 1 billion tweets per month, according to Pingdom.

"With BitTorrent, Twitter is planning to distribute files faster and more efficiently, saving time and precious resources and improving the scalability of Twitter’s operation," writes Ernesto at TorrentFreak (via Glyn Moody).

Here's the full description: "Twitter’s new project, codenamed ‘Murder’, will not use the bandwidth of Twitter users. Instead, it will transform the site’s servers into a large BitTorrent swarm that will distribute file updates using BitTorrent technology. The setup is pretty straightforward. Murder uses a ’seeder’ server where the new files will be distributed to thousands of ‘peer’ servers. Because all servers assist in the deployment of the files, it will only take a fraction of the time it would otherwise take when files are distributed from a central server."

The TorrentFreak story described the project as being in the development stage. But Twitter staff have since confirmed that it is live, and the Murder code is available on GitHub. "Murder is already in production and has been for several months," Twitter engineer Ryan King shared in our comments. "We use it to deploy all our code."

Murder "allows for scaleable and fast deploys in environments of hundreds to tens of thousands of servers where centralized distribution systems wouldn't otherwise function," according to the GitHub description from Twitter's Larry Gadea. It's based on the BitTornado client, and requires one server to act as a tracker, and another to seed the files to the servers that will share them, distributing the load across tens of thousands of servers.  

BitTorrent Inc., the company formed to commercialize the open source peer-to-peer software, expressed interest in helping other large Internet platforms explore the potential for BitTorrent to boost back-end file transfer.

Read more at TorrentFreak. For additional background reading, here's some of our previous reporting on the the evolution of Twitter's infrastructure: