Subscribe to the Data Center Knowledge Newsletter
Get analysis and expert insight on the latest in data center business and technology delivered to your inbox daily.
How Dropbox Stores Stuff for 200 Million Users
How does Dropbox store files for those 200 million users, who store 1 billion files every 24 hours? The company uses more than uses more than 10,000 physical servers to manage user content, along with Amazon Web Services. Here's an overview from the Velocity conference.
Rich Miller
October 23, 2013
3 Min Read
Along with expanding enterprise features, Dropbox is working on deeper integration with Microsoft’s Office 365 service, which offers Microsoft Office software as a service. (Image: Dropbox)
Andrew Fong, Site Reliability Engineer at Dropbox, discussed the company's infrastructure last week at Velocity 2013 in New York. (Photo by Colleen Miller.)
NEW YORK - It took the online storage service Dropbox about four years to accumulate its first 100 million users. It took just 10 months to add the next 100 million.
"The growth has been really immense, especially recently," said Andrew Fong, a Site Reliability Engineer at Dropbox. Fong and other members of the Dropbox team described the company's infrastructure growth last week during a presentation at the O'Reilly Velocity 2013 conference.
The powerful growth of Dropbox has been driven by its ability to offer cloud file storage that can be easily synched between multiple devices, including desktop and mobile devices. Files stores in Dropbox can be easily shared, and are accessible through a web interface or mobile apps, with support for Windows, Mac, Linux, iPhone, iPad, Android and BlackBerry.
How does Dropbox store files for those 200 million users, who store 1 billion files every 24 hours? The company uses more than uses more than 10,000 physical servers to manage user content, along with Amazon Web Services. User metadata is stored in the company's data centers, while the actual files reside on Amazon's S3 storage service. Dropbox also uses Amazon EC2 instances to help the data centers "talk" to its cloud storage.
"We think we have a formula that works for this hybrid architecture," said Fong. "I believe that hybrid infrastructures are what you're going to see going forward for large infrastructures."
Changing Architecture
It's always difficult to engineer for growth when you're not sure what that growth curve will look like. But Fong urged web architects to assume that the earlier you confront the challenges of a hybrid approach, the better off you'll be.
"Don't think it's easy to combine systems later," he said. "It's better to do the work up front. There's a huge difference between dealing with Amazon and dealing with data centers. With a data center, we actually have to care about the underlying hardware, and we have a lot of it. We would have built differently if they'd understood that up front."
Working with data centers and Amazon requires two completely different tool sets, Fong said. But Dropbox is taking steps to simplify across the two platforms. Rather than "baking" a standard virtual machine image for cross-platform use, the Dropbox team is using the Puppet configuration tool to unify its approach to installations, inventories and lifecycles.
What are the tough challenges for Dropbox? Ziga Mahkovec, a member of the Dropbox engineering team, identified several areas:
Serving Photo Thumbnails - Dropbox needed a more efficient "pipeline" to convert newly uploaded photos into thumbnails. By batching http requests, Mahkovec said, the Dropbox team was able to improve performance by 40 percent.
Desktop Client Throughput - This problem was addressed by uploading file chunks in parallel using multiple connections, leading to a 2.5x speedup across Dropbox, and a 4X improvement in Japan.
Storage Latency on Amazon S3 - "The problem with using a service like S3 is that we don't have much control over performance," said Mahkovec. "It's a black box to us. So we have to do good monitoring." In this case, Dropbox pooled its requests to avoid making new connections.
Mahkovec said the answers to these type of problems are not always intuitive.
"When running at scale, what might seem the obvious solution doesn't necessarily apply," he said. "When you can't control all the variables, we run a lot of experiments and iterate."
About the Author
You May Also Like
