The Data Center Operating System

Can the Internet-scale processing power of a huge computer cluster be harnessed by small development teams to build web apps that can support millions of users - all over the course of a long weekend? That's the goal of the Reliable Adaptive Distributed Systems Laboratory (RAD Lab) at Cal-Berkeley, a testbed for bringing the power of the Internet giants to the masses. The initiative made some headlines in 2006 when it gained sponsorship from a bevy of tech titans, including Google, Sun and Microsoft. RAD Labs is back in the news after Lee Gomes wrote about it recently in the Wall Street Journal. Todd Hoff has followed with a good explainer at High Scalability.

"Although large-scale Internet services such as eBay and Google Maps have revolutionized the Web, today it takes a large organization with tremendous resources to turn a prototype or idea into a robust distributed service that can be relied on by millions," RAD Lab says in describing its mission. "Our vision is to enable one person to invent and run the next revolutionary IT service."

You can read more about RAD Lab's technical vision (PDF) and check out some of its working projects on its web site. Its approach incorporates statistical machine learning (SML) and recent insights in distributed systems in tying together technologies including the Ruby on Rails programming language of the datacenter, the Chubby and MapReduce libraries, and storage in services like BigTable, Google File System, and Amazon's Simple Storage Service. Some reaction from Todd Hoff:

Especially coming from Berkeley, which has a legacy of developing significant Internet technologies. Any software and applications emerging from the RAD Lab will be made freely and openly available to the public, with source code distributed using the Berkeley Software Distribution (BSD) license.