Mesosphere Turns Data Center into One Huge Computer

Mesosphere stands to make a major impact in the data center. The company formally launched its Data Center Operating System (DCOS) into early access earlier this week and plans public launch in the first half of 2015. It also recently raised more than $36 million in a Series B funding round, which the company will use to build out its engineering and sales teams even further.

Remember all of that cool software on computers before you got Windows? Remember how that whole world opened up to you even more once you had a visual operating system, and how Windows ushered in the creation of other very cool software? Mesosphere stands to do the same with data centers. It is accomplishing this task in an age where the data center is becoming increasingly distributed and virtualized -- the age of web scale.

DCOS helps with better data center resource utilization and better fault tolerance. Mesosphere abstracts the complexity and handles the overall orchestration, helps prevent failures, ensure failover, and respond to demand surges. There’s even a tool called Chaos for simulating failure, modeled after Netflix's Chaos Monkey.

The data center OS provides an API and software development kit that lets programmers develop for a data center like it’s one big computer.

“You can write new applications and fire off tasks into your cluster by specifying how many resources each task should occupy," Mesosphere CEO Florian Leibert said. "You don’t need to wire different machines, just use the Mesosphere fabric.”

Mesosphere does a lot of complex things but makes them look really easy. For example, Chronos is a distributed and fault-tolerant job scheduler that supports complex job topologies. The tool is normally used by sophisticated engineers, but Mesosphere makes it dead simple to install it on a Mesosphere cluster and use it across data centers.

VMware recently integrated Mesosphere with VMware vSphere to help run applications and services at scale. "Mesosphere will have a positive impact on the data center," Kit Colbert, VMware's vice president and CTO for cloud-native apps, said via email. "As applications become more distributed, their scale and complexity will increase."

Colbert said non-web-scale customers can reap the same benefits of Mesosphere web-scale companies like Twitter and Airbnb have. "The challenge is how these technologies can be implemented in the data center so they meet all enterprise requirements around security, compliance, SLA management, and more."

Khosla: a Perfect Philosophical Fit

New investor Khosla Ventures led the recent round with additional investments from Andreessen Horowitz, Fuel Capital, SV Angel, and others. The company raised a $10.5 million series A in June, and its total funding is now approximately $50 million.

In June, venture capitalist Vinod Khosla said at GigaOm’s Structure conference in San Francisco that the most important opportunity in the business of IT was getting rid of all the IT people. The opportunity is in building a data center OS that will automate resource management much like a computer OS does.

He made the comments directly after Mesosphere had raised its Series A, which was led by Andreessen Horowitz. Now, it comes as no surprise that Khosla Ventures would take a more active role in shaping Mesosphere’s future.

“The industry needs a new type of operating system to optimize and automate the complex landscape inherent to the agile IT era: a growing fleet of distributed web, mobile, analytic server applications, operated as application-centric abstractions on commodity server and storage pools in dedicated data centers and public clouds,” Khosla said in a statement this week.

Foundation of a Data Center

DCOS is built on Apache Mesos, the tool famous for helping Twitter get a handle on data center operations and killing the “fail whale.” Mesos abstracts CPU, memory, storage, and other compute resources, creating virtual pools.

The work Mesosphere has done around Mesos is significant. Its data center OS isn't just a commercial version of Mesos, but a wider platform and ultimately operating system that uses Mesos as a cornerstone.

“When it comes to commercial pieces, Mesosphere adds modules and extensions to the open source that are really relevant to large deployments,” said Leibert. "Most of the things we’ve developed that are all parts of the bigger picture are free and open source and widely used, while some are not -- like allowing you to start up a service and makes sure that it fails over and scales up.”

A user interface abstracts these complex functions and makes sure DCOS is not only powerful, but intuitive and pretty. There’s also a command line interface if a user wants to bypass the visual interface.

The company recently acquired a design studio in New York. “Design is important. It’s a challenging task,” said Leibert. “We're coming from Airbnb, where we had a lot of design resources in terms of public-facing, but internally we also took design seriously. Sophisticated engineering was previously done with unsophisticated design tools, and we wanted to change that.”

Roots at Twitter and Airbnb

Connection between the founders goes back some years, said Leibert. Co-founder Benjamin Hindman was co-creator of Mesos and former Twitter lead engineer. While he was creating Mesos, his family was Leibert's host family while he was an exchange student. Leibert was involved in helping both Twitter and Airbnb scale. Another co-founder, Tobi Knaup, is also a longtime friend and previous tech lead at Airbnb.

All three led very interconnected and parallel lives, and like many other startups, saw what was bugging them and fixed it. "I can count back to the Airbnb days; in the middle of night I’d get a call and have to go in," said Leibert. "This automates the solutions to those problems."

Beyond Web Scale Market

Mesosphere is often only spoken of in the context of the largest web-scale companies, but its market is wider. "If you are starting a new company, you should build atop of Mesosphere day one," said Leibert. "Our system is built for the enterprise as well."

Its wide appeal prompted developer-focused cloud provider DigitalOcean to partner and offer Mesosphere. DigitalOcean has grown immensely due to its developer-friendly take on cloud.

“It’s the new and better way to sort of extract DevOps work," Mitch Wainer, DigitalOcean co-founder and chief marketing officer, recently commented on Mesosphere. "To organize and structure your large-scale infrastructure environments.”

What’s in DCOS

The DCOS consists of a distributed systems kernel with enterprise-grade security, based on Mesos. It includes a set of core system services, including a distributed init system (Marathon), distributed cron (Chronos), service discovery (DNS), storage (HDFS), and others, all capable of launching containers at scale.

The company’s growing team has worked to extend the libraries, platforms, cloud hosts, and Linux distributions supported by DCOS, as well as features around security, cost-accounting, alerting, and other core enterprise features.

Apache Spark, Apache Cassandra and Google’s Kubernetes are all natively supported. Kubernetes helps manage the deployment of Docker workloads. It supports all modern versions of Linux and runs on-premise, on bare-metal servers, or in a virtualized private cloud, such as VMware or OpenStack.