Mesos Powers Data Center Backend for Apple’s Siri

Offering an extremely rare behind-the-scenes glimpse at the backend that powers Apple’s services, the company’s engineers recently revealed that Siri, its natural-language-based interface, runs on thousands of servers managed by a system based on Apache Mesos.

Mesos is a popular open source software for large-scale cluster management. Its most widely known deployment is at Twitter data centers, where the use of Mesos is credited with being one of the things that made Fail Whale, the page that signaled a Twitter outage and used to appear quite frequently, a thing of the past.

Mesos powers JARVIS, the Platform-as-a-Service Apple engineers created to make it easier for developers to add or change Siri’s features and capabilities, Derrick Harris, an analyst and blogger at Mesosphere, wrote on the company’s blog. JARVIS stands for “Just A Rather Intelligent Scheduler.”

Apple engineers talked about their Mesos-based PaaS during a meetup at the company’s Cupertino, California, headquarters earlier this week. The team has been working with the cluster management software for several years, and Siri’s backend is now in its third generation.

JARVIS stores data in HDFS (Hadoop Distributed File System).

Mesosphere’s data center operating system, its core product, is also based on Mesos, and the company is “obviously thrilled about Apple’s public validation of the technology,” Harris wrote. Last year, Mesosphere raised $10.5 million to commercialize Mesos.

Benjamin Hindman, one of the original creators of Mesos, joined Mesosphere last year after a four-year stint at Twitter to lead the design of the product.

Mesos abstracts disparate computing-infrastructure resources, such as CPU, memory, and storage, be they physical or virtual, hosted on-site or in the cloud, and presents a unified view to applications that consume them. Applications like Hadoop, Spark, Kafka, or Elastic Search get presented with an API (application programming interface) for resource management and scheduling across data center and cloud resources, according to the open source project’s website.

Also this week, Mesosphere announced that its data center operating system is now fully integrated with Kubernetes, the open source framework for managing Linux containers built by Google, bringing cluster management and container management closer together.