Why Cloudera and MapR Opted Out of Pivotal’s Hadoop Consortium

When it announced it would open source its entire suite of big data analytics tools earlier this week, Pivotal also announced formation of an industry group that would bring what the EMC- and VMware-owned software company said was a much needed common set of standards to ensure compatibility across technologies in the Apache Hadoop ecosystem.

Founding members of the group, called Open Data Platform, include enterprise Hadoop heavyweight Hortonworks (which now has an expansive partnership with Pivotal), GE, which owns a 10-percent stake in Pivotal, as well as EMC and VMware. The list also includes numerous non-affiliated heavyweights, such as IBM, Verizon, CenturyLink, Capgemini, Teradata, and a couple of others.

Glaringly, two of Hortonworks’ biggest competitors in the enterprise Hadoop market – Intel-backed Cloudera and MapR, which last year closed a $110-million funding round led by Google Capital – were not part of the ODP announcement at all. Sundeep Madra, who leads Pivotal’s Data Products Group, told DCK he’d love for them to join. But top execs at both companies told us they don’t want anything to do with ODP. They said the initiative seemed self-serving, and its stated goal of creating a “common core” for Hadoop was redundant, since a framework for interoperability already existed within the Apache community itself.

“We haven’t really seen interoperability across projects as a major issue at all,” Jack Norris, chief marketing officer at MapR, said. “We think the market, and especially the Apache Software Foundation, is working fine. That’s why it seems like [ODP] is more about a partner program than a community initiative.”

MapR management were approached about joining the ODP but declined. “We had the chance to participate and basically decided not to.”

Mike Olson, a Cloudera co-founder and the company’s chief strategy officer, said ODP was a “marketing ploy,” and a “mistake.” He agreed that Apache ensured interoperability in the Hadoop ecosystem aptly. “The APIs and services of HDFS [Hadoop Distributed File System] and MapReduce have been carved in stone for years,” he wrote in an email. “There’s no incompatibility among those APIs from any vendor. Zero.”

Cloudera management, Olson said, learned about the pending ODP announcement about one week ago and approached Pivotal to learn more. Once they did, they decided not to join. “Pivotal and Hortonworks claim that the ODP is driven by an industry-wide longing for standardization in the Apache Hadoop ecosystem,” he wrote in a blog post explaining his company’s position. “I don’t believe them.”

Cloudera has about 1,450 companies in its partner ecosystem, and its management has not heard from that ecosystem that there were companies that were confused about building applications on core Hadoop, Olson wrote.

His biggest objection to ODP was that it seemed like a club for companies with deep pockets that’s very different from the Apache community of developers that drive Hadoop forward. “Developers in Hadoop already collaborate to design and implement standards, using tried, tested, successful open source collaboration tools,” he wrote.

Notably, one of the announcements Pivotal made this week was that its analytics tools would soon support Hortonworks’ Hadoop distribution. Pivotal has its own distribution and said it would continue to support it along with Hortonworks’.