The Network Gear Economy Is Becoming Commoditized

People attending Amazon Web Services’ re:Invent conference in Las Vegas finish their dinners early so they can return to the main stage, just to hear the company’s celebrated VP and distinguished engineer, James Hamilton, spew ideas in what would be a stream-of-consciousness monologue were it not for the presence of gently guiding slides.  During a late evening keynote session on Tuesday, Hamilton introduced attendees to his company’s custom network routers, explaining that no single vendor of vertically-integrated network gear is capable of responding to architecture-centered network problems within a timeframe any narrower than six months.

The solution, he explained, was for Amazon engineers to collaborate with multiple, competing manufacturers of network gear along a loosely coupled stack of technologies.  Those technologies would enable Amazon to produce simpler, more versatile custom routers adhering to standards of its own — even the less universally accepted ones, like 25 Gbps Ethernet instead of 10-gig or 40-gig.

Attendees leaving Hamilton’s talk early might have come away thinking he was advising them to go build global retailing empires of their own, just to give them the leverage they needed to conduct negotiations with the Broadcoms and Marvells of the world.

But just then, Hamilton suggested something potentially profound:  The ASICs that Amazon ordered to power its custom routers are becoming cheap.  Their cheapness enables them to be ordered in bulk.  And with those competing network stack vendors’ products becoming more open, perhaps one need not be an Amazon to build a more moderate data center using custom networking gear.

“Vertically-integrated networking equipment, where the ASICs, the hardware, these protocol stacks [were] supplied by single companies, is [like] the way the mainframe used to dominate servers,” said Hamilton.

“If you look at where the networking world is, it’s sort of where the server world was 20 or 30 years ago.  It started out with, you buy a mainframe. . . and that’s it.  And it comes from all one company.  The networking world is the same place.  And we know what happened in the server world:  As soon as you chop up these vertical stacks, you’ve got companies focused on every layer, and they’re innovating together, and they’re all competing.  You can get great things happening.”

For any given server size, Hamilton explained, the overhead of networking required to service that node is rising.  That wasn’t the case, he said, back when network equipment was artificially expensive.  In a more commodity-driven economy, this no longer can be the case.

Amazon’s custom-built routers are built with the assistance of the company’s own protocol development team, he said.  This fact is responsible for what Hamilton characterized as a steep reduction in costs.  When a vendor is bound to supporting the protocol needs of a vast plurality of customers, he argued, the complexity of the resulting product renders its reliability, in practice, unmaintainable.

“And the next release comes out, and they don’t test all that stuff that people like me ask for, because nobody uses it anyway,” the AWS VP continued.  “And it doesn’t work!  It just doesn’t work!”

By comparison, Amazon’s protocols only have to serve Amazon’s requirements.  It’s the flip side of the argument in favor of standards, which in the older economy were vitally necessary to ensure interoperability.  In certain facets of Amazon’s network, standards are absolutely not required at all.  In fact, Hamilton argued, the demands on standard protocols are so high that they end up working against customers with Amazon’s scale.

Later, Hamilton showed off one of AWS’ ASICs of choice: a custom-built Broadcom Tomahawk.  Because the vertical networking stack is breaking loose from the ties that bind, he said, custom ASICs are becoming a market.  And that means costs are coming down — for everyone.

“These are absolute monsters!” he exclaimed.  “This is a 3.2-terabit part.  What does that mean?  It’s 128 ports of 25-gig.  All ports can be running flat out, with no blocking.  It’ll flow at 3.2 terabits at the same time. . . Why do I like that?  Non-blocking is a wonderful place to be, but the real reason I like it is, there’s a healthy ecosystem.  Cavium, Mellanox, Broadcom, Marvell, Barefoot, and Innovium are all building parts.  There’s 6-terabit and 13-terabit parts coming.  And they will be around the same price, in just the same way the server was.”

Hamilton described an environment for networking gear whose situation was similar to the bright, beautiful world that Intel brought forth, when Moore’s Law was in its heyday.  (I would call it a new stack, but that phrase has been taken.)