Why Merchant Silicon Is Taking Over the Data Center Network Market

Merchant silicon is burrowing its way deeper into data center networks. It will be in 63 percent of all Ethernet switches that ship in 2022, a jump from the 56 percent last year, by IHS Markit analysts’ estimate.

“It’s going along with the trends we’ve seen in the market in terms of adoption by many enterprises, as well as telco and other cloud service providers, whether it be hyperscalers or tier-twos,” say Devan Adams, IHS principal analyst for cloud and data center switching.

“When you have a vendor like Cisco start to accept merchant silicon being used within even some of their up-and-coming switches – like the recent announcement they made releasing their 400G switches, which are very hyped – that’s big.”

Adams says that “everyone’s looking to try to introduce 400G.” After Juniper and Arista made their 400G switch announcements, many expected Cisco to follow with models based on its own proprietary silicon, “which was the case, but only half the case. They made the announcement that they’re also going to offer the merchant silicon version.”

It’s a big win for customers, he says, pointing out an announcement that Cisco has Innovium as a third-party vendor. “That’s interesting because they’re very new; they’re really a small silicon vendor.”

Related:To Get to Six Data Centers Per Region, Facebook Rethinks Its Network (Again)

Homogenizing the Data Center

While Broadcom is the undisputed leader in merchant silicon, there is a trend of increased adoption from white-box and traditional switch vendors that used to primarily deploy their own custom silicon, according to IHS’s Data Center Network Equipment Market Tracker report, which predicts proprietary or custom silicon will top 25 percent of all units shipped in 2023. The report also says that programmable silicon will account for 12 percent. This compares to 38 percent proprietary and 6 percent programmable for 2018.

“Proprietary or custom silicon vendors are mainly traditional switch vendors, but many are expanding their portfolios to include non-proprietary third-party commodity silicon,” Adams says.

Juniper is one example, with its QFX1003 switch, which will use its proprietary ZX ASICs and its QFX5220 switch, based on Broadcom merchant silicon. Both will offer 400GE speeds.

“Virtually everybody’s moving towards merchant silicon and the scale-out model,” says Mike Bushong, VP of enterprise and cloud marketing for Juniper. “Adoption in the data center is going to be aggressively on the merchant-silicon side. There’s going to be a place in the data center for custom silicon, although I’d say that in terms of buying behavior, I fully expect merchant silicon to dominate.”

Related:Cisco Embraces Disaggregation of the Data Center Network

The first benefit end users will see is economic, Bushong says, and it’s the main reason they care about the move to merchant silicon. “But it’s not the only benefit they’ll see.” There’s a very real agility angle to merchant silicon. “In data centers, if I can reduce diversity, if I can make everything look the same, then I can become more efficient. I can become faster,” he says.

Part of what merchant silicon does is provide operational uniformity even with different vendors, Bushong says, meaning whether you’re buying from Juniper or from someone else, you leverage some of the same underlying characteristics.

The question people doing procurement should ask is whether they are making decisions that make their operations more uniform, he says.

“Who’s in the room when you evaluate?” Bushong says. “If end users view merchant silicon as only about what the box looks like and how they’re buying it – if they fail to make the connection to how they are going to manage it and what is the operational side – then they’re not getting the full benefit of merchant silicon. They’re making an uninformed decision and only looking at part of the problem space.”

What’s stalling things now is consideration of architectural changes, he says.

“Are the protocols different?” Bushong says. “The hardware’s largely made that transition out. What some of the other companies are trying to do now is say that if that transition is there, then is there a monopoly that can be broken up? This is what Barefoot [Networks] is trying to do. If you can break those pieces apart, they think they can provide greater economic leverage. That’s kind of the next frontier.”

AT&T’s Programmable Network

AT&T in 2016 was the first telecommunications provider to announce it was using programmable switches from Barefoot in its network. It installed Tofino-based white boxes running SnapRoute’s FlexSwitch network operating system in parts of its existing MPLS-based networks. AT&T then used data-plane programmability using a language called P4, which Barefoot says has since been adopted by many large data center operators. The telco not only applied merchant silicon to their core MPLS network, but also used it for in-band network telemetry (INT) on all traffic going from a link between San Francisco and Washington, D.C.

It was significant that the INT concept was used on such a fundamental part of AT&T’s network, says Ed Doe, Barefoot’s chief business officer, “and one where they wanted to be able to measure, with very fine control, exactly what were the latency utilization, the SLAs – service level agreements – that they were able to maintain.”

Tencent’s Mixing and Matching

Doe says the concept of telemetry has gone into many hyperscale data center customers. As recently as late last year, other hyperscale customers started to go on the record about using technology from Barefoot and applying the telemetry concept to it.

One of those hyperscalers was China’s Tencent. The gaming and social media company is using Cisco switches that contain Barefoot silicon to advance telemetry in the core of its standard cloud offering. Barefoot’s Doe says customers like Tencent are paving the way for more mainstream enterprises, “especially if they can take advantage of a typical switch from a company like Cisco that harnesses the Barefoot technology in P4 and telemetry.”

Tencent used not only Cisco’s NX operating system (which is the standard operating system on these switches), but it also incorporated SONiC (Software for open networking in the cloud), an open-source network OS developed by Microsoft.

“That was unique in that Cisco worked with Tencent to bring that to bear onto these same switches powered by the Barefoot Tofino tech and using the P4 programming language,” Doe says.

Incumbents Come Onboard

Customers are benefiting because new entrants into the merchant space, like Barefoot, are giving people not just an alternative but also the benefit of new technology. “This new tech is being brought to them or packaged and made easily accessible by companies like Arista and Cisco, which are customers of ours, and providing these solutions to their customers, which are usually different-sized data center companies,” Doe says.

Until vendors like Arista and Cisco came on board with merchant silicon, end users had to be more technologically advanced to take advantage of white boxes, often writing their own software to recreate what the hyperscalers were doing. “But now, with those two – and in the near future many more – you’re going to have many options … no matter what the size is,” he says.

Why Stordis Chose Barefoot Over Broadcom

One company working closely with Barefoot is Stordis, which announced its own switches at the Open Compute Project (OCP) global summit in April. Having started as a niche distribution company in the storage space, Stordis is repositioning itself as a networking-focused business. The German company started doing open networking several years ago.

“We got interested in having a different switch ASIC than the usual ones, and we always liked the concept of programmability that Barefoot offered,” Stordis CEO Alexander Jeffries says. “We were selling all these bare-metal white tops, just with a Broadcom base, and then we also started selling quite a few of the Barefoot switches.”

Those were essentially the same switches as Broadcom’s but with Tofino chips.

“Since we were doing quite a lot of work in the OpenFlow space, we actually knew that customers needed specific requirements to be able to use two switches properly,” Jeffries says, adding that existing switches didn’t have enough compute power, memory, RAM, or storage space. “We also saw there was not a single switch with time synchronization and time stamping, so we added all these features and got these new switches built. There was basically nothing else available with this kind of feature set [and] making use of the Barefoot Tofino.”

The Stordis switch is designed like a bare metal switch, Jeffries says, “so you have commercial-grade software like Kaloom, but we will also offer open-source software like ONL [Open Network Linux].”

Stordis is signed up to the ONF Stratum project, as well as OCP on the hardware side. “We’re also looking at supporting things like Redfish and OpenBMC for the management of the switches to give the statistics of the switch, like fan speeds and temperature,” the CEO says. “We’re trying to support as much open source as possible with these units.”

While most other switches have dual- or quad-core CPUs, Stordis puts an 8-core CPU into the switch for extra processing power. The switch has 32GB of memory and a 128GB SSD.

From a hardware perspective, much of the box’s functionality would have been possible with a Broadcom switch, Jeffries says. The reason Stordis went with Barefoot is that it’s programmable, enabling the chipsets to perform a certain feature or functionality. A Cisco switch comes with an abundance of software and functionality, which frequently is overkill. The beauty of Tofino is that you can program the switch to perform only one certain functionality.

“The kind of customers we’re talking to for these kinds of switches are quite a bit different to what we were talking to in the past,” Jeffries says. “We’re getting a lot of interest from service providers – so telecommunication companies – the security environments, lots of interest in academic research, [interest] from all kinds of use cases.”

In-band telemetry is a huge plus, Jeffries says, “so when you have traffic queues and congestion in your network, this kind of monitoring you can do very well with P4.” He says Stordis engineers have many ideas for new switches and new switch models, “so it will be interesting to see how quickly we grow and where we are a year or two from now.”

Stordis customers are talking about what they need to put in place if they’re no longer going with Juniper or Cisco, but they also don’t want to use open networking gear, Jeffries says.

“What does it actually mean in terms of resources, training people, perhaps having to develop your own code?” he says. “How market-ready is the open-source option? Many of the open-source things look very, very interesting, but it probably will take another year, or two, or three, to get really market-ready and stable. It’s a real challenge to get these open-source things going because at the moment it’s more like proof-of-concept or basic foundation for many of these open-source initiatives, and one needs to take what’s there and actually get it into production-grade software that you actually can deploy in a mission-critical environment. This is the real challenge, because not everyone’s got the resources and funds like the Facebooks and Amazons and Microsoft Azures. [The hyperscalers] can go out and employ whatever they need to at whatever cost, but that’s not as easy with other kinds of organizations, and that’s going to be the challenge for the next two to three years.”

But if it does goes there, Jeffries says, “you could see the networking industry – or these open-source options – becoming like a Red Hat to networking, where you have your open-source switch and some kind of enterprise license, which offers support as well. You could see something similar to that happen in networking and I think that’s not too unrealistic.”