Data Center Networks are Getting Faster and Smarter

Few components of the data center have seen more innovation than the network. Besides the obvious fact that it interconnects everything inside and therefore performs an outsize role compared to other pieces of the puzzle, it’s also the layer that’s expected to progress longer than others.

Facebook’s infrastructure engineers, for example, decided to pay more attention to the network than anything else five years ago. It was already clear in 2013 that processors wouldn’t keep up with the Moore’s Law curve, Jason Taylor, the company’s VP of infrastructure, told us in an interview earlier this month. Growth in data density of storage media, including Flash, was also predictably finite. Things were different in the world of networking technology.

“In general, the other components were doing OK, but they weren’t doing as well as network,” Taylor said. “Network was just incredible.” Data center networks have gone from 1 Gigabit Ethernet links out of a server rack just under a decade ago to 100GbE today and 400GbE already around the corner.

“That’s a tremendous increase in the capacity of network, far outstripping the densification of any other component in the data center,” he continued. “And we see that network resource being something that will continue to grow and scale very well for approximately fixed cost. We still see probably a solid 10 years of great scaling out of the network.”

DCK spent a part of August looking at some of the trends in data center network technology, examining in addition to speed the way some networking innovation by operators of hyperscale cloud platforms like Facebook has been seeping into the enterprise data center market.

At Facebook, which has been more open than other hyperscalers about its data center technologies, 100 Gigabit Ethernet in the data center is now commonplace, and the company’s engineers are already preparing for a 400GbE future.

Enterprise data center networks aren’t far behind. As our contributor Mary Branscombe reported, the transition to 100GbE this year went “from a steady march to a jog.” By August 2018, 100GbE port shipments were already double what they were in 2017, according to Dell’Oro Group. Vendors have observed a similar trend.

A lot of the network capacity progress is driven by scale, and scale drives the need to automate – another area where hyperscale platforms have been trailblazing. Some of that innovation, through vendors and open source communities, is now finding home in enterprise data centers.

In another report, Branscombe highlighted the growing amount of attention enterprise operators now pay to automating network management and being able to apply the DevOps principles to their work, so they can better support the teams of developers that write software that runs on top of their infrastructure.

Looking to take network automation even further, incumbent vendors and a handful of startups are actively working on technologies that support the concept of Intent-Based Networking, or IBN – the idea that networks could tune themselves based on policies set by the administrators, removing the need for all the manual network management processes used today.

In a deep dive, our contributor Wylie Wong explored IBN strategies of the two largest data center network vendors – Cisco and Juniper – that are just starting to take shape. He also took a look at the products and strategies of four of the leading startups developing IBN solutions for data center networks.

Some of these companies have developed features that enable users to simulate their networks and apply changes virtually to test them before rolling them out to production. Microsoft, which operates one of the world’s largest hyperscale cloud platforms, has been doing something similar for some time now. The company recently said it would open source the technology behind this capability.

Even if Facebook’s projection that network performance will continue to improve by leaps and bounds while cost remains steady ends up being true, there’s another kind of cost to consider besides dollars and cents.

Having faster, smarter, more automated networks means putting more and more processing power in switches and routers. Power consumption by network equipment continues to go up, which Taylor expects will become a challenge within several years.

“But that seems like an engineering challenge that we’ll be happy to take on,” he said. “I expect in the next five years we’ll start to see more challenges around really just getting the optics close enough to the network in order to minimize power.”

One idea with a lot of potential is integrating optical components directly into processor packages. Facebook is a big supporter of the idea. A group of vendors, including Microsoft, Cisco, Juniper, and Arista, recently created the Consortium for On-Board Optics to set standards in this developing space.

“This is something people have talked about for a long, long time: when do you see optics to your processors or chips,” Taylor said. “It’s the first time I’ve seen a future where – maybe in the next seven years – we could actually have optics” directly inside chip packages. “That was research just 10 years ago.”

Explore our series on the latest in data center networking technology here:

The Year of 100GbE in Data Center Networks - Here’s what’s driving the current step change in evolution of data center networks.

How DevOps Gained a Foothold in the Data Center Network - Expert silos and manual methods are on their way out, as scale and software-defined networking demand ever more team integration and automation.

Intent-Based Networking in the Data Center: Cisco vs. Juniper - How the two networking technology giants’ visions for self-driving networks differ from each other.

4 Intent-Based Networking Startups Innovating to Disrupt the Data Center Network - These self-driving network startups are out to eat Cisco and Juniper's lunch.

Microsoft to Open-Source Its Secret Weapon Against Cloud Network Outages - The Open Network Emulator simulates the entire network that powers the company’s hyperscale cloud platform. Microsoft has been using it for about a year to test changes made to the network before they’re deployed in production.

Machine Learning Is Becoming a Must in Data Center Network Security - The volume of data traveling on networks and sophistication of attack tools are outpacing human experts’ capabilities.