AT&T Says the Edge Isn’t Where You Think It Is

 -- Updated with clarifications about infrastructure for the Plano project and additional comments by AT&T's Vishy Gopalakrishnan

It was announced in February at Mobile World Congress in Barcelona and later, in May, at Microsoft Build 2019 in Seattle: an airborne drone sensor system linked to a traffic management and incident response platform called VigilAir using Azure Stack along with AT&T’s infrastructure. AT&T calls this model “Network Edge Compute” (NEC, no relation to Nippon Electric), and recently touted the model’s ability to boost performance and reduce latency by moving compute resources to the edge of the network. (The AT&T marketing materials for the project imply that it’s core goal is to apply 5G and edge computing to improve VigilAir’s performance, however, after this article was first published, AT&T spokespeople said that the test does not use 5G, relying instead on 4G LTE wireless infrastructure. A February blog post on the company’s website is titled “AT&T and Microsoft Test Network Edge Compute to Enhance 5G for Business,” and a June post about the same project is titled “For 5G and Edge, the Sky’s the Limit.”)

In an exclusive interview with Data Center Knowledge, Jeff Shafer, AT&T’s assistant VP of product management for edge solutions, revealed one detail that edge computing specialists may not have seen coming: The network edge that AT&T has in mind today is not, as he put it, “a little micro data center at the base of every cell tower.”

“There’s reasons why that doesn’t work,” he said. “It sounds good in theory, but in practice, it doesn’t work.”

The experiment is collaborative, produced by the AT&T Foundry in Plano, Texas, along with Microsoft, and Tel Aviv-based situational awareness startup Vorpal. The startup has already deployed its VigilAir portable sensor network for drone activity detection, designed for metropolitan areas and airports but also temporary venues, such as sports stadiums and public events.

Shafer described the Plano experiment as a precursor for a network edge deployment, or “a way to simulate that type of an environment so that we could test a real-world application.” Prior to experimenting with edge deployments, Shafer said, Vorpal would have to transport entire server racks to remote stations, including airport perimeters.

“They weren’t doing anything with edge, really,” Vishy Gopalakrishnan, AT&T’s VP of ecosystems and innovation, told us. “If they had to go monitor a space, they would go to the rooftop, and they’d have a bunch of souped-up servers that they carried along everywhere they’d go. And they’d use it as the physical site where they’d do the ID and detection.”

The goal AT&T had for the project, Gopalakrishnan told us, was to offload the computing capacity of those remote servers to “somewhere within our network, so you don’t carry around a two-ton server infrastructure everywhere you want to do detection and identification of drones.”

Initial tests showed a 40 to 50 percent decrease in system latency as a result of deploying through NEC over regular cloud servers, Vorpal founder and CEO Nir Raz said in a emailed statement.

‘That’s Not How the Mobility Network Works’

Moving computing resources into the AT&T NEC enables VigilAir to be more effectively deployed, especially as clients will require higher quality-of-service levels, Raz said. But where would this NEC be, once the 5G edition of VigilAir comes online?

According to AT&T’s Shafer, with 4G/LTE today, this location would be one or more of the facilities the company calls National Technology Centers (NTC). As company literature provided to investors describes them, NTCs are data centers that “host data and internet service for our wireless customers.” But with 5G, AT&T is virtualizing the mobile core in AT&T’s Network Cloud, allowing for more distributed access to NEC at AT&T network facilities in metros across the U.S.

“Somebody who doesn’t necessarily know how the mobility network works may think, ‘Oh, if I put a mini data center at the base of the tower, then I can just siphon off the traffic there and process it locally there,’” Shafer said. “That isn’t how the mobility network works.”

At the edge of a cellular access network – which AT&T now calls a “mobility network,” since 5G may alter the structure of “cells” – is the radio access network (RAN). This comprises the connections between wireless devices and cellular towers or transmitters. Data traffic is encapsulated and backhauled to AT&T’s mobile core, which for 4G/LTE is located in centralized NTC facilities and which an AT&T spokesperson now tells us will be distributed more pervasively in metro-based network facilities for 5G. It is here, Shafer explained, where the handoff between the RAN and the internet takes place — no place else, and certainly not in the immediate vicinity of the tower itself.

“If someone were to just have a data center somewhere in that path between the cell site and the NTC today or to the distributed 5G mobile core in the future, it wouldn’t make any difference, because you’re going to hairpin through the mobile core and it’s going to come back,” he said. “Network topology matters and how the traffic gets routed in these networks matters. Just being geographically close isn’t enough, because depending on where the nodes are in the network and how the traffic is getting routed and getting handed off – maybe to a different network, like mobility-to-internet, as an example — just having a data center nearby isn’t necessarily going to help.  It might, but it may not.”

We asked Gopalakrishnan what parts of the Vorpal demonstration were particularly 5G, as opposed to just 4G LTE or “5G E.”

“What 5G gives you is a tremendous lift in terms of the bandwidth, and tremendous reductions in the latency,” he told Data Center Knowledge, “in addition to ultra-high reliability. What that allows you to do is actually envision the breaking out of the architecture.

“Instead of keeping, for example, the compute part of it at the facility,” he continued, “if we were to deploy it in 50 locations across a city, in a non-5G environment, what would end up happening is that you would actually end up supplying a bunch of compute at 50 locations. What 5G allows you to do, because of ultra-low latency, is actually distribute that compute back closer into the network itself, taking advantage of the fact that now, I have the ability to get some economies of scale, by pushing out my compute away from the edge itself — meaning the physical edge, where I’m doing the drone detection and monitoring. Now, I have just moved a bunch of compute, storage, and all of the processing into, if not one location, multiple locations. But then I keep the footprint that needs to be deployed for any of these drone detection scenarios, down to a minimum.

“You can only do this kind of distribution with 5G because of the latency,” concluded Gopalakrishnan, without clarifying that the project in Plano was not using 5G infrastructure.

Walking Back the Edge

Shafer’s explanation would appear to contradict the theory of edge computing circa 2017, especially among architects and builders of µDCs designed to reside next to cell towers. While AT&T never explicitly endorsed a µDC configuration or design, it never seemed to close itself off to the idea. As early as 2016, the company had given indications that it was planning to utilize an open-source technology called Central Office Re-imagined as a Datacenter (CORD) to equip existing facilities with commodity x86 servers. As its 5G message matured, the company advised it would eventually provide data and internet hosting services for customers at these facilities.

“What 5G gives you is a tremendous lift in terms of the bandwidth and a tremendous reduction in latency, in addition to ultra-high reliability,” AT&T’s Gopalakrishnan said. “What that allows you to do is envision a breaking out of the architecture instead of keeping the compute part of it at the facility.”

Without 5G, if you were to deploy in 50 locations in a city, for example, “you would end up supplying a bunch of computers to every location.” With 5G’s ultra-low latency, much of that compute horsepower can be moved “closer into the network,” away from where drone detection and monitoring take place, to take advantage of infrastructure economies of scale. That enables you to keep the footprint that does need to be deployed where detection and monitoring take place down to a minimum. “You can only do that distribution with 5G, because of the latency.”

Playing the role of the data center in this experiment were servers installed inside the AT&T Foundry facility, company officials said. The demonstrations were not meant to simulate or test the capability of any servers, only the connectivity.

‘Operational and Financial Model, Not Latency’

Edge data center and bare-metal cloud builder Packet was one of the early partners in the Kinetic Edge Alliance, along with Vapor IO. That group has already been building RAN tower-adjacent, or tower-connected, µDCs that Packet co-founder and CEO Zachary Smith said will power “new low-latency services at the network edge.” We asked Packet to comment on AT&T’s assertion that processing data traffic at the edge of towers “isn’t how the mobility network works.”

“AT&T is exactly correct and aligned with Packet’s view that the edge data center opportunity is simply not about latency,” Smith said. “Being at the base of a cell tower doesn't give you automatic direct access to the networks. Quite the opposite. It’s true that new wireless options like CBRS/Private LTE and tier-three-market cloud and peering points are making inroads to bring network breakout closer to the end user. But the compelling thing about data centers at cell towers is the operational and financial model, not the latency. A shared-infrastructure tower deployment model is a proven example of a repeatable, scalable, and operationally cost-effective approach to distributed custom infrastructure.”

As the topology of the edge takes shape, it’s much clearer now that this mixture of new facilities and repurposed central offices will become  potential mobile core locations where AT&T’s 5G data network can break out traffic to NEC. For Vapor IO’s part, chief marketing officer Matt Trifiro characterized Shafer’s explanation of today’s mobility backhaul to the NTC as a flaw in AT&T’s own architecture which it now has the opportunity to ameliorate. (An AT&T spokesperson later clarified that the opportunity is to ameliorate a flaw in its 4G/LTE architecture with 5G.)

“It’s a tautology,” Trifiro said. “If all of AT&T’s network traffic must pass through their national data center, then there is zero opportunity to apply edge computing to those workloads, by definition.”

Trifiro acknowledged Shafer’s explanation of how data traffic routing presently works, stating that all wireless data passing through an NTC “must take a long, trombone-shaped routing path before it can be delivered to its destination.”

“This is the inherent flaw in the current design,” Vapor IO’s CMO said. “Before traffic from each device reaches the NTC, it cannot be routed to another client device, nor to other networks. Even in this example, where both devices are connected to AT&T, their traffic would still be sent to the NTC first. This stable but antiquated architecture, which was derived from some of the original voice-based telephone networks, becomes a significant obstacle to achieving the performance and cost targets required to make use cases like city-scale IoT viable for mass-market use.”

Break Out

The solution that Trifiro is suggesting is called Servicing Gateway Local Break Out (SGW-LBO), which would extend traffic steering functions to the network edge, thus enabling selected streams to be diverted to local applications. In a February 2018 white paper, Kinetic Edge Alliance partner Athonet suggested that an LBO approach would be necessary for latency-sensitive applications such as autonomous vehicle control systems, which can’t afford to wait for traffic to be “tromboned” through the carrier network and back again. The report implied not only that such an application might be impossible under the current wireless routing scheme, but that even the 5G network might break under the strain of this application unless localized data is allowed to be broken out.

Diagram of the relationship between user equipment (UE) and computing at an MEC edge site, which in Athonet’s view is separated from a core computing site

The SGW-LBO methodology is now considered an official part of Multi-Access Edge Computing (MEC), a structured definition of the concept that has been formally adopted by the 3GPP organization and incorporated into its methodologies for 5G wireless. According to a report produced for the ETSI standards organization, “local breakout at the SGWs is a new architecture for MEC that originates from operators’ desire to have a greater control on the granularity of the traffic that needs to be steered. This principle is dictated by the need to have the users able to reach both the MEC applications and the operator’s core site application in a selective manner over the same APN.”

If you ask the organization governing the worldwide implementation of 5G, this is how the mobility network will work in the near future. But for now, according to AT&T’s Shafer, “it’s not going to matter.”

Even if you would somehow “break open another traffic path at the tower,” he said, network topologies are different in every metro, making the data center-at-tower model not universally applicable.

“You’ve got to look at how fiber is connected within a metro. We’ve been doing this, as we’ve been looking at our metro-by-metro rollout plans: Where does the traffic actually go, and at what point would it be handed off, say, from a mobility network to a wireline network? Where would that happen, and where would it get routed back? In many cases, you’re talking about a hairpin from somewhere else. And it might be in that metro, but it could be somewhere else — hundreds of miles away. It depends on the scenario. You can’t just have many data centers somewhere and just expect that it’s going to happen without understanding the basic topology of the network.”

For now, service providers and data center architects are marching forward with a plan endorsed by the organization responsible for defining 5G — an organization that includes AT&T. Yet, at least today that plan is something AT&T describes as ranging from infeasible to impossible. The truth about edge computing will evidently differ from the 2017 edition of the future. There may yet be more than one edge in 5G, and where those edges clash, there could be friction.