Inside the Famed Black Hat NOC

It's been called one of the most "hostile" networks in the world, but the managers of the Black Hat network operations center (NOC) contend that it's merely the most unique. After all, they can't just block all malicious-looking network traffic because they could inadvertently disrupt legitimate Black Hat activities, such as on-stage hacking tool demo or Trainings course exercise.

Bart Stump and Neil Wyler, aka Grifter, are eager to get back to the business of running and monitoring the network on-site this year at the Mandalay Bay in Las Vegas. The plexiglass-enclosed NOC went dark last year for the first time, when Black Hat USA shifted to a virtual show at the height of the COVID-19 pandemic. 

After that unplanned hiatus in 2020, this year's NOC team of more than a dozen volunteers will operate a network for what will be a scaled-down version of the on-site portion of Black Hat USA, which pivoted to a hybrid physical-virtual event in this latest phase of the pandemic. Most of the Trainings instructors will operate virtually, except for a few who will be on location and streaming their courses. The event itself will host an estimated 5,000 attendees onsite, far fewer than the 20,000 who crowded Mandalay's halls in 2019.

But even with a smaller number of users on the physical show's network — most registrants will be attending via Black Hat's virtual platform, Swapcard — NOC managers Stump and Wyler say their job this year won't be much different: ensuring network availability and detecting badness amid the famed experimental ethical-hacking on the network.

Their main worries are denial-of-service attacks or any disruption to the network, Wyler says. And they'll be hunting for any actual malicious network activity amid the legit traffic. 

"If we see any traffic that deviated from what is normal, we know to immediately act on it," he says, likening the process to finding "a needle in a stack of needles."

"Most SOCs would be exploding in alarms," Stump quips, but for the Black Hat NOC, it's the norm.

Busted

Each year at the end of Black Hat, Stump, who is a senior security consultant at Optiv, and Wyler, who is a principal threat hunter at NetWitness, deliver a postmortem presentation on what they found on the network that week, a talk they initially thought no one would attend or even care about. In fact, the first time they gave their NOC wrap-up presentation, they initially thought the packed crowd in the room were attendees from a previous session that had run past its time slot.

And each year they spot something notoriously notable on the network. There was the time a couple of years ago when NOC analysts tracked suspicious traffic from one of the Trainings sessions going to a police department website. (The NOC team, using intelligence from their security tools, can map the logical traffic to the physical location floor plan, and their tool marks the location of an incident with an image of Cookie Monster's head.) They traced the traffic to a classroom in the Mandalay Bay — and to a student in one of the Black Hat Trainings sessions who apparently had decided he would apply his newfound Web attack knowledge against his hometown police department's website, using various tools including OWASP's ZAP (Zed Access Proxy), which scans Web apps for vulnerabilities and is often used in penetration testing.

"We were sitting and hunting on stuff and saw some weird traffic going to what looked like a police precinct website," recalls Wyler. '[We realized] this isn't a demo — this is someone attacking it ... because they had learned something new in Web attacks," he says. They quickly shut it down and confronted the student.

In 2017, the team investigated some other "shady"-looking traffic on the network. 

"[It] was somebody actively exploiting a vulnerability that had been discussed from the stage, like only 12 hours earlier," Wyler recalls. "We were seeing someone who wrote some code for [the vulnerability] and was actively trying to exploit this."

It turned out the person messing with the vuln was weaving an exploitation of it into the Emotet Trojan. That rate of turnaround from vulnerability disclosure to an exploit mirrors the challenge businesses and security vendors regularly face: Attackers don't need a lot of time to find ways to abuse a software bug. 

"These are the kind of things we're up against as a security industry," Wyler says.

This year at Black Hat, Wyler's only "concern" — and he puts air quotes around the word — is that the crowd of DEF CON attendees who roll into Vegas early next week might get bored and decide to "mess around" on the Black Hat network in the runup to the hacker convention, which kicks off on Aug. 5. Many of them would typically be in town early to attend the BSides conference, but that was moved online this year due to the pandemic.

NDR Debut

But the Black Hat NOC operators admit their access to, ability to test-drive, and sometimes even shape new technologies isn't typical for most enterprises. For several years the NOC has deployed what some enterprises still consider "new" technologies, such as behavioral analytics, security orchestration, automation and response (SOAR), threat intelligence platforms, and endpoint detection and response (EDR) tools.

The automation piece has been key in speeding up their response to any shenanigans on the network. "We can't be running all over three floors of Mandalay Bay" to a classroom if they detect something suspicious, Wyler says. So if they spot command-and-control traffic, for instance, one playbook has their personal area network (PAN) firewalls place it in a captive portal with a warning in case it's not legit traffic in the class, for instance. If it is deemed malicious, the user gets a message to see the NOC to troubleshoot the issue.

The newest member of this year's NOC will be network detection and response (NDR), drawing from full packet capture, machine learning, and threat intelligence platforms. 

"We have three different solutions in there, all getting traffic mirrored" so the team can compare and analyze which system finds what, Wyler explains. "This year our big focus is on the NDR piece and [to] see what different solutions see or don't see [it on the network]."

This year's network was in the process of construction progress as of this posting, so the details weren't final. But like the 2019 network, it is expected to include PAN firewalls; Cisco Threat Grid analytics, Umbrella cloud security, and endpoint security products; Ruckus wireless access points, controllers, and switches; RSA's NetWitness threat detection and response platform; Gigamon data center monitoring tools; and CenturyLink 10G circuits, according to the Stump and Wyler.

Peering Into the Fishbowl

The NOC team won't host their regular walk-in tours of their command center this year due to the pandemic, but they plan to stream live via Twitch from their location at the Mandalay Bay conference center.

"We still wanted to have that interaction, so we'll do a Twitch stream to watch the same 'fishbowl' voyeuristically," Stump explains. "We'll have some dashboards still in the background ... and keep that interaction and at least the spirit of the [traditionally open] NOC."