How Red Hat Is Dealing With the Spectre of the CPU Meltdown

By now there's probably no one who follows tech even a little who hasn't heard about Meltdown and Spectre, two related security vulnerabilities that differ from garden variety security issues because they're hardware and not software vulnerabilities. They're also notable because they're built into just about every CPU that's been manufactured for decades.

Those reading media accounts published Wednesday and Thursday are probably under the impression that while Spectre affects Intel, AMD and ARM CPUs, that Meltdown affects only Intel products -- or perhaps all Intel CPUs and some ARM chips.

Not so, says Red Hat's Jon Masters. Both vulnerabilities are basically architecture agnostic.

"I think Intel got a lot of unfair attention this week," Masters told Data Center Knowledge. "The reality is that it's a cross industry, cross vendor issue that affects pretty much every architecture. It's not just Intel, AMD and ARM, it's actually every modern architecture."

Masters should know. Although his day job is as Red Hat's chief ARM architect, he's been one of the people leading the company's efforts to deal with Meltdown/Spectre.

The Intel issue came up as he was answering a question about the impact reported performance issues with the Meltdown fix would have on organizations running a large number of servers. Some press reports have indicated servers could take a performance hit of up to 30 percent after patching -- with Red Hat putting the number at between 0-20 percent.

Related:Intel, Microsoft Deal With Widespread Chip Security Vulnerability

"Our updates follow a Red Hat policy of security by default," he said. "Our updates will always be the most secure we can make them, so the consequence when customers install these updates is that they will see, by default, the associated performance hits.

"But they can do a risk analysis and choose to what extent they want full mitigation. For example, if they're in a closed lab environment, for HPC or something like that, and they've got a very controlled environment, maybe they will decide that they don't want all of these mitigations enabled. They can choose to disable them and they'll get 99 percent of the performance back."

The reduction in performance seen by those opting for full protection will depend primarily on the workload of the machine.

"In our testing, there are very few workloads that are on the order of 18 to 20 percent," he said. "Most workloads are in the 5 to 10 percent range in terms of the performance impact, depending on the vendor and depending on the generation of processor."

Meltdown has been the vulnerability that's been getting the most attention because it's readily exploitable, with proof of concept code publicly available.

Related:Red Hat Bets Data Centers are Ready for ARM Servers

"I really think it's not as big a deal in the longer term as people are making out," Masters said. "Yes, it's a significant threat. It is something that we obviously need to patch very swiftly. That said, it's a very complicated attack. It's existed for potentially up to several decades on different families of microprocessors as a class of attack that was theoretically possible, but finding it took a lot of effort. It's something that is very hard to reproduce. It's unlikely that many people out there would have come up with this independently prior to the disclosure, and patching it is relatively straight forward."

Spectre, which is actually two related vulnerabilities, also raised some fears when some news sites reported that it's unfixable. That's not exactly true either, according to Masters, although the fix isn't as easy or straightforward as it is with Meltdown.

"The first half of it is relatively straightforward and actually easily patched," he explained. "The second half potentially has a cross virtual machine exploit. You can have virtual machines attack each other or attack the hypervisor. We also have a mitigation for that in all of our updates, but the mitigation relies on a microcode or system firmware update in order to be firmly effective. I think that's going to complicate the process of deploying this."

That's not too worrying for Masters, however, because the attack for Spectre is, for all intents and purposes, impossible.

"The Google research team said they were able to do it one time, with one particular processor, and with one particular operating system version," he said. "It took hours to set up. It was very difficult to do, and the bit rate of extraction of information in that case was one-and-a-half kilobytes per second. That's enough to be very concerning, because you could extract the security key, but it was very difficult to pull off and as far as we can see there are no public exploits in the wild for this."

Because this is essentially a hardware issue, the steps being taken to close the vulnerability on existing processors are like long term bandaids. Ultimately, this is an issue that will have to be addressed by the processor vendors for future generations of CPUs. The good news is this appears to be doable, even though part of the long term fix will require some changes in software design.

"It's actually not a very difficult logic change in future generations of processor to mitigate for Meltdown," Masters said, after cautioning that he's not a CPU designer. "It's actually a fairly straightforward change, and very quick to integrate. Of course, rapidity in terms of hardware design is a different order of magnitude from software updates. For the Spectre attack, the fix is very simple. And again, that one is pretty straightforward to impliment in future generations."

Part of Spectre will require some changes in software design, which is probably why some news reports have called it "unfixable."

"The first half of Spectre is easy to patch in software and basically has no performance overhead," he explained, "but it's very difficult to fix in hardware. We will probably be dealing with the first half of Spectre for many years to come. That will involve things like toolchains, compilers and things of that nature being updated to automatically address it, rather than what we're doing right now, which is using scanning tools that have been developed to find where these problems are and insert these little extra instructions into our kernels to prevent problems."