Red Hat: We Didn't Pull CPU Microcode Update to Pass the Buck

Red Hat got into a bit of a PR snafu this week after it pulled from distribution a CPU microcode update meant to address the Spectre Variant 2 CPU design flaw. The Register, the tech news site that broke the story about the Meltdown and Spectre vulnerabilities in processors earlier this month, characterized the move as Red Hat washing its hands of the responsibility to provide customers with firmware patches to address the vulnerability by instructing them to get firmware updates from their hardware vendors instead:

It's true that Red Hat distributed microcode to address Variant 2 of Spectre to its customers and then removed it after Intel announced that some customers were experiencing unwanted system reboots after applying the firmware updates that included Spectre mitigations. But Red Hat isn't actually involved in writing the firmware updates. It passes the microcode created by chipmakers to its users "as a customer convenience."

"What I would have said if they'd asked us ahead of time is that microcode is something that CPU vendors develop," Jon Masters, chief ARM architect at Red Hat, told Data Center Knowledge in a phone interview Thursday. "It's actually an encrypted, signed binary image, so we don't have the capability, even if we wanted to produce microcode. It's a binary blob that we cannot generate. The only people who can actually generate that are the CPU vendors."

Related:How Red Hat Is Dealing With the Spectre of the CPU Meltdown

Although his day job is Red Hat's chief ARM architect, Masters has been one of the people leading the company's efforts to deal with Meltdown and Spectre.

Christopher Robinson with Red Hat's security assurance team concurred. "The microcode is not software that Red Hat owns. It's given to us by these different vendors. We've historically been kind of a middle man in supplying that. Right now, because we don't have a complete repository of all the updated microcode software, it's just easier for our subscribers to go straight to the source and grab the microcode update from their silicon provider."

In this case, the problem arose because Red Hat didn't have all of the firmware necessary for the fix to work for everybody.

"The microcode that was supplied to us did not cover all of the microprocessors that our customers possibly could use," Robinson said. "It appears, subsequently, there may have been two versions that could have some regressions. We're looking to provide our customers a consistent experience, so it's very difficult for us to say in this case, apply the package but in these other two cases talk to somebody else, or in a third case talk to another third or fourth party."

Related:Google Technique Offers Spectre Vulnerability Fix with No Performance Loss

Part of the confusion may have come from the differences in the way proprietary and open source development models work. Unlike Windows, which is developed by Microsoft 100 percent in-house, much of Red Hat Enterprise Linux is developed upstream. Although Red Hat is often part of the upstream development process, it's often prudent to wait until changes are made upstream before incorporating them downstream.

"We decided that since it wasn't a complete set [of microcode], the best thing to do is basically to wait for the community project to catch up, and once it's all there and it's all well tested and known to work well, then we'll revisit it," Masters said.

Until then, Red Hat is recommending customers to apply the Spectre and Meltdown related patches they supply and to contact their CPU vendors for firmware updates.

"That's the stance several vendors are taking as well," said Robinson. "It's not just us."