Log4Shell Vulnerability Highlights Software Supply Chain Issues

Open source software has eaten the world, but last month's Log4Shell vulnerability chaos highlights the potential dangers when enterprises don't treat it with the respect it deserves.

The danger is two-fold. First, the software supply chain is full of known vulnerabilities that companies aren't patching. And, second, attackers have also begun exploiting the lack of attention paid to software project security to deliberately add backdoors and other malicious components.

Open source is everywhere, and it's all vulnerable

According to Synopsys' 2021 open source security and risk analysis report, 98 percent of enterprise software projects, both internal and commercial, contain some open source code.

And it wasn't just the occasional code fragment or library, either. For an average application, 75% of the codebase was open source, the report said.

"The pace of technology development these days is only possible because we have great quantities of open-source software at our disposal," said Nicko van Someren, CTO at Absolute Software. "This lets us build bigger, better products much faster than if we needed to write every line of code ourselves."

A similar report released by Osterman Research and GrammaTech last summer showed that 100% of commercial off-the-shelf applications contained open source components.

As open source usage goes up, so do the risks.

According to Osterman, all those applications -- all 100% of them -- also had open source components with security vulnerabilities, 85% of them critical.

The open source code can be buried several layers deep in an application, in a dependency of a dependency of a dependency. When it's embedded in a proprietary software package, it may be extremely difficult for a company to identify and patch.

In fact, according to a report released by Sonatype on Monday, the Log4j library has been downloaded more than 10 million times since Dec. 10, when the Log4Shell vulnerability was discovered. Of those downloads, 40% were of the older, vulnerable versions of the code.

Even more ominously, that percentage hasn't changed much over the past month. Between Dec. 10 and Dec. 12, the percentage of vulnerable downloads dropped from 100%, before the patch was available, to 50%. By Dec. 18, it was down to 40% -- and then has stayed roughly at that same level ever since.

Patching open source software also carries an additional challenge compared to commercial software, said Tim Mackey, principal security strategist at Synopsys.

"Commercial software vendors know who their customers are," he said. "But open source projects rarely do as anonymous downloads don’t allow for push notifications."

That means that it's up to the companies themselves to stay on top of necessary updates and patches.

Another challenge, according to Roy Illsley, analyst at Omdia, is that there are many cases in which companies aren't going to want to install patches automatically, without testing them first.

"Very few people are fully automated," he said. "Because they don't trust it. They've got to go through the testing."

Legacy systems are also a challenge, he said; if new versions of software don't support the older hardware, then a company can't update without doing a major systems overhaul first.

Finally, a company might not know that there's code that needs to be patched because it's buried inside of proprietary tools.

Data center cybersecurity managers should be taking this opportunity to ask all their vendors to provide software bills of materials, he said, though they'll have the most leverage during the initial purchase or at contract renewals. Still, the best vendors are going to be paying attention and stepping up.

"All the surveys we run, the data that comes back, one of the things that customers value is trust in their partners," he said.

A software bill of materials is a list of all the software components in a digital product, explained Kelly Rozumalski, vice president at Booz Allen Hamilton. If all vendors provided one, it would have been much easier for companies to respond to challenges like Log4Shell.

"The lack of SBOM resulted in many organizations struggling to figure out where the vulnerability existed on their network," she told Data Center Knowledge.

We might start seeing this happening soon.

Last May, President Biden issued an executive order requiring a software bill of materials from vendors that provide software to the federal government.

Two days later, the Cloud Native Computing Foundation released a best-practices white paper recommending that all vendors provide an SBOM where possible, with clear and direct links to dependencies.

The Log4Shell vulnerability may well be the final kick in the pants that the software industry needs to step up.

Open source projects are a soft target

"Attackers are finding many systems harder to attack directly, so they have started shifting to a softer target: software supply chains," said David Wheeler, director of open source supply chain security at Linux Foundation.

Attacks on the open source software supply chain increased 650% last year compared to 2020, according to Sonatype's state of the software supply chain report, released in September.

Attackers will name malicious projects with the same names as legitimate ones, just in different repositories. Or create ones that use the most common typos of those names.

Attackers can also steal a developer's credentials and submit malicious code under their name -- or join an open source project and submit malicious code themselves.

Many open source projects have few developers involved, said Wheeler, and the malicious code injection gets missed.

He suggested that companies take steps to ensure that the open source software libraries they're using are from the correct repositories and are spelled correctly. Companies can also demand a software bill of materials from their vendors so that they have a list of all the open source code in their environment -- and check those lists for typosquatting packages as well.

"A bigger challenge is malicious code injection," he said. "It’s a less common kind of attack today, compared to some others, but it’s important."

He suggested that if a particular piece of open source software is important to a company, that they actively participate in that project -- or, at least, look for projects that have multiple maintainers and multi-person code review processes.

"In the longer run, get involved in the Open Source Security Foundation," he suggested. "This is a consortium of many organizations working on systemic solutions. For example, the OpenSSF is starting a project to distribute multi-factor authentication tokens to some OSS developers, so that a stolen password will not easily lead to posting maliciously-subverted packages in their names."

OpenSSF is also encouraging good security practices, he said, such as multi-person review, and is investigating ways to directly detect typosquatting and malicious code.

To underscore how dangerous a situation it is when a critical open source project has too few developers, the volunteer maintainer of the popular colors.js and faker.js libraries, Marak Squires, deliberately sabotaged the code as a protest, adding an "infinite loop" that caused thousands of other software projects to crash.

According to a Sonatype report, colors.js has been downloaded 3.3 billion times and more than 19,000 other software projects depend on it, and faker.js has been downloaded 272 million times and is used in 2,500 other projects.

It's time for enterprises to step up and contribute to the open source community, experts say.

“Enterprises shouldn't see open source contributions as charity, because it isn't," said Dan Petro, lead researcher at Bishop Fox, a security testing firm. "It's an investment in the company's own technology. This goes for security just as much as ordinary development."

Companies need to stop thinking of their open source work, such as security audits of open source projects, as pro-bono work that they do out of the goodness of their hearts, he said.

"Companies should invest in the technology they use themselves, and see it as auditing their own security posture. For example, a media company may rest heavily on a particular open source video processing library to do business-critical tasks. Finding and fixing vulnerabilities in that library just makes good business sense.”

If few folks want to audit and test open source software, even fewer want to actually get their hands dirty and assist the mostly unpaid maintainers with fixing and regression testing the vulnerable software, said Casey Ellis, founder and CTO at Bugcrowd, a San Francisco, Calif.-based crowdsourced cybersecurity company.

With Log4Shell shining a spotlight on this problem, even the FTC is considering stepping in.

"The Log4j vulnerability is part of a broader set of structural issues," the agency said in a statement released last week, where it also pledged to prosecute companies that fail to patch the vulnerability and put consumers at risk. "It is one of thousands of unheralded but critically important open-source services that are used across a near-innumerable variety of internet companies. These projects are often created and maintained by volunteers, who don’t always have adequate resources and personnel for incident response and proactive maintenance even as their projects are critical to the internet economy. This overall dynamic is something the FTC will consider as we work to address the root issues that endanger user security."