If you ask someone on your engineering team to name something in your SBOM, you might hear the names of a handful of well known open source libraries, especially if they have been in the news lately due to a serious vulnerability. These components are typically the ones they use day in and day out as part of the code they write for your product. They’ll likely be written in the programming language your team uses and managed by the repository manager they use as well. In the newspaper business, Above the Fold is the place where important and critical news of the day is reported, while the news “Below the Fold” is perhaps perceived as less important or at least, less flashy. That said, 99% of the news in the newspaper is “Below the Fold”
In the world of SBOMs we find the same distinction in the types of SBOMs we build and manage. We often will be striving to make a SBOM for the dependencies our code touches directly, while neglecting or ignoring the foundations under our product; the operating system, containers and infrastructure pieces.
While there are many historical reasons for this, the ability to ignore these layers is long over.
What are you less likely to hear about?
Just as important but often neglected are what I call “Below the Fold” components, the infrastructure and operating systems that your entire stack is built upon, but they are often treated as an opaque box with little understanding of their internal dependencies or even existence.
Important components for your stack such as Operating Systems, Databases (you may have multiple optimized for different purposes), Message Queues or Monitoring/Observability components may be indispensable but are often ignored.
The inverse pyramid of SBOM knowledge
When you create or receive a SBOM for a product, the layers your product developers work on will be concentrated at the top of the stack with less and less knowledge as you get closer to the bottom. You might even see some components from each layer mixed in your final SBOM, but that is not a sign of completeness. Your SBOM might be 80% complete at the top level, but only 1% complete at the lowest.
MORE TRACKING
Your Product
Infrastructure and System Services Layer
Operating System Layer
LESS TRACKING
The complexity of a modern software product means that multiple teams assemble layers of software, network together services and download container images, all while expecting that “someone, somewhere” will be keeping an eye on things. If you are reading this, you might be that someone!
Who selects and builds the Operating System layer?
Traditionally, an IT person or System Administrator was responsible for selecting, provisioning and updating the operating systems that products ran on. As time went on, this task became something that developers took on. As the march toward containers and Cloud Computing continued, it became easy to grab an operating system image from the Internet and move on. The component you depend on might even recommend an image name from a container service. This one component might bring in hundreds or thousands of other open source components silently. Your product may depend on more than one image, selected or created by different people at different times. Some may be kept up to date, others are a “one and done” and left in place with no updates or fixes even if serious vulnerabilities are discovered.
As attacks on components increase, concerns around poisoned components and legal requirements for accurate and complete SBOMs also increase. The need to track and manage these layers is becoming more important.
Why do people neglect the operating system layer?
Much of the industry has moved away from a dedicated job title and role (System Administrator) while at the same time, systems have become more complex and expectations around security and SBOM management have increased
Often there is a belief that Firewalls or Network scanners are performing this task, especially since they may see some of the same open source component names, CVEs and devices in the reports generated by them.
Network scanning tools give only a thin slice of the open source in use at these layers. These results will be concentrated on services that advertise their existence or explicitly transmit information about their name and version when prompted. This is a very small percentage of the actual number of components to be managed.
Additionally, containers and other operating systems are seen as opaque boxes. “That’s Linux” or “That’s our Postgres Server” and tracked as a single line item if they are even tracked at all. There may be a belief that a vendor or open source project will be sending out updates or alerts, even if this is not the case.
Even if you are doing container scanning, the SBOM might not be sending active alerts depending on the tools used to scan it before deployment. The depth of analysis might not be appropriate for the current threats and many times their alerts might be ignored since they are going to the wrong people
What serious concerns come from the operating system layer?
Depending on your industry and distribution model, you will be concerned about a few different things that a SBOM will help you identify and manage. These include:
- Vulnerabilities, CVEs and Security issues
- Open Source License Compliance
- Poisoned Components and Malware
- Supportability and End of Life issues for OSS Components
By creating SBOMs for these layers, you will be in a better place to manage each of these concerns.
Where are we going with Below the Fold SBOMS?
As teams look lower in the stack, I expect to see a few trends occur. The first is that the tools available for teams to perform better analysis and SBOM creation for the lower levels will appear. These might be part of your existing SCA toolset, or may be from a separate vendor with a concentration on container and infrastructure experience. It is okay to mix and match tools to get a better result!
SCA products performing Binary analysis of existing images and binaries will help do spot checks for the Vulnerability of the Day, especially before vendors and OSS projects have their SBOMs created and shared.
There is movement to use Hardened and Minimal images. These are images specifically designed to be as secure as possible, published often with vulnerabilities fixed and in some cases, with unneeded functionality removed in order to reduce the attack surface. This will help, but you will still find cases where your teams have built their own images, or are layering open source components on these hardened images.
Additionally, scanners to help understand the interactions and full service dependencies of a software system are appearing. These help collect and visualize the network and data paths between different silos. By getting a complete understanding of all the containers, services and teams involved with running your product, you will have a more complete and useful set of SBOMs.
Who Manages the security for each layer?
It’s important to understand who is responsible for the security response for each layer. Do you have a DevSecOps team who can handle it all or do you need to pull in a classic IT team to help with operating system and VM issues? It is important to examine every container, layer and service and explicitly assign it to a security response team. Doing this over a holiday weekend idue to a high severity vulnerability is a recipe for disaster.
Questions to Ask Your Team and Yourself
- Do we know all the containers our system depends on?
- Where did these images come from, what is the update process?
- Are these images being scanned and managed by a SCA product?
- Can we produce a SBOM for these images?
- Do we depend on any “real” server hardware with an Operating system on it?
- Do we use Virtual Machines (VMs) anywhere?
- Where did those images come from, what is the update process?
- What infrastructure components like Databases, Queues do we use?
- Do they have their own operating system / container to manage?
- Are there any temporary systems that are still running? Something for a single customer or integration?
- What is our process to keep our SBOMs up to date for everything we discovered in this review?
Wrapping Up
As the legal requirements for complete and sharable SBOMS become more stringent, the need to examine the lower foundations of your software stack becomes more important. You should bring up these concerns with your software team and discover the layers and containers that should be brought under management. You may find varying levels of security and compliance even in the same team. Putting pressure to produce SBOMs on suppliers of foundational components can help you get a better understanding of your complete supply chain and dependencies list. Bring in the right tools for each layer and make sure that the results are viewed by a team with the responsibility to protect the company based on the results of these scans.
As said before, while sometimes you need to read the first page, 99% of the news is in the rest of the paper!