Category: SBOM Skills

The importance of “Below the Fold” SBOMs

Jeff Luszcz

 

If you ask someone on your engineering team to name something in your SBOM, you might hear the names of a handful of well known open source libraries, especially if they have been in the news lately due to a serious vulnerability. These components are typically the ones they use day in and day out as part of the code they write for your product. They’ll likely be written in the programming language your team uses and managed by the repository manager they use as well.  In the newspaper business, Above the Fold is the place where important and critical news of the day is reported, while the news “Below the Fold” is perhaps perceived as less important or at least, less flashy. That said, 99% of the news in the newspaper is “Below the Fold”

 

In the world of SBOMs we find the same distinction in the types of SBOMs we build and manage. We often will be striving to make a SBOM for the dependencies our code touches directly, while neglecting or ignoring the foundations under our product; the operating system, containers and infrastructure pieces. 

 

While there are many historical reasons for this, the ability to ignore these layers is long over. 

 

What are you less likely to hear about?

 

Just as important but often neglected are what I call “Below the Fold” components, the infrastructure and operating systems that your entire stack is built upon, but they are often treated as an opaque box with little understanding of their internal dependencies or even existence.

 

Important components for your stack such as Operating Systems, Databases (you may have multiple optimized for different purposes), Message Queues or Monitoring/Observability components may be indispensable but are often ignored.

 

The inverse pyramid of SBOM knowledge

 

When you create or receive a SBOM for a product, the layers your product developers work on will be concentrated at the top of the stack with less and less knowledge as you get closer to the bottom. You might even see some components from each layer mixed in your final SBOM, but that is not a sign of completeness. Your SBOM might be 80% complete at the top level, but only 1% complete at the lowest. 

 

MORE TRACKING

Your Product

Infrastructure and System Services Layer

Operating System Layer

LESS TRACKING

 

The complexity of a modern software product means that multiple teams assemble layers of software, network together services and download container images, all while expecting that “someone, somewhere” will be keeping an eye on things. If you are reading this, you might be that someone!

 

Who selects and builds the Operating System layer?

 

Traditionally, an IT person or System Administrator was responsible for selecting, provisioning and updating the operating systems that products ran on. As time went on, this task became something that developers took on. As the march toward containers and Cloud Computing continued, it became easy to grab an operating system image from the Internet and move on. The component you depend on might even recommend an image name from a container service. This one component might bring in hundreds or thousands of other open source components silently. Your product may depend on more than one image, selected or created by different people at different times. Some may be kept up to date, others are a “one and done” and left in place with no updates or fixes even if serious vulnerabilities are discovered.

 

As attacks on components increase, concerns around poisoned components and legal requirements for accurate and complete SBOMs also increase. The need to track and manage these layers is becoming more important.

 

Why do people neglect the operating system layer?

 

Much of the industry has moved away from a dedicated job title and role (System Administrator) while at the same time, systems have become more complex and expectations around security and SBOM management have increased

 

Often there is a belief that Firewalls or Network scanners are performing this task, especially since they may see some of the same open source component names, CVEs and devices in the reports generated by them.

 

Network scanning tools give only a thin slice of the open source in use at these layers. These results will be concentrated on services that advertise their existence or explicitly transmit information about their name and version when prompted. This is a very small percentage of the actual number of components to be managed.

 

Additionally, containers and other operating systems are seen as opaque boxes. “That’s Linux” or “That’s our Postgres Server” and tracked as a single line item if they are even tracked at all. There may be a belief that a vendor or open source project will be sending out updates or alerts, even if this is not the case.

 

Even if you are doing container scanning, the SBOM might not be sending active alerts depending on the tools used to scan it before deployment. The depth of analysis might not be appropriate for the current threats and many times their alerts might be ignored since they are going to the wrong people

 

What serious concerns come from the operating system layer?

 

Depending on your industry and distribution model, you will be concerned about a few different things that a SBOM will help you identify and manage. These include:

 

  • Vulnerabilities, CVEs and Security issues
  • Open Source License Compliance
  • Poisoned Components and Malware
  • Supportability and End of Life issues for OSS Components

 

By creating SBOMs for these layers, you will be in a better place to manage each of these concerns.

 

Where are we going with Below the Fold SBOMS?

 

As teams look lower in the stack, I expect to see a few trends occur. The first is that the tools available for teams to perform better analysis and SBOM creation for the lower levels will appear. These might be part of your existing SCA toolset, or may be from a separate vendor with a concentration on container and infrastructure experience. It is okay to mix and match tools to get a better result! 

 

SCA products performing Binary analysis of existing images and binaries will help do spot checks for the Vulnerability of the Day, especially before vendors and OSS projects have their SBOMs created and shared.

 

There is movement to use Hardened and Minimal images. These are images specifically designed to be as secure as possible, published often with vulnerabilities fixed and in some cases, with unneeded functionality removed in order to reduce the attack surface. This will help, but you will still find cases where your teams have built their own images, or are layering open source components on these hardened images. 

 

Additionally, scanners to help understand the interactions and full service dependencies of a software system are appearing. These help collect and visualize the network and data paths between different silos. By getting a complete understanding of all the containers, services and teams involved with running your product, you will have a more complete and useful set of SBOMs.

 

Who Manages the security for each layer?

 

It’s important to understand who is responsible for the security response for each layer. Do you have a DevSecOps team who can handle it all or do you need to pull in a classic IT team to help with operating system and VM issues? It is important to examine every container,  layer and service and explicitly assign it to a security response team. Doing this over a holiday weekend idue to a high severity vulnerability is a recipe for disaster.

 

Questions to Ask Your Team and Yourself

 

  • Do we know all the containers our system depends on?
  • Where did these images come from, what is the update process?
  • Are these images being scanned and managed by a SCA product?
  • Can we produce a SBOM for these images?
  • Do we depend on any “real” server hardware with an Operating system on it?
  • Do we use Virtual Machines (VMs) anywhere?
  • Where did those images come from, what is the update process?
  • What infrastructure components like Databases, Queues do we use?
  • Do they have their own operating system / container to manage?
  • Are there any temporary systems that are still running? Something for a single customer or integration?
  • What is our process to keep our SBOMs up to date for everything we discovered in this review?

 

Wrapping Up

As the legal requirements for complete and sharable SBOMS become more stringent, the need to examine the lower foundations of your software stack becomes more important. You should bring up these concerns with your software team and discover the layers and containers that should be brought under management. You may find varying levels of security and compliance even in the same team. Putting pressure to produce SBOMs on suppliers of foundational components can help you get a better understanding of your complete supply chain and dependencies list. Bring in the right tools for each layer and make sure that the results are viewed by a team with the responsibility to protect the company based on the results of these scans.

 

As said before, while sometimes you need to read the first page, 99% of the news is in the rest of the paper!

 

The S in SBOM should stand for Sharable! 

Jeff Luszcz

a person hands another person a document

 

What serious issues besides vulnerabilities and compliance issues will keep you from sharing your SBOM?

 

You are going to find yourself in a jam. An urgent request for a Software Bill of Materials (SBOM) is going to come across your desk. Thinking that it’s a simple “push a button, get a report” process you’re going to ask an engineer to “push a button and get you a report”.

 

When you look at that report, your first SBOM, you are going to find a long list of problems that you won’t want to share with your customers or the government or whoever else is demanding a SBOM ASAP!

 

Let’s talk about what problems you’ll encounter.

 

In a previous article “Your first SBOM is going to stink. Don’t panic, get started fixing it!” I discussed common mechanical problems in getting to your first complete SBOM. These included Completeness, Depth, Unremediated Vulnerabilities, Open Source License Violations, and Over Delivery.

 

As part of the process, you will clear out vulnerabilities and open source license violations but along the way you will find other issues that may prevent you from sharing your SBOM.

 

These include Commercial Components with licensing problems, information you can’t share due to NDAs, architecture issues you don’t want exposed, export control issues, competitor’s technology, perceived over dependence on open source, and dependence on old code or technology.

 

Commercial Components and Technology

While much of the discussion around SBOMs has revolved around Open Source, there is still a large dependence on Commercial technology in the modern software stack. Most Software Composition Analysis (SCA) tools do not discover or report on Commercial components. If the contract or legislation your SBOM is being generated in order to fulfill requires a list of “all third party software” or explicitly calls out Commercial components as reportable, it will be required to discover and add these components to your SBOM. This is typically a manual process. 

 

These commercial components often require payment or contract terms in order to be used. It is important to confirm that the usage is being tracked and properly paid for under the terms of the agreement. It is not uncommon to find commercial code that should have been removed at the end of a contract term still being used and shipped in a codebase. This may require re-licensing for continued use or rework to remove.

 

Additionally, you may find yourself with a conflict between the requirement to produce a complete SBOM and a Non-Disclosure Agreement with your technology vendor. It is important to understand any restriction on your ability to tell others about your usage of this commercial technology. It is also important to understand how to resolve any conflicts between a requirement to share everything with an agreement to keep some items confidential.

 

Competitors’ Technology or Relationship Disclosure

The larger your organization, the more likely that products have had a long lifetime, possibly being brought in through mergers and acquisitions or through technology partnerships. The technology stack that was appropriate a decade ago, may no longer be politically or commercially appropriate now. While reviewing your SBOM you may find indicators of components or technology coming from current competitors. It is very important to understand the licensing framework that this technology requires. Are there restrictions on using this technology? Has the cost become prohibitive? Will this potentially public use cause marketing, business or legal issues for your organization?

 

Will sharing of this SBOM alert competitors to your technology usage or business relationships?

It is common for business relationships to be confidential or at least kept quiet. Through this sharing of a SBOM, will these relationships become public? For example, does your new release swap out one technology (commercial or open source) with another, will this be a surprise to your own vendors or community? Is your executive management aware of this? Are there additional NDAs or agreements that could be put into place in order to limit sharing of this information? Are you prevented from sharing this relationship information in your SBOM? How can you legally satisfy both sides?

 

Export Licenses and Review

The shipping of certain technology may require export licenses or compliance review depending on your location and the type of technology being used. By creating a SBOM you may find yourself aware of technology that requires this process and review. It is important to bring in experts who are aware of export restrictions and requirements especially if there has been potentially missing compliance.  Additionally, your organization may already be producing this required documentation which can be helpful in producing a complete SBOM. SCA scans often miss the commercial (and sometimes open source) components that may require export review. This is a manual review step when producing your SBOM.

 

Perceived Overdependence on Open Source Technology or Engines

Even though 90% of a project’s technology stack is typically open source, it can sometimes be a surprise to customers or the public to see the actual contents of the SBOM. If a company has made a big deal about “their secret sauce” or cutting edge technology, but the SBOM shows that it’s based heavily on an Open Source engine, you may find yourself with a marketing problem. If your organization is concerned about a customer asking “Why are we paying so much for this if it’s just rebranded Open Source?” it’s important to get ahead of the question and have a well thought out response to change this impression. This also might also be a time to look at your public and financial support of the open source you depend on. If you are heavily dependent on certain open source technology are you a financial supporter or major code contributor to the project? “Of course we are heavily dependent on it, we’re a major financial and code contributor to the project” can be a great response to this objection!

 

Dependence on Old code or Technologies

Before SBOMs, many commercial products were Closed Boxes where customers had little insight into how and when they were put together. By sharing a SBOM, it will be clear the years that the majority of the project was built in, the types of source code languages being used, and technology choices that can be second guessed.

 

If no new components have been added in years, you may have to manage customer expectations about perceived lack of development. If the technology stack is based on a programming language or architecture that is out of the mainstream or has become passe, you also may find yourself with a customer management problem. It is important to get an understanding of what objections customers may have when they read your SBOM for the first time based on age, architecture or design choices. These choices are often set in stone though, and management may need to provide staff with clear and concise responses to prevent customer concern.

 

Sharing what you Know

The process to getting to a Sharable SBOM requires attention to detail, coordination of multiple teams, and an understanding of how technological decisions can be impacted by legal, commercial and political concerns. It’s important to be aware of how SBOM creation and dissemination is occurring in your organization and to make sure that the business needs of the organization are reflected in the SBOM process. By being aware of these issues, you can prevent legal and commercial issues, as well as preventing ethical breaches when teams find themselves stuck between two contrary requirements. A little early planning goes a long way to showing that you are aware and in control of your technology and will continue to be a good partner going forward.

 

 

 

 

How Falsehoods, Folklore and Foul-ups hurt your SBOM

Jeff Luszcz

 

 

In a perfect world, the software components our teams select are flawless. They have no software vulnerabilities, have an appropriate Open Source license, and are well maintained by a loving team forever. Unfortunately, we don’t live in that world and have to manage software that is built and selected by humans with differing levels of knowledge, webs of complexity, and physical and emotional lifetimes. These can lead to Open Source license violations, software vulnerabilities and a general feeling of unease in your customers when they read your Software Bill of Material (SBOM). Let’s take a look at some common problems in your SBOM and what caused them.

 

Foul-up: Someone claims “It’s public domain” or “It’s Open Source!”

There are two common mistakes that developers make when first encountering open source software licenses. The first is mistaking and misusing the term “Public Domain” in order to mean that something is Open Source. The second is to believe that the term Open Source gives them carte blanche to use the component any way they want with few or no restrictions. While there are open source components that are considered “Public Domain”, many times this term is being misused in place of the phrase “Open Source”. Additionally, “Open Source” is an umbrella term that includes obligations ranging from “releasing all your source code” to ones with which have almost no requirements at all. I have often seen the wildly inaccurate statement “This code is public domain under the terms of the General Public License” which should be your sign to walk, not run, away from that component!

The fallout from this misunderstanding is that the actual open source licensing of these components are not visible until a SBOM is created and reviewed (often by customers!). This shows up in your SBOM as open source components with licenses that are contrary to your license policy (if you have one) or cause a serious license conflict. Examples of this would be components licensed under the General Public License (GPL) in a distributed closed source application like a smartphone or desktop app or components with an Affero General Public License (AGPL) in a SaaS app.

 

Foul-up; “I didn’t know I needed permission and/or read the company’s license policy”

A similar mistake is caused by a developer or team member not knowing about your organization’s open source policy or guidelines. Introducing new open source or commercial components without understanding the impact of their inclusion can lead to serious and expensive problems. Periodic training (starting at onboarding and continuing on in the future) is important to teach best practices and prevent license violations as well as help reduce vulnerabilities and other security exposures. Additionally, having a centralized Open Source Portal with training and policy documents and scanning using Software Composition Analysis (SCA) software is a great way to prevent and catch problems as they occur.

 

Folklore “We don’t ship software so we don’t need to worry about Open Source licenses”

Another problem occurs when licenses are simply not checked at all before selecting a component. This often happens when a developer misunderstands the impact of software distribution models on open source licenses. While it is true that many open source licenses only come in effect when the software is shipped, there are many open source and commercial licenses that are in effect no matter what the distribution model is.

An important consideration is that distribution models change. While you might not be distributing the software today, you may find yourself supporting on-premises installations for a large customer. Now suddenly every component you have built into your application is distributed and the open source license obligations are required to be complied with. 

Even the most SaaS company may find itself making software distribution of utilities, onboarding tools and integrations. Misusing a SaaS open source policy for these projects can cause serious license violation surprises.

 

Foul-up “It doesn’t have any vulnerabilities….right now”

When a component is first selected by a developer it is less likely to contain known vulnerabilities or CVEs associated with it. Typically vulnerabilities are discovered as time goes on and after security researchers get access to a component version in question.

A SCA tool is a great way to keep on top of the vulnerabilities that occur in the components you select. Make sure you are examining both the top level as well as the transitive dependencies.

Even without a SCA tool, if your versions are years out of date, you almost certainly have vulnerabilities present in your dependencies..

 

Foul-up: “I didn’t know I needed to check the component’s transitive dependencies”

While your developers may be aware of the license and vulnerability status of the components they select, they may be unaware of the sub-components brought in automatically by the repository management system (e.g. NPM or Maven). These components, called transitive dependencies, are just as important as the top level component, but are often overlooked or ignored. Your ability to fix any problems with these components is limited (i.e. you are unlikely to be able to fix a problem faster or better than the parent open source project themselves!) but you may face pressure from your customers to fix these problems ASAP.  By scanning at component selection time, you can get ahead of core license conflicts and ancient unmanaged vulnerabilities. By running a SCA system with alerting, you can get warned of changes in vulnerability status and plan required upgrades as necessary. Additionally, you can help your upstream open source projects by becoming aware of issues they may not have the technology or knowledge to manage.

 

Foul-up “I thought my SCA tool handled Commercial Vulnerabilities”

If you are using any commercial dependencies, you may be surprised that most SCA scanners do not report on their vulnerabilities/CVEs and many are unable to report on even the Open Source components used by those commercial dependencies. This is a large hole in many companies’ open source management processes. It shows one important reason why requesting your vendors’ SBOMs and reviewing their contents not just at ingestion time, but continuously if possible.

 

Folklore: “I got it from the Internet, I must be able to use it”

No news is NOT good news. Another common source of unexpected SBOM churn is when code is downloaded from web pages, forums or project documentation sites. Any time you take software or source code from another place, it is required that you know what permissions to use these resources are. A clear open source license, or permission statement is typically required in order to legally use this source code.  If components appear in your SBOM with unknown, blank or generic sounding license metadata, it is important to dive deeper into what the true permissions exist for this component. If the original author has not put any in place, it may be helpful to suggest to them a license that works well for you (perhaps a Apache 2.0 or MIT license). Be aware that the author may select a license that is incompatible with your business model or desires and may even require commercial payment or forbid your use!

While it’s always best to know your obligations before you build something into your application, clean up work is always required.

 

Code from forums like Stack Overflow often have published license guidelines that may be a surprise for you. For example, source code from Stack Overflow is licensed under the Creative Commons Share-alike 4.0 license by default (though individual authors may override that) See https://stackoverflow.com/legal/terms-of-service#licensing and https://meta.stackexchange.com/questions/347758/creative-commons-licensing-ui-and-data-updates for more information.

 

 Foul-up: “I’m sure someone is still maintaining this component!”

Deciding to incorporate an open source component into your project is the beginning of a long term relationship. While you are not explicitly owed anything by the open source maintainer, there is often an expectation that vulnerabilities will be fixed, features will be worked on and the project will maintain some sort of line of communication with the community.  In some cases projects die. This might be due to burnout, change in employment, lack of interest or even the maintainer dying!

When selecting a component, you do not want to select a component that is already dead. Look at release cadence, open issues, known vulnerabilities and indicators of response from the maintainer. If a project is already dead or dying, don’t select it unless you are able to shoulder the maintenance of the component for as long as you use it.

Periodically check your components’ health status. If projects die you should replace them or take over maintenance if possible.

 

Humans are making these decisions, remember them!

Humans are making the decisions that impact your product every day and it’s important to understand the thinking that goes into these decisions. By understanding the reasons why problems occur you can better put in policies, procedures and products that help prevent serious impacts and future mistakes. Overarching policies that don’t take into account the human factor are doomed to seeing the same mistakes being made over and over again.

Humans make mistakes, but also learn from their mistakes. Talk to the developers and the development teams about what needed to be fixed. Make it part of your retrospectives as well as release checklists. SCA tools allow policies to be enforced automatically, try to automate as many of these as you can, but also understand the limitations of current tools. 

Onboarding training and yearly training should go beyond simplistic “GPL bad! Upgrade often!” slides.  

The more your SBOMs are living documents, the healthier they will be. Your first experience reviewing a SBOM may be scary but with time and experience your company will be healthier and safer. Good luck!