Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2021, 29 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Back to the Drawing Board: Building Containers with SBoMs - Nisha Kumar, VMware

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Back to the Drawing Board: Building Containers with SBoMs - Nisha Kumar, VMware

A Software Bill of Materials or SBoM is a list of software components that comprise a software artifact, be it firmware, OS, a VM, and yes, a container. We can generate an SBoM for container images post build using image scanners like Claire, Trivy, Tern, and Syft. This method is not foolproof, however, as they rely on metadata existing in the container filesystem (such as package manifests) in order to report on them. If a container goes through a multistage build or tools like Docker-slim to reduce the attack surface of the container, all that metadata is gone. How do we get more accurate and consistent SBoMs for containers? We generate them at container build time. This talk demonstrates how we can do that with tools like Tern, Buildah, and the OCI specification. We will get back to the basics of building containers, learn about the OCI specification, and make a container builder which can generate an SBoM at build time.

A

Hello and welcome to my talk back to the drawing board, building containers with s-bombs, I'm nisha kumar and I'm a senior open source engineer at vmware.

A

So I'm hoping I'm hoping you got enough messaging about s-bombs and why we need them, but if you haven't here are a couple of analogies: if you've ever looked at the ingredients on a product's packaging, you may be surprised at what you find. For example, twinkies are not vegetarian and coconut milk conditioner does not have coconut milk.

A

As with these ingredients, a software bill of materials helps you make informed decisions about the distribution and maintenance of the software you consume either as a tool or as a dependency included in the software. You distribute.

A

You may be asking yourself: why do I need to generate s-pumps for containers specifically well, this is a picture of a node.js container. As you can see, there are plenty of components included that may not even be needed by a node.js application.

A

One of the my friends on twitter noted that there was a libra office hanging around there somewhere. Why would node.js require libreoffice?

A

Who knows now think about the several hundred node.js applications comprising your sas, offering the number of software components being consumed becomes exponentially large and it gets very hard to keep track of so many components, so naturally they are hidden behind many layers of abstractions.

A

A kubernetes distribution consists of hundreds of components, but it is a dependency in and of itself for a number of services.

A

The abstractions are needed, as we cannot keep track of so many components, and so many moving parts as developers. We focus just on the thing that we are building. We know we need certain top level dependencies, but those dependencies need dependencies.

A

It's turtles all the way down, as they say, the tools we use to build containers are not repeatable and they're. Not hermetic, building a typical container requires downloading many artifacts from all over the internet and several configuration steps, not to mention the volume mounts to the host and copying files from the host into the container.

A

Many of these steps are not repeatable. They are dependent upon the state of the network and the state of the internet and the state of your host machine. You may not be able to rebuild the same container a few months ago from when you originally built it s. Bombs help bring some transparency to the container ecosystem, even highlighting what the known unknowns are and what the unknown unknowns are.

A

They also provide a separation between the source code and the rest of the system required for software to run. You don't need to look at an s-bom until it becomes necessary to look at them.

A

One question I get asked a lot as an engineer working for a rapidly transforming sas company. Why do sas providers need to generate as firms in the first place? They are hosting all of the services they're, not delivering any software to a customer.

A

Here's the thing service providers, don't ship software, but they do take custody of the customer software and provide all the machinery to deploy it and run it. So customers entrust sas providers to take good care of their assets with great power comes great responsibility.

A

The burden of software management now falls on the sas provider. Customers are now entrusting their code to sas providers and expect that sas providers know their infrastructure and their deployment pipelines inside and out. They expect their assets to be secure and auditable.

A

They don't care about what tools and processes are used to do this, but they do care about the risk they will be taking by entrusting a significant amount of their business operations to the sas provider.

A

So there is actually more on the line with regards to reputation, and that should mean more investigation into what exactly gets built and shipped. Can a service provider today be sure about what they use to build and ship someone else's code?

A

They may not need to provide this information to a customer, but they certainly need it for themselves.

A

So why not use static analyzers to generate these s-bombs to understand why, let's take a minute to consider how static analyzers work in general, static, allonizers, look for patterns and make inferences about what's in a file based on some rules and heuristics?

A

Typically, there is some software that reads: a file look looks up a public database for known patterns checks to see if those patterns exist in the file and then generates a report.

A

The security solutions that you pay for may make use of a proprietary database may have an api and a client to query that database and may generate some fancy reports that makes the decision makers feel good about their decisions.

A

Now static, analyzers or static analysis in general works really well on a large container image such as the ones you build like how you would provision a vm or a desktop oss have metadata and they come with a package manager.

A

One can use the package manager to install some system dependencies, including a programming language tool chain. Then one can use the tool chain to install the app's dependencies and finally make some configuration changes for the apps to work. So all along the way things like package manifests, license. Files, documentation and other metadata may get installed onto the container's file system and that's what static analyzers read in order to make their inferences about what packages are installed.

A

But all this metadata makes the container too large and two large containers make it difficult to deploy them and distribute them to reduce the size of the container. We use this thing called multi-stage docker bills, so these multi-stage builds are used to copy out just the artifacts needed for the application to run.

A

We put those artifacts into a minimal operating system file system that doesn't have any of the metadata scanners, except expect to find further. We use tools like docker slim, to ready to remove all of the files. The app does not open at runtime.

A

So by this stage in the container build process, all the metadata about the container build is gone, so this process is great for reducing the attack surface of a container image, but scanners will not be able to detect any useful information about the container image at the end of it.

A

So how do we build transparency into container builds? How many of you thought, let's use each ppf? Well, we don't have to get that complicated.

A

The thing is we don't need to do things like inspect the docker, build logs or watch the kernel or do nmap scans container builders already know some portion of what they are installing and we already have tools that read package manifest or invoke the package manager or the tool chain to list the transitive dependencies.

A

If container builders can create accurate s-bombs for the pieces, they are installing and re-usbs forms created by folks like os suppliers or software packages or any other automation, then they can include them in the container build and distribution ecosystem.

A

What that means is that we have the best of both worlds. We can reduce the size of the container images and we don't have to sacrifice any of the metadata that we collect along the way.

A

So today, I'm going to show you a workflow, that's a good starting point to generate s-bombs during container bills and we're also going to sign the container image and the s bombs and push them to and push them to a local oci registry.

A

So we will be making use of several tools to accomplish this, we'll be using builder to build the container, we'll use turn to inventory the container to generate an s-bom, we'll use ora cli to push the s-bombs to the local registry and we'll use six doors cosign to sign all of the artifacts.

A

Now all of these tools make use of a concept called oci, artifacts and I'll talk about a little about that later. But first, let's go on to the demo.

A

Okay, let's see if we can do this 10 minute demo.

A

I have here a vagrant box which I have provisioned by the way you can provision your own vagrant box by using the tools um this this repo containers with s-bombs that is hosted on github and I'll share. The links to that later so part of the provisioning of this vegan file is vagrantbox is creating a minimal root fs using deb bootstrap.

A

So let me show you what that looks like.

A

So that's a debian root file system and I created that with devbootstrap.

A

Let's look at the containers with s bombs, repo, as you can see, there's a debian tarball right there. That's just a tar of this debian root file system.

A

Okay, we need a local oci registry, we'll use part man to set that up and that's just pulling this docker image and running it as a daemon. So let's give that a whirl.

A

And it's done, let's check if it's running and there we go it's running, let's check if we have any images.

A

Yeah we just have the registry all right. Let's get to building our first image.

A

I have a convenience script over here that I'd like to go over. So what we're doing is we're using builda to build an image from scratch. That's what this is and that'll return a container and then we'll mount this container, I'm using build.unshare, and that lets me mount without root privileges.

A

And I need this mount point, because that's the one that I'm going to point to turn to in order to generate an s-bar and then I'm going to add the debian tarball to the container. That's going to take a little while when it runs but the. But what should happen at the end is that we have now a minimal debian container and we'll commit that container to an image.

A

And then we will inventory that container.

A

We point the mount point to turn and we'll generate an s bomb called debian s bomb and we'll use the format spdx json. So spdx is a standard, s-bom format and one of the like sub-formats that spdx supports is json.

A

Okay, so we have an image and we have an s-bomb and then we will use build-up to push the image to the local registry and then we'll use auras to push the s-bomb also to the local registry.

A

And then we'll use cosine to sign both the image and the s-bomb and we'll do some cleanup over here all right. Let's see how this goes.

A

So this is where we add the debian tarball to the container.

A

A

Now we generate the s-bomb that was done. We now have the image in the s-bomb. Now we push the image, the s-bomb and the signatures, so yay we're done. Okay um now you should see that there's!

A

Oh, I don't have the debian s form here, never mind it is up. It is in the registry and we're gonna use that to create the next container that we're building on top of this debian container. Let's take a look, so this is a convenience script.

A

We'll start off with verifying the container that we built and that should work, and then we will use builder to build on top of this image, and we do this same thing that we did last time. We did build our from and that debian image, and then we mount it because we need that mount point.

A

We need to give that to turn and then we're gonna uh just install some stuff, we're gonna, install python3 and then we're going to commit this image as uh python and with the tag three okay, then we're gonna, download the debian s-bomb and where and before that, we'll verify whether it's signed and once after we verify that it's signed, we pull it using auras, and then we make turn inventory the container with the debian s bomb as a context, and we will use the same format svdxjson and we will produce an s-bomb for just the python bits.

A

We can now push the python image and the s-pawn you'll use build-out to push it and push the image and we'll use auras to push the python s-bomb, and that would include the debian s-bomb and the python s-bomb, because we needed this guy to be able to inventory this guy and then we're gonna sign our new artifacts using cosine all right. Let's give this a wall.

A

Okay, we verified it.

A

And this is building installing python 3.

A

We're storing signatures and we verified our s-bomb and we're done. Okay. Now I want to show you uh the two s-bumps that we had we'll use jq to filter it, because there's a lot of data in it. So let's do jq. I just want packages, that's a list, and I want the name and I'll give.

A

Hopefully it should be there on the system there. It is okay, so those are all the packages that are installed that uh that came that I installed on the debian file system.

A

Now we'll see what that looks like for the python as well.

A

Okay, not that many, because, along with the python 3 that we installed, we also brought in all of these transitive dependencies and that's what this s-bomb contains.

A

All right so cool, that's the end of my demo.

A

Let's see, I have 20 more minutes and I can switch back here. Hopefully there we go.

A

Okay, so there are certainly a lot of gaps that we need to fill out for this to become a viable or at least have like an easier ux than what I've shown, and one of them is using artifact management, that's with oci artifacts, so oci artifacts is basically using the open container initiatives, image and distribution specification to store artifacts in container registries that are not container images.

A

So, in order to support that requirement, there are a number of changes that need to be done on the oci image spec and on the oci distribution spec and there's a working group under the oci that aims to do this. So this this would help us not manage so many tags and instead you'll be able to refer to all of the supplemental artifacts for the container image, with just one tag.

A

The other thing that maybe you've noticed is that the method in which we used to build the container was still using the old method of using the package manager. Apt-Get install python 3.

A

that is not really reproducible, so repeatable bills in regards to containers is basically basically just recovering a file system diff from a cache, and that's not what we want. What we really want is the ability to redo that docker run or build a run such that you get the same uh set or similar set of files at the end of it. Every time you run it, and this is a problem- that's already solved by linux distro tools, and these are good places to look for ideas and inspiration on how to make repeatable container builds happen.

A

Where you can get involved, I have some links to the oci artifact references proposal that steve lasker has against the open container initiative.

A

Please come and join the turn community meetings because we talk a lot about these this ecosystem and how turn can help? You can read up on the update framework, which is the backbone of six door and cosign and, of course, there's a link to our friends at sixdo.

A

Okay, some resources. This demo is hosted on github, it's public. You can check it out. I don't think I'm going to. um I don't expect to have any contributions, but if you have a question or an issue feel free to file a github issue here are a number of blogs, uh discussing oci artifacts and it's a concept that is a little difficult to get your head around, and I hope that these blogs would be useful to you.

A

uh Dan laurenc of cosine fame has also written a blog about the update framework, which is a nice gentle introduction to the update framework. This is also concept that is difficult to get around. I'm still trying to wrap my head around it and then there's a link to the turn repo and the spd expect, and you can talk to me on twitter at these places and these slides will be available.

A

Okay, and with that, that's the end of the talk talk. Thank you for listening. I'm nisha kumar.