Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022, 1 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Securing Your Container Native Supply Chain with SLSA, Github and Te... Laurent Simon & Priya Wadhwa

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Securing Your Container Native Supply Chain with SLSA, Github and Tekton - Laurent Simon, Google & Priya Wadhwa, Chainguard

Supply chain security has been a huge topic of discussion in recent months, and protecting your supply chain has become more important than ever. In this talk, Laurent Simon and Priya Wadhwa will discuss how to practically apply the principles of SLSA to secure your container native build system. They’ll start by covering how to use the in-toto project to create and verify source code attestations. They’ll also do a step-by-step demo of achieving SLSA Level 2 in common build systems like Tekton and Github Actions. If you’ve been wanting to secure your supply chain, but haven’t known where to start, then this talk is for you! Priya has given a related talk at SupplyChainSecurityCon on integrating Sigstore with Tekton. That talk focused on the theoretical integration, and this talk will practically show users how to secure an existing Tekton instance. This talk will also cover other build systems (e.g. Github Actions) which users may be using as part of their cloud native deployments.

A

Hey everyone welcome to securing your container native supply chain with salsa at github and techton.

B

Hi everyone, I'm laurent, I'm a security engineer at google. I work on the open, ssf scorecard project and supply chain security.

A

And I'm priya, I'm a software engineer at chain guard. I work on open source projects like six door and techton, which we'll be talking about today and a bunch of other open source projects as well.

B

So you've probably heard a lot about software supply chain. Security in the news recently, as the graph shows attacks are on the rise and it's becoming really important to start to understand how you can secure your supply chain. So, let's take a recent attack as an example: the npm caller package attack.

B

The attack happened just a few months ago and one of the maintainers decided to add an infinite loop in the package. The code was never changed in the repository itself. The maintainer pushed directly to the npm registry.

B

Around 9 million projects depend on this package, either directly or as a transitive dependency, which is really scary as a developer. It might be intimidating to know where to start when it comes to securing your supply chain. So, in this presentation we'll discuss practical steps, you can take to start improving the security of your supply chain.

A

So the good news is that if you already use tacton or github workflows today, there are a few easy steps that you can take to start securing your pipelines today. So that's what we're going to be talking about in this talk, let's jump right into it.

A

So a quick agenda for today when discussing supply, chain security, it's important to have some metrics or a framework to actually quantify how secure your supply chain is. So that's why we're going to start by talking about salsa, which is the framework we're going to be using to determine how secure our for our supply chains actually are.

A

Then we'll jump into talking about six door, which is a project we use for signing and verification, and at the end of the talk, we'll talk about how we bring salsa and sig stores together to achieve uh higher security levels in both tecton and github workflows, with demos for each of those platforms.

B

So what is salsa salsa is an acronym that stands for supply chain levels for software artifacts. You can think of salsa as a framework or a checklist. Basically, a list of things that you want in your supply chain, depending on how secure you want it to be. Salsa is not a tool or a service. It's more a set of guiding principles.

B

Some key principles about salsa include the following: artifacts should be signed. An artifact should also include metadata, also called salsa provenance about who and how the artifacts are created.

B

Now you might be wondering why we don't just use signatures.

B

And the reason is signatures do not protect against all sorts of attack that happen on the supply chain. Let's take again the example of the npm color attack. As we said earlier, the npm maintainer pushed directly to the npm registry, so the maintainer had access to the credentials.

B

The maintainer would probably have access to the signing key as well. If signatures had been used, salsa provenance solved this problem by including who created an artifact and what source code was used to create it.

B

So, in a nutshell, salsa provenance provide the following benefits. It allows tracing an artifact back to its origin, including the exact source code, repository name, hash, commit and branch.

A

So the salsa framework has four levels ranging from level one to level four. It's meant to be incrementally adoptable and it's a set of security guidelines established by industry, consensus level. One is supposed to be the most achievable level, while level four represents the ideal end state level. Four can be difficult to achieve so levels. Two and three are meant to be stepping stones um on your path to achieving salsa level.

A

Four, in your, so in your secure supply chain, um depending on your build system, you can start making progress towards salsa level, 2 and salsa level 3 today, which is what we're going to be talking about.

A

So a quick overview of salsa level, 1 salsa level 1 is the most achievable level with only two requirements. The first is that your build process is scripted, and the second is that you have some kind of provenance available, some sort of metadata about how your artifact was generated, and maybe what also went into it.

A

Salsa level two builds on top of salsa level 1 and has some additional requirements. So this is where we start getting a little bit more secure, salsa level. 2 requires that your source code is in a version control system like github, hopefully you're already doing that.

A

It also requires that your builds are built in a service and that providence is service generated provenance should also be authenticated, which means it needs to be signed so that end users can verify that the providence for the artifact they're looking at is coming from where they expect it to come from.

A

So now that we've talked about the framework, we're going to be using to evaluate how secure supply chain pipelines are, how are we actually going to start implementing these salsa requirements?

A

So this is where six store comes into play and both of our uh demos, involving github and techton later in this talk, are going to be using six door under the hood.

A

So six store is an open source project aimed at making signing and verifying software free, easy and accessible. And how does six store accomplish this?

A

It provides a few tools and services which are available for public use and the first tool that it provides is called cosign, which is a cli tool and library used for signing and verifying software. Artifacts, this could be things like oci images, s-bombs or provenance.

A

One of the services that cycler provides is called focio, which is a certificate authority service that issues short-lived code signing certificates.

A

It also provides a service called record, which is an append, only transparency, log for storing signatures, certificates and providence. But why do we use six store instead of traditional signing methods? What kind of problems do the six store actually solve?

A

So the main benefit of six store is that it eliminates the key management problem. So we have like a diagram of a traditional signing situation here, where we have a maintainer who holds onto a public private key pair signs their piece of software and then distribute distributes it to an end user. The end user is responsible for verifying that software before they actually use it, but there's a few issues in this scenario.

A

First, it requires that the maintainer holds on to the private key and keeps it safe for long periods of time, potentially years on end, which can be really difficult. What, if that private key get lost or ends up in the wrong hands? It's really hard to. Then revoke old signatures.

A

It also means that the maintainer is responsible for distributing the public key to end users figuring out how end users can get their hands on it a lot of times instead of verifying signatures and users just choose to use the software without doing the verification, because it's not easy to do the verification.

A

So this is where six store comes in. It solves this key management problem so that maintainers no longer have to keep track of private or public keys. But how does this work so? You can see we have this updated diagram now, which includes the six store tools and services that I had mentioned earlier.

A

So, instead of holding on to a private key pair, the maintainer actually just has to request a certificate from the full co certificate authority. This certificate is a code signing certificate and contains and is bound to, the identity of the maintainer. In this case, it's the maintainer's email address the certificate, also embeds the public key that was used to sign this.

A

uh That was used to sign sorry that is associated with the private key that was used to sign the artifact, and the signature is bundled along with a certificate and stored in the record transparency log. This is really convenient because the log is public. So when the end user attempts to verify the software artifact, they can use the cosine cli tool which, under the hood, will retrieve the correct entry from recore and do all the verification for you all.

A

The end user has to do is decide if they trust the identity that is embedded in the certificate.

A

So the nice thing about focio is that both people and systems can request full co certificates. People can prove their identity by signing in with an email address, so this would be like a sign in with github or sign in with google type flow that you might have seen on different websites. Yourself systems can also authenticate and request full seo certificates, so workloads can use the spf specification and there's also support for using kubernetes service accounts and github actions invocations.

A

So we're going to be using these last two in our demos later on in this talk, so how do we bring salsa and six store together to actually start securing some build systems so we'll take tecton first and we'll talk about github after so what is tekton? This is a big question and there's a lot to techton. So in the interest of time, I'm just going to cover the important bits for today's talk, but I would highly encourage you to go. Learn more about tecton if you're interested for now.

A

All you really need to know is that tecton is a continuous delivery system built on kubernetes. It leverages custom resource definitions to run tasks and pipelines on your cluster.

A

You can kind of think of tekton, just as a ci cd system, where you would do anything you would normally want to do in cicd, including running unit tests, running integration, tests, building some sort of software artifact and then storing it somewhere. For your end, users to be able to use the basic unit of techton is the tacton task, which is you can deploy your tacton task to your cluster. um If you look on the right side, we have some sample yaml for a typed on task.

A

If it looks like a pod or it reminds you of a pod specification, that's because the tekton controller is responsible for turning this task into a pod so that it can actually execute in the cluster.

A

You can instantiate this task with the tkn cli tool to turn this task into a task run. Task run is actually executed in the cluster and we'll be doing this in the demo later on in this talk.

A

So what do we know once a task run has has completed? Well, there are a few key pieces of information. First, we know whether the task run was successful. We also know what steps we're on and exactly what arguments were passed in and then finally, we know what artifacts artifacts may have been built in this task run. This is really useful when trying to achieve certain salsa levels.

A

uh We know exactly what was run so we can put that into our salsa providence and we also know what artifacts were built, so we can generate providence for those artifacts and also try to sign them.

A

So how do we do this? This is where tecton chains comes in. You can think of captain chains as a supply chain, security manager for tekton. It leverages sigstor under the hood to find artifacts that are generated by tecton and also generate signed provenance.

A

Here's a little diagram I drew of how this works. Basically, you have your cluster running and you have techton pipelines, which is the core techton project running in your cluster. When you start a task run, the techdown pipelines project is responsible for executing that task run. Meanwhile, tekton chains is observing all the task runs in your cluster and once it sees that the attachment has successfully completed it'll, try and extract any artifacts that it recognizes from that task. Run it'll generate signatures for that artifact generate provenance and then store those two things.

A

Alongside the artifact in whatever storage back-end you have configured, this could be an oci registry or a gcs bucket. You can also configure various types of signing keys to actually sign these things with tactile chains, including kms keys, public private key pairs that you might have set up yourself and fulso, which we had talked about earlier.

A

So uh I'll do a quick demo of everything we just discussed with tekton. So let me share my terminal.

A

So, in the interest of time, I've already created the cluster that I will be doing this demo on um it's an amazon, eks cluster and the most interesting thing about how I created this was I passed in this with oidc flag. This is basically going to allow my kubernetes service account to authenticate to vulcio and request a code signing certificate for the piece of software I'm about to build.

A

So I already have tecton pipelines installed in this cluster, installing chains to add in supply chain secure the supply chain. Security manager is just a simple control, apply command. I've already run this, so I won't run it again, but um the last piece of configuration we need to do is tell tacton chains exactly how we want to generate our provenance and sign it. So there are four commands here.

A

The first one is telling chain that we want our the format of our provenance to be the in total format, which is what's recommended uh by the salsa framework.

A

We want to store our signatures in an oci registry and we've also enabled the transparency log, which means we want to store our signature and certificate bundle in the record transparency log we've enabled signing with full co, so we don't actually have to set up our own public, private, key pair or point to kms, we'll just request a certificate and uh that's how we're planning on signing our artifact.

A

So, let's take a look at the uh the artifact, we're actually going to build we're going to be using canaco to build an oci image and push it to a registry. So this is the task that we plan on running there's just a few parameters passed in including the name of the image. We want to build um a few results here, which is how we're going to tell pekton chains that an artifact was built.

A

The first result is the digest of the image we're going to be building, and then we have the url or the name of the image that we're going to be building.

A

There are three steps in this task, so the first one basically just creates a simple little docker file, which we want to build. The second one actually does the build and push of the container image, and the last one writes the name of the image to our result, so that we know what image was built.

A

So we can apply this task to our cluster.

A

Taking a second amazing- and we can kick it off with the tkn cli tool.

A

So while this runs, um I basically passed in and I've stored already stored this image name as an um environment variable or I have not- um let's try this again, I'm gonna delete this task run that I just created.

A

Let's start over.

A

This is the image that we want to build and let's kick off another task run so now we're passing in the image name that we want to build and we're running it under the service account I created earlier called ecr pusher, which has uh permissions to push to my ecr registry.

A

If we take a lot a look at the logs for this image, we can see that we're running through those steps, adding the docker file, building and pushing the image, and we have written the results. So this task screen has completed successfully.

A

If we take a look at the task run using that same cli tool that tech tom provides, you can get a little bit more information about the image that was built.

A

So we have the image url, which I'll just store into an environment variable and the digest.

A

So we should now be able to run this image.

A

It'll pull it down and we can see that it has printed out hello world, which is exactly what we expected, but now we want to make sure that the image that we built was signed and that provenance was correctly generated. How do we do this? The nice thing is that under the hood, tecton chains has already realized that an artifact was built attempted to sign it generated, signed provenance and stored it in our oci registry as an end user. It's really easy to verify that this happened using the cosine tool.

A

So at the moment, transparency, log and full c of support are experimental in cosine. So I have to set this experimental flag, but all we have to do is a simple cosine verify and then the name of the image, and we should see that verification is successful.

A

If we pipe this into jq, we can get a little bit more information around the certificate and the signature, and you can see that the subject in the certificate is a service account that we were running under and that the issuer is uh points to my eks cluster that we ran in. So it's really cool. We didn't have to worry about keys. We didn't have to do anything everything sort of just happened under the hood, because we already installed tecton chains.

A

We can also take a look at any providence that was generated with the cosine verify attestation command.

A

And looks like it was successful again we see our certificate subject, our certificate issuer url and we can take a look at the provenance that was generated by decoding it and typing it into jq, and you can see that we have our provenance now in the salsa providence format. um The subject is the image that we just built, and you can see that all the steps that we ran um exist in this provenance, so we know exactly how it was built and also when it was built.

A

Let's transition back to the slides.

A

Cool, so you can see that by just by adding tecton chains into our existing tecton deployment, we were able to achieve salsa 2., um assuming your code is already in version control. Our builds are scripted and tucked on this build services. Techton providence is available, authenticated and service generated since it's been generated by tecton chains, so tekton plus tecton chains equals also level 2.

B

uh Yes, so now, let's talk about github, workflows and github actions, so many projects are natively developed on github. So it would be great if we could also generate salsa provenance on github, and this is the motivation behind this presentation.

B

So, on github the standard technology that you run, that you use to run ci are github workflows and github actions, whether you want to run simple unit tests or end-to-end integration tests or maybe release pipeline. You can do all this using using github, workflows and actions.

B

Typically, you define github workflows by creating a file under github workflows on your repository. Github workflows are very powerful and you can run arbitrary commands. You can also define so-called triggers to start your run. For example, you can start to run on an incoming pull request or when there's a new push to a branch or when a new tag is created. On your repository.

B

And here's a simple example of a hello world: github github workflow: all it does is print hello world. Every time, there's a push to the main branch.

B

One powerful feature of github workflows is the ability to support isolation between jobs, so you can define different jobs in the same workflow. Here we have two jobs, one in red and one in green called pr and secret, even though they're defined in the same workflow those jobs run in isolation in red. We have a job that runs an incoming pull request.

B

The pull request may contain malicious code might not have been reviewed, so we want it to be isolated from other jobs in the workflow and in green we have a sensitive job that accesses credentials or other sensitive information like a secret.

B

This could be the job that publishes container images to your registry and needs a secret to log in the registry.

B

So what I'm presenting today, uh using github actions and github workflows, meets the salsa level three requirements. So let me tell you a little bit more about what those requirements are.

B

The first requirement is that the build must use an ephemeral environment.

B

The motivation behind this is that it makes it harder for an attacker to gain persistent persistency to your release pipeline if they work again to gain uh if they were to be, if they were to to attack it on github, each job in a workflow runs in a fresh vm, so we get that that requirement for free.

B

A second requirement is isolation, meaning that the provenance generation must be isolated from the maintainers.

B

So what this means is that the maintainers that are building an artifact shouldn't be able to interfere with the provenance generation, and this takes us. This leads us to the the last requirement, which is that the provenance should be non-forgeable, meaning that maintainers shouldn't be able to lie or tamper with the provenance information.

B

They shouldn't be able to lie about what source code was used, for example, to build the artifact.

B

So, let's see how we can generate salsa 3 provenance on github using an example. All you have to do as a user is define a workflow and call the trusted builder.

B

Once your the trigger, once the the run is triggered, the execution is handed over to the trusted builder. Here on the right. The trusted builder is hosted in a different repository in our example. It's called github.com trusted builder and the trusted builder is not under the control of the original workflow on the left and therefore the workflow on the left cannot tamper with the execution of the trusted builder.

B

So we can do anything we want in the trusted. The trusted builder and the first step we take is building and publishing a container using code.

B

And then, in a second step we can generate the provenance sign it and upload it to a registry, and here uh we can just use cosign api and then the six store tooling. To do that.

B

So something I want to point out is how signature generation uh happens. I think priya already uh talked about this, but I want to mention it again uh in the example that you saw, we didn't have to provide any sign-in keys, and this is a key feature of generating salsa pronouns using six store.

B

Key generation and management happens transparently through workflow identity. So when the trusted builder runs, it is provisioned with a unique x509 certificate that unique, uniquely identifies it, and for this we use oidc as priya mentioned earlier. This certificate is then used to sign the provenance.

B

So during provenance signature verification, we can inspect the certificate and learn who produced the bill.

B

Just to make things even clearer, I want to compare the web pki and the salsa pki that we've built using six store on the web. A ca may be verisign and when you visit a google website, the x509 certificate that you get during the tls handshake will have the google identity, such as 3w.google.com in the salsa pki, with six store. The ca is six store and when you verify the provenance, the certificate contains the identity of the builder.

B

So during verification we can inspect the certificate and identify the builder by name, and if we trust the builder, then we can trust the sun provenance and its content.

B

So it's pretty easy to verify. uh Provenance we've built a tool called salsa verifier that can help you do that and under the hood, it's using the cosine api. All you have to do is pass it. An artifact oci of your container and tag.

B

And one one option that is mandatory for you to pass is the source code that you expect this container to have to have been built from in our example. Here it should come from github.com, orig, origin, repo.

B

Optionally, you can also pass a branch which, by default, is set to main.

B

And then you can also pass a tag uh if you expect the the uh the container to be built from a a particular version or tag of the repository.

B

So as you as, as priya mentioned before, uh thanks to to seek store, we don't have to handle public keys and you can just pass the identity of the repository. You believe the container should come from and the verification happens automatically for you.

B

Okay, so let's have a quick demo.

B

Let me share my screen.

B

All right so the example we're going to take um is hosted uh on github, it's called kubecon minus eu22, it's a simple hello world which will print a random random uid. So we're gonna take this gopro. This go file and package it up as a container using co doing this is pretty easy.

B

As a user, what you have to do is define a workflow, a github workflow. Here it's called code minus publish.yaml, and then here we define different jobs. You can ignore this part. um You can look at it if you're interested, but but the most interesting part of this workflow is this this spot here, the build part all we do here is we basically call a trusted builder, which is hosted under this repository, and then we pass a bunch of arguments to cool.

B

So here we're saying: build a multi-architecture image and tag the container with fact 5 and tag 6..

B

We also pass environment variable that we expect such as code docker ripple, which is just just the the docker account where we want to publish the resulting container image and we pass the password to be able to publish to the registry.

B

Then all we have to do is run this workflow, so you can go to the action tab here on github and here on, the right run, workflow and we run the workflow.

B

So it's going to take a few minutes for this to complete. So uh I've already run uh the uh the release a few times to save up, save up, sometimes.

B

And here on the left, you can see all the different steps that were taken by the builder to build this image and publish the to publish it and sign the provenance, and this is the resulting container image that we got. So, let's try to verify the problems.

B

Okay, so let's first verify that we can run our container image.

B

Okay, so this seems to be working. We have our yellow hello world. We can also run it by using the hash, which is the same thing. Okay, so at least our container image is working. Now, let's verify the problems.

B

Okay, so salsa pronouns was verified and docker the docker container is printed. That's that's what it resolves to these tactics now, let's take this command and change the expected source repository, let's say 23 and as expected, it doesn't pass the verification because the source repository doesn't match.

B

As we said earlier, we can also pass it a main branch oops.

B

Which verifies and if we were to pass a different, a different branch, it would fail.

B

We can also print the provenance and, as expected here, uh it looks very similar to what priya presented with tecton. Here we get the source origin repository and the commit hash that it was built from.

B

We also see a list of environment variables that were passed to co, to build and also the command that was used by co to build the container.

B

There's additional information about the environment used by the trusted builder uh architecture was amd64 running an ubuntu on an ubuntu machine, and we see that the trigger event was a workflow dispatch, which is just saying saying that I clicked on the button to run the workflow.

B

We also see up here that the reference is the main branch of the original repository. So we have all the information that we need.

B

So, let's quickly go back to the action run, it just completed.

B

And we can see again that this build is reproducible and we get the same container image that we verified.

A

Awesome that was super cool. um Let me present again.

B

A

B

Github actions and github workflows gives you salsa level 3. If you use a trusted builder.

A

Thank you. Everyone for attending, um if you have any questions about this talk, feel free to reach out to either of us I'll, be there in person. So look out for me um and yeah. Thank you. So much for attending.

B

Thanks everyone.