Cloud Native Computing Foundation CloudNativeSecurityCon 2023 - Seattle, 3 Feb 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Spicing up Container Image Security with SLSA & GUAC - Ian Lewis, Google

Description

Don't miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from 18 - 21 April, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Spicing up Container Image Security with SLSA & GUAC - Ian Lewis, Google

Understanding and verifying the content of images that you deploy in production environments is difficult and error prone. Images could be built in an insecure environment, by a malicious actor, or include dependencies that are insecure. Users often don't have enough information to determine if images are trustworthy. Two new tools can help; Supply chain Levels for Software Artifacts (SLSA), and Graph for Understanding Artifact Composition (GUAC). In this talk attendees will learn how to add SLSA provenance metadata to their container images and strongly link images back to their source code on multiple build systems including GitHub Actions and Google Cloud Build. We will also cover how to verify images and their metadata before use; both when running locally and when running images in Kubernetes. Using policy engines like Kyverno and Sigstore policy-controller we can verify an image's source code repository, builder identity, build entry points, and more to protect production environments from malicious images. Finally we'll discuss how to understand your image's supply chain using GUAC. We'll discuss how we can combine SLSA with GUAC to better understand the contents and build provenance of your images from the base layers on down.

A

Okay, I think we'll get started, uh so this talk is on splicing up containers from image security with salsa and guac.

A

My name is Ian Lewis I'm, a developer Advocate at Google, Cloud, I focus mostly on container and supply chain. Security and I am currently on loan to the Google, open source security team or the ghost team, as it's colloquially only known working on salted tooling.

A

um So I said that this talk is about spicing up container image security, but I'll be pretty mild. No hot takes or anything like that.

A

uh I want to start by I, guess uh kind of level setting a little bit um and give a little bit of context for like why some of the things I'm going to be talking about are important like why we're doing these things in the first place. So hopefully, this kind of like gives you an idea of like why some of these things or how some of these things are actually solving problems in the real world um and uh because it's really easy to get lost in the details. But essentially you know software supply chain.

A

Security has been a Hot, Topic I guess in security space for the last couple years, and this is basically because attackers have shifted their focus towards the software supply chain and part of the root causes of some of these are that that development in CI systems are essentially um not treated like production or are not.

A

uh You know given the same level of love in terms of like security hardening as production systems are, and so there's been a culture of essentially like you know, developing civilized systems are essentially YOLO and, like you know, production is where we actually Harden things, um and that's not actually the case, because there's uh channels to get software or malicious software into production through these CI and development systems.

A

um Another aspect of this why it's a a good Target for attackers is that there's lots of places that attackers can insert themselves.

A

um Apologies for this slide is a little bit harder to read, but, like essentially, attackers can uh slot themselves in essentially anywhere in the development life cycle so from source to uh you know: source code, the developers, development environment through the build systems through to the package managers and Distribution Systems you use for deployment, as well as all the deploy the dependencies that go into the software that you're building today we'll be like pretty much primarily focusing on the build aspect of this, so like your CI systems and the build systems um and having traceability from there into your production systems.

A

um But it's good to know that there's a little bit of context here for like why we're uh talking about these things and another like kind of overarching theme, is that there's a lack of information about the the uh software that we consume. So as we develop these things we and we consume software, we download it.

A

We run it, there's a lack of understanding about like where that software came from uh how it was built, who built it and being able to verify that and make sure that it was actually built by the people we think it was built by.

A

So this comes out in a lot of ways right, so you have something like you know. How do you know essentially where your software came from right like we have, we can download artifacts, but and we can download them from exam, for example, from a GitHub repo um or something like that.

A

um But how do you know that that artifact was actually built from the source code in that repo there's? No real guarantees for that, and similarly like this is even worse in the cases of where the software is being distributed on a totally different platform, for example, Docker Hub or um you know, on GCR or on like other container registries.

A

So, for example, right, like you'd say like is Ian Lewis on Doctor Hub, the same Ian Lewis as uh the Ian Lewis on GitHub, there's, no real linkage there uh that you can be sure is is something you can verify.

A

um We won't actually solve that particular problem, uh but the the that's kind of one of the root root problems for this, but, like also like, is the actual image say like Ian Lewis Foo, that's on Docker Hub, the same or built from the source code. That's in the Ian Lewis Foo repository on on GitHub, for example, and how do you know that? How can you verify that? That's actually the case so essentially like? How do you know like how the sausage is made? How like what goes into the food that you're eating?

A

uh How do you like kind of maintain some of that hygiene? um That's a pretty difficult thing right. So, for example, let's say like that we have a software repository right like we have a uh a GitHub repository. We have the salsa and guac demo right a repository here, and we want to be able to run this as a container right and but we're we're going to first take a look at the source code. We've kind of found this thing out there that looks like we're it does what we want it to do.

A

Right, we'll look at the source code here, we'll say like okay: this is going to like pronounce some version, information right, it'll print out the git uh version and uh what tag it was built from and things like that. That's cool! That's what I want right and um we can even kind of look on Docker Hub and see that we found a an image there. That's built by a guy named Ian Lewis and it looks like it's the same name.

A

It's got like a link to the to the GitHub repository and the readme. um You know we can look at the tags. The tags like seem to match up. We've got like you know the same sorts of tags as we have on the releases over here.

A

um We can actually click on this tag and see like some of the image layers right. If we look at our Docker file, you can see like okay. This is a a um a two-step build, so we can only see the the second half of this, but we can see that these steps here, you know, line up with the steps that are uh that are present in the docker image.

A

Okay, this, like pretty much, looks good right like this is something that like looks like it's linked to this repository. You know it was built from this repository um and all that, but if we actually like go and try to run this- and this is like just a Docker- run command to to run it- you know this is essentially like building and running code.

A

That's like completely different from the code we just looked at, um and this is because, like we have no linkage between the source code repository and the actual image that was being that we're downloading and running um so, for example, that, like the Ian Lewis on Docker Hub, could be a completely different person like if we don't verify that.

A

Also like the the names don't need to necessarily match up like somebody could just make up their own uh account on Docker Hub um or, for example, like the account credentials could be, have been leaked or gotten out on Docker hub, for example. So um you could have like a personal access token that was Excel traded or it was compromised and that allowed an attacker to upload an image.

A

So this is kind of an issue that is fairly pervasive, like you're not able to trace back to the source code.

A

So how do we kind of like solve this problem? So one of the ways that we solve this problem is by doing something called the software attestation right.

A

So what we're going to do in plain English when we say something like is an attestation or we attest to something we're basically asserting that this is true right, like if I say something like my name is Ian Lewis, like I, miss I'm attesting to the fact that my name is Ian Lewis and then ideally, I would have something that verifies or that, like that, somebody else could use to verify like that.

A

I am actually Ian Lewis or that might the statement that I said is correct right, like my passport or like some statement that I've actually signed with my own signature right. So in the case of software, what we're going to do is we're going to actually use some metadata which we'll call provenance that comes from that describes, information about how the software was built and where it was built and we're going to combine that with cryptographic, signing and identity in order to create a software access station.

A

So essentially the identity is going to be either a person or some kind of workload, for example a machine identity. What people call a machine identity, but essentially this is an abstract identity, and then you combine that the identity then signs this this metadata to say that it's asserting that it's true and then that attestation can then be verified by the consumer when they're actually consuming the the software.

A

So one of the the big things is that people say like kind of throw these terms around a lot, but like I like to think of attestation as being something that you can actually verify in a strong way so like it needs to be signed by somebody, um and it needs to make sense right if you don't have an identity that says like hey. This is the thing that signed it like.

A

Who do you who's attesting to what like that becomes like kind of an issue if you're just providing metadata, for example, and you're, not signing it or you're just signing stuff, but you have no kind of like understanding of what the identity was or where that came from right.

A

So, as kind of a building on that topic right, we have the idea of kind of software attestations um but who's kind of, like in the context of building software and deploying software who's, actually um kind of attesting to what and like what are you actually going to be attesting to um so. Basically, what happens is when you're going to do a build. uh The build system will essentially attest to the fact that that build was run, that it built that uh um or ran that build right.

A

So the build system is essentially some sort of machine Identity or has some sort of machine identity, and then that is used to essentially make the attestation which says that I ran this build I used this source code, I got this artifact out, and so when you consume the artifact, you can verify that. Okay, this is the same artifact that was built by this and is described by this out of the station.

A

So you can verify that that artifact was actually built by that build system at a particular time, using particular source code, and we make sure that we are cryptographically signing this so that we can sign. So that we can verify this in a very strong way that we actually know we don't have to like do signature like human level, signature comparisons or anything like that. It's not a it's a very objective rather than subjective process.

A

um Another like issue here is that we need to make sure that the build System since the build system itself is attesting to running the build that the build itself isn't actually doing the attestation right. So it in the sense that um we need to have the build system needs to have some level of separation or isolation from the build itself, um because effectively you get like a softer equivalent of a conflict of interest.

A

If the build itself is attesting to the fact that it was run right like it could essentially make up whatever it wants to at that point right, so so the build system it needs to be trustworthy right. We need to make sure that we trust the build system itself, um and so, if the build itself like could be running any kind of software, uh we need to make sure that the the build system that we trust is actually the thing. That's making that a station.

A

And then we're going to combine these the different ideas right together, along with Sig, store uh and you've, heard a lot about six store, I'm sure during the conference I'm not going to go too detailed into it, but effectively. A six-store is a a set of tools that allow you that we're going to use for Keeley signing.

A

So um what we do basically here is instead of having using a key that we've like that, we have to manage, and that is long-lived and that we have to store somewhere and maintain access to we're going to use a quote-unquote keyless signing.

A

um And what that allows us to do is take essentially an identity, the identity from the uh the machine identity from the build system and use that identity to Mint a new cert or to use the uh use a certificate Authority in this case, folkio, which is the Sig store service, to create a new certificate that is short-lived and is only going to be used by the the build system. One time in order to start to sign our provenance.

A

uh So that basically allows us to take the key sign the provenance and then throw away the private key. We don't need it anymore, because we're only going to use it one time and then the public key. What we can do is upload that to another service, six door service called wrecker uh with uh that we can then allow like create a log of when the key was created and then allow that allow us to also retrieve that certificate later.

A

When we need to verify so effectively, we don't have to really keep the key around the private key around. We can keep the public key. We can put that in a log service that we can then use to pull down and verify it, and so we don't actually need to store the keys anywhere in our actual build system. We can essentially be keyless uh in that sense.

A

um So there's a lot. uh There's a lot been a lot of talks, but like one uh talk by uh um colleague on the ghost team, uh Hayden was really good, and so, if you are really interested in a much more of a deep dive into the six-story ecosystem and the services I recommend you look at his uh the recording for his talk.

A

So, okay, so one now that we've kind of got some of the concepts out of the way.

A

um Let's talk a little bit more about the concrete, like way that we're going to implement some of the uh some of these ideas, and so what we're going to do is like apply some of these ideas or one application of these ideas is in the uh supply chain levels of software RTX. So this is the salsa framework and The Souls of framework is essentially a framework that is used to uh Define a set of requirements or levels that Define like an increasing set of security requirements.

A

So right now it has a like levels defined one through four, which are progressively more secure, more hardened levels of security for a build system, um and these are make it so that it's like increment, incrementally adoptable. So um each level has a set of requirements and those get more stringent, as you go up the the uh the levels.

A

This also said some things like common terminology. So when we talk about provenance, we kind of know what we're talking about. uh When we talk about like signing and keyless, we kind of know what we're talking about like I, guess, keyless isn't really necessarily defined by salsa, but um it defines a bunch of common uh terminology that can be used for uh for build systems and for supply chain security.

A

um It also, as part of the framework, defines a provenance format, uh which is essentially just a Json format that you can use to define metadata that describes the build uh and what how the build occurred.

A

Okay, so those are a lot of kind of like abstract, Concepts and Frameworks.

A

Now, let's like kind of get down into like actual implementation of some of these ideas, um and so what we've done is we've started a project called the salsa GitHub generator project, and what this is doing is utilizing some of these Concepts, like keyless signing uh Salsa Frank like salsa provenance and some of the features of GitHub actions in order to achieve isolation for our build between the build system and the build itself, and so we have a number of different uh workflows that we've implemented, and this is using GitHub actions, reusable workflows and the reason why we did.

A

That is because this gives us a level of isolation from the actual build part of the system. So when you build something, you can separate the build uh process and the generation and signing of provenance into separate build jobs that are effectively isolated from each other right to run in different VMS, um and so what we have. As part of this, we have a number of reasonable workflows. One is the language agnostic kind of generic generator. We call the generic generator.

A

This is like a a workflow that you can use to generate provenance uh mostly for file based artifacts, so things like binaries things like s-bombs. Maybe that type of thing we have another workflow called the container generator, which we've just uh essentially G8, and so we are basically saying that this is stable um and this is also uh used for container artifacts, and this is partly to make it so that it's easier to integrate with all of the other Sig store, tooling.

A

Around containers, um as well as like, have support for things like uploading, the r, the uh attestations to The Container registry. Alongside of the containers themselves, so it's easily, you can easily get the uh the provenance. We also have a go builder for go projects which allows you to build and generate prominence kind of in one step.

A

We also have a project called the salsa verifier project, which is used to verify the provenance that's generated by these Builders. So this uh verifier project is essentially a command line tool which will verify the provenance for trusted builders that um that we essentially have identified as being well built, and so some of those are the ones that we've built ourselves and have a full kind of like threat model.

A

So like these, uh the workflows that we built, as well as a Google Cloud, build and we're looking to support other uh CI systems, as we have more ways of generating provenance safely, um and this also allows us to do things like verify. The source code is the source code we expect and that it was built with the tags that we expect and for with the builders that we expect et cetera, et cetera.

A

So let's take a little bit of a look at what that looks like right. So let me blow this up a little bit.

A

um Also, one thing that we can do as well with containers as we upload them or we when we generate provenance, we can actually look at some of the like inspect the provenance and take a look at what it looks like using some of the um six door based tools so as a way of just kind of looking at at this, like. Let's look at the provenance uh and the address station is for the uh container that I just built or that I just ran. That was like a malicious container right.

A

So if we look, we can use the cosine tree uh command to actually run to actually check the container registry and look for added stations. So we can actually notice that the kind of malicious container that was here like actually has an attestation itself right. So this is a malicious container, even though it had, and but it has like an attestation, it's just uh when we actually look at what the attestation contains.

A

So in this command, we're going to see, use the cosine download at a station command in order to download from this this image and then we'll look at the internal payload or the internal Json, for that was for the prominence that was generated.

A

And so this is the the actual Json that was generated, um and so we can look at some things like the subject. This is a what's called an in Toto statement, and this defines you know somewhat of like subject: outbreak, verb kind of a thing where, um but the in this case the object is actually the subject but like what we have here is you know this?

A

The subject is the thing we're talking about so in this case the container image um and here's the digest for that, and then we can look down and see uh more inside. This thing called the predicate.

A

We can see the Builder that was used uh to build it as well as information about what source code was used and uh which workflows were used in GitHub, and so we can see here that if we look at the actual source code that this came from, this is actually not the source code uh repository that we that we expected it to be coming from. It's actually a different source code Repository, and so we can use things like the uh salsa GitHub generator or the salsa verifier uh in order to run the assault.

A

The the verifier to verify this, and so we'll look and we'll try we'll say: okay, let's verify the this particular image and we'll expect that the source code, Came From, My, Salsa and guac demo repository at this particular version right, and so, if we run that this is duplicated a little bit. But uh we can see that when it tries to verify it that the generated uh artifact does not have the expected Source right. So the expected source is social block demo, but we got uh the fuldia repository.

A

So in this case the verification failed because we were expecting a different repository than the one that was actually present in the provenance.

A

So if you want to like actually see what it looks like to verify, one that actually does verify um effectively like this is going to check the provenance like check the signatures on the provenance and check the the uh the fields in the provenance uh for the source code and the tags. In order to verify those.

A

um Those are correct, and if those are correct, then we can. uh The verification will pass and then we can go on to use the container as we normally would.

A

So it's also kind of worth noting that, um as part of six door like in using the Sig store, tooling um and the uh and salsa provenance, we're able to integrate really well with a lot of other tooling that supports six store and solsa provenance. So tools like kiverno and the Sig store policy controller are two uh two tools that are used in kubernetes to verify prominence before you actually deploy containers into kubernetes.

A

So these are used as what's called an admissions controller, so that admissions time when you try to create a pod, it will actually verify the provenance for that image against the policy that you've defined inside of the cluster and in our salsa GitHub repo. We have a number of examples about how to use those, in example, policies that you can use to verify containers um so effectively.

A

What the admissions controller does is it verifies the prominence and it will reject the pilot outright if the the verification doesn't pass, um and these can be used with a salsa GitHub generator because, like our uh workflows, use this kind of uh use the format the same formats that are accepted by uh by kuverno and sigster uh policy, controller.

A

Okay, so now that we've kind of like gone over like how you can kind of generate provenance and uh how you can kind of verify that before you actually run your containers, um there's also there's still like, we haven't totally solved. The problem of you know understanding getting information about the artifacts that we're consuming so the uh so not just like how it was built. But, like you know, what's inside of it, um do these things have access Stations, don't they like, which artifacts do we have?

A

um How do they relate to each other uh things like that, so one of the tools that we're also developing is a tool called guac, which is a kind of graph based tool that is used to kind of ingest, metadata and information about artifacts and then allow for querying and understanding and visualization of the relationships between those different types of artifacts.

A

um So this is really useful in a number of contexts. You know essentially, like you have artifacts that are node artifacts and metadata. There are nodes, and then you have like relationships between those nodes as they're linked. So in the graph uh you would have like nodes that are artifacts and metadata and the edges or these relationships that go between these.

A

So, in our case, like so, guac can really apply to a lot of different uh areas of discoverability and auditing and so kind of across the life cycle, I, guess of a uh vulnerability like so from the kind of being reactive. uh How are you affected by something that actually happened um to trying to Implement safeguards into uh your system?

A

So like adding policies based on this information and then also on the proactive side of like trying to understand the wider set of security implications of like different uh artifacts, like which areas or which artifacts are have higher risk profiles which need more attention or need to be prioritized instead in terms of like improving their supply chain security?

A

um That sort of thing so I'm going to take like a little bit of a uh and I'll, also say that one of the uh folks that is working on the ghost team working on guac uh did a keynote earlier talking a little bit more about guac, so I definitely recommend checking out the recording of that as well, um but effectively like guac is a system that will ingest um artifacts and give you a graph kind of representation of those artifacts which you can use to kind of visualize and understand the uh the tools right now it's effectively like it's a little bit of an early uh early.

A

It's an image development life cycle. So right now it's mostly just ingestion and then you can do queries on the database directly. So in this case I'm using neo4j to directly query and visualize in a database. But in the future there will be much more targeted type of apis for developing policies, and things like that- um but we can see here here- is the here's, a set of metadata that I've ingested for the images that I was using earlier.

A

So we have our uh the image that we uh that I built, that is, that is good, and then we have the image down here. That is uh the malicious container, and so like say, for example, in a reactive context, where we figure out that this one malicious container is in there. How did that get in there and what what happened?

A

um We can actually see that this container has um had was built by the same Builder as our good container and that it has a software attestation. So this orange node is an attestation node, so we actually built it yeah. It was actually built with the same Builder and generated as a station in order to try to fool uh fool us, but it was built from a different repository. So that's one key knowledge Point here is that we, even though uh we could verify the key signature and that it had to add an added station.

A

We really still need to check the repository information, the provenance metadata itself, in order to make sure that that's what we expected was.

A

So we can see here a little bit about the relationships here and how this this uh artifact was built. These orange these red nodes over here are part of a uh information that was ingested from an s-bomb, so this was built from an image, a container image and we used the the sift tool to generate an s-bomb which tells us the different uh golang packages.

A

These are not quite as relevant for for our in the this reactive use case, but we can see that uh they have uh at least one kind of common dependency within the two, uh the two containers.

A

So there you can kind of see like an overall set of um relationships between the different types of metadata about our artifact.

A

So in we just used like kind of more reactive context for for guac, but in reality what we want to be doing is, like you know, quote-unquote shifting, left I'll, maybe take a a second or two to let you guys like check off your bingo cards, um but uh really what we want to be doing is uh using this in a much more proactive context, be able to like look at the metadata in a much more proactive way and better understanding.

A

Overall, our um you know this: the situation of our supply chain uh and our um the artifacts that we're building and using uh so with that I'll um end and I. Think I have a couple of just a couple of minutes for questions if they're already, but we have a blog post that we released uh I think today uh on the uh GA in the container workflow.

A

If you want to find out more about salsa, you can check that out at salsa.dev, our GitHub generator and social verifier repost are on GitHub under the salsa framework organization and guac is under the glocksec organization on GitHub as well um I'm Ian Lewis on GitHub and on E M Lewis on Twitter. So if you're inclined, you can hit me up later as well.

A

So thank you all.

A

I have about three minutes for questions, so maybe one or two. If there are any.

A

A

So the question was like there's a lot of GitHub on the slides. Is there an equivalent project for gitlab uh the there is? uh We are working with gitlab to uh to implement some of these things like right now. Some of the issues around on GitHub are around isolation.

A

So, like um we are kind of Lucky in a sense on GitHub that we are able to create these generators um using uh features of GitHub actions that allow us to isolate that from the actual build, um but with gitlab there's not really a way to do that quite yet, and so we're working with them to actually like be able to generate the problem safely and then be able to like do essentially the same things we're doing on GitHub as well. So, yes, we are thinking about that and working with them on that yeah.

A

Any other questions.

B

C

Like a developer,.

A

Right, so the question is, um as a user of of a CI system or a smaller CI system. uh Is this something that we have to rely on the CI system itself to implement? Or is this something that we can build ourselves?

A

um This is really something that usually the build system itself would have to kind of implement um or that, ideally, it would Implement, uh but um in the case of GitHub, we're like kind of using uh a uh a feature of the system in order to like build it ourselves. But we are kind of essentially working with other, like build tools like uh like build kite in order to be able to implement these things.

A

But there's still a lot of that is still a work in profit address on other CI systems, particularly with regard to the isolation that we mentioned about gitlab and also about around the identity portion, so the uh being able to have an identity provider, so an oidc provider, for example, for providers like build kite and gitlab in order to provide the identity piece that allows us to sign and do keyless signing.

A

One more the red shirt here.

A

Okay, so the question was like: did the salsa verifier like walk a tree of like dependencies in order to verify the dependencies as well um in our case like what we're doing is essentially just verifying the provenance that was generated for that image, instead of doing any sort of dependency walking, um that's something that we can explore in the future if we have, if the artifact itself has a way of like uh defining or if we have a way of figuring out what those dependencies are um right now, like you know, the only way we do, that is like if we were explicitly provided an s-bomb or we like tried to generate one ourselves, but we don't really necessarily want to.

A

Trust the artifact itself to provide us with those dependencies we'd like to be able to get those like kind of out of band, but yeah. That's something! That's an area that we kind of like want to think about and explore.

A

Well thanks, everybody for coming um I'll be sticking around. So if you have other questions- and you want to talk to me in person- then I'll I'll be available thanks a lot.