Cloud Native Computing Foundation SigstoreCon NA 2022, 28 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Securing Kubernetes Manifests with Sigstore and Kyverno - Jim Bugwadia, Nirmata & Yuji Watanabe

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Securing Kubernetes Manifests with Sigstore and Kyverno - Jim Bugwadia, Nirmata & Yuji Watanabe, IBM Research

Kubernetes offers a powerful declarative configuration management system which allows users to specify the desired state using a set of resources. In this talk, Yuji and Jim will show how you can establish trust and protect the integrity of Kubernetes resources. They will use Sigstore to sign YAML definitions and Kyverno to verify resources during admission controls. They will highlight real-world use cases for resource signing such as tamper-prevention and approval workflows which can be driven using OSS tools like Cosign and Kyverno.

A

Thanks folks, it's about six months ago, Eugene presented at a caverno community meeting and he talked about you know, yaml signing and the first question. You know when we were kind of discussing this is we were asking well? Why would you want to do that? Why sign the animals? What's the benefit, and what can we derive from this and, of course, once we got past that discussion, the next thing that came up as well. How does this actually work? What does it mean to sign a yaml manifest?

A

How do you get this into your cluster and verify at the right levels, so we're going to cover those things today, we'll also discuss you know some of the challenges with signing and verifying yaml files or resource manifests in kubernetes and then we'll you know end with the full demo of how this flows end to end using a six-door CLI as well as caberno, which is a validating admission controller that runs inside your clusters.

A

So, let's start with the Y right. So why would you want to sign yamls and just to kind of recap, signing anything gives you three things. The first thing is: it makes sure that what you, the data, you're receiving the message or payload is authentic. It comes from who you think it comes from. The second is: it makes sure that the it hasn't been tampered with, so there's Integrity protection, and finally, it also makes sure that you know once something is signed. You cannot kind of go back and reneg on that or change that.

A

So those are the three things you would get with signing pretty much anything now putting this in the context of kubernetes. You know like it or not. Most kubernetes resources are declared as yaml Manifest files. In most cases, maybe perhaps not in all. These would be.

A

um You know in a git repo somewhere and you're, going to use a git Ops controller to push these yaml manifests into your cluster, and then controllers within your cluster are going to look at that declared State and the current state of the system and reconcile the two so in kubernetes driving its declarative configuration through yamls becomes pretty much the standard practice. So putting you know that, together with now signatures, the benefits you're going to get is you're going to make sure that the data in those yamls the contents of your resource manifest is authentic.

A

It hasn't been tampered with once it's deployed into the cluster, and you can also make sure that you know you're you're, preventing any inadvertent or unauthorized changes, but which may occur in your cluster itself. So signing can also be used to build some pretty interesting workflows like approvals. If you want, you know multiple signatures before a resource is deployed, you can do that or if you just want to prevent you know some resource from being going into production until your QA or your production team signs off on it all right.

A

So you know, hopefully that gives you an idea of why we would want to do this or why sign you know, resource manifest, so diving into the how and you'll see some of this live in a demo itself, but the two components we're going to use in the demo, so there's a six door CLI.

A

So this is in a repo under the six-door repository, it's creators manifest six store and what this does is it contains pretty much a command line, as well as a library everything you would need to sign and verify yaml resources, so it builds on top of cosine. It has you know, logic you know to to be able to inspect the yaml to be able to. You know, convert that into a couple of hashes which are ingested in the yaml file itself.

A

So you end up with annotations within the yaml file, which are then used for verification. The other component you would see is cabernovich runs as a admission controller inside your cluster, so caverno can act as a validate as well as the mutating webwork.

A

Its role is to be able to inspect any admission request, apply your configured policies in this case. That would be a policy which you know for signing your yaml files itself, and you can also you know, generate resources, but that's obviously a different use case from what we'll look at today, so putting that all together in the end-to-end flow.

A

Typically in your CI CD pipelines- or you know, based on your approvals workflow, you would use a CLI or some tool, like you know, a GitHub action which would do the signing much like you would for a image signature and then, once that resource is deployed, it's pretty much self-contained, there's no additional fetch, which happens from an oci registry, which is slightly different from an image validation, workflow.

A

But given every all of the data and these annotations, which are inserted into the yaml, what we'll be able to do is then verify the contents and either allow or deny that request. Now. This request may also originate inside the cluster, so in case somebody edits that GMO it would be. You know you would be able to deny that if it wasn't signed and didn't flow through the right phases of your deployment pipeline.

A

So, given that you know, ug is going to dive in into details about some of the challenges encountered while building this solution and then we'll look at a live demo itself. Thank you. Gene.

B

B

So let me introduce a little bit detail of the how the side this mechanism works. The first signing Miami manifests. So this is a tool of the cly. uh Actually, the cube CTL program, the it's a original some annotation is in injected to the original- must be via my best. Basically, it gets this manifest body and encode the it's signed by the cosine sine below command. Then it encoded message and encoded signature is embedded into The annotation, so the to verify this sign the yellow manifest. So there are three steps.

B

The first step is uh put put through the encoded message from The. Annotation then check the significance. Signature of against this encoded manifest yamu manifest so then sorry same. The once does it encoded VMware manifest is correctly signed, then compare the origin, the matched.

B

So yeah this mechanism is actually embedded in in the key value control, so the to configure the uh this uh signature identification. We can use Cabernet policy, so actually the new declaration uh validate manifest is defined by this, so the um normal, and so the mass of so resource to me to be signed is specified and also the as public key is specified in this position.

B

So uh this policy has another extended use case like list of desktop keys and multiple signature, verification and more more complex cases, logical and or operation of the between the keys. So that is, you can Define such a complex case in in this given policy.

B

So the challenge to verify this signature, the in admission controller, is a mutation. So the uh when you uh create apply a send the request from the QVC TL comma capacity command, so the uh originally Amino manifest is not coming to the uh admission control point. So some mutation happens before the before that checked.

B

Like the native admission control, May introduce some mutation to the original Emerald, so the thing to verifying the con yeah my best, we need to think to consider how which part is generated from the uh those trusted uh admission controller or this. This change is not allowed or not so that that uh consolation is required.

B

So the solution is first solution. Is you can specify the which, which part of the uh manifest can be a should be ignored? So maybe that's some. You can specify the this. This mutation is introduced by the some native communities, admission control, or maybe you, if you know you, you have some uh your own Uh custom admission controller. You can specify that ignore configuration.

B

Okay. That is one approach. Another approach is a driver and a computer, so this is a approach to compute the diff by using the duratum output. So I will explain later, but this this approach is pretty good because you don't need to specify the which part should be ignored because it is a expected. Mutation is automatically computed by the driver in admission control phase, so that part is good. But but another requirement is the you need to add some addition.

B

You need to put additional permission to the keyword, admission control, so actually the you need to do the driver in the admission contract, so the create permission is required so, but by using this drawer you can remove the cost of specifying the each individual ignore of a unit configuration.

B

So approach is like this, so the admission requests come, but the admission people coming to admit admission request is coming to the keyboard. No, the the some attribute, some um uh library or annotation. Some part of the Manifest is mutated, maybe added to something so, but this change is also computed in the uh in the driver in the admission cultural. So then you can check you can compare those two. So do I run this out and coming the admission code rules. Okay, the uh you can you can automatically filter the uh expected uh expected mutation.

B

So let me let me show them: okay,.

B

B

So now this policy is okay,.

B

Okay, this is a policy already deployed in this uh cluster, so the this polish specializer, which resource should be protected and the public key here and.

B

B

This is a sample deployment and I want to deploy this deployment configuration, but I already set the policy, so so policy says the signature is must be provided for the so for this deployment. So if I create.

B

Deployment to the Target system, so this keeper, you know automatically protect the deny this request, because the signature is not provided so next I will do the.

B

Put the signature to the original vcm required: okay,.

B

This is generated, so some annotation is already included here. So if I apply.

B

Now I I did previously I did this one, but if I use the signed yamro manifest the this creation. This deployment is also successfully deployed.

B

So the deployment is already running so now I if I. So this is expected so I successfully deployed, but maybe later someone comes comes in then change change. The configuration, for example,.

B

I will start the edit the.

B

Okay, this part another privilege escalation. This part is originally set as by true, but if I change this part by two.

B

Then this edit is prevented because, as you can see, it's at this part the originally this value is Big Force, but after that too, so this part is not. This part is the gap a bit from the side state, so this is prevented. So that's a demo.

B

A

So, just to quickly recap: I guess: in the last section, yeah what we saw was by signing and verifying your yaml manifests. What that will, let you do is protect critical resources.

A

um It's probably not the best idea to use this for every resource in your cluster, but certainly things like cluster roles, policies Etc you might want to protect and make sure that those are signed- and you know, can be not tampered or not changed later in your pipeline itself. The signing and verification is pretty simple: six door. Of course.

A

Cosine can sign everything, so you know might as well use it, and it's using the you know the blob signature format underneath so you can utilize that to sign, and then you can use admission controllers like kiverno to do the verification in the cluster itself. So certainly, you know feel free to you know either reach out on the kuberno channel or on the cosign Channel. If you have any questions or would like to see any other examples of using. This also want to give a quick welcome shout out to Rico.

A

She did most of the implementation for this feature. Unfortunately, she couldn't attend, but it was a fantastic job in getting this implementation in and feel free to reach out to either ug or I. If you have any other questions and if you have a few minutes happy to answer any right now,.

A

And like why, why do you think that makes sense.

A

Images right, yeah, so the example uses deployment, but it could be in any resource. Typically, you might want to choose a few resources in your cluster that you're signing protecting in this manner.

A

um Certainly if it's a deployment of something that is critical like a security tool, would make perfect sense, but could also be done at a pod level and matching a particular namespace. So really a lot of flexibility in that.

A

A

So sorry was your question: uh if yeah so yeah and it's interesting, you know with githubs and something to kind of you know obviously you're we're mutating the yaml. You would probably want to do this on a pre-commit at the right point in your life cycle right so yeah.

A

A

A

You can- and that is uh interesting case, so you could have a policy which verifies every other policy, including itself. So then the only thing you're really getting from that because that policy is already deployed is you're, preventing somebody from changing that policy.

A

Any other questions or, if not then oh God, yeah.

B

Yeah we need to have the original content to match the mutated one and the original one. So that's nice.

A

Yeah, one of the other schemes we had explored is, if there's some other way to get a canonical form of the resource, but it becomes a pretty hard problem right because you need to know what the actual initial form was and strip out all of the other things that could have changed in your pipeline or through other mutating controllers or in cluster controllers like when a pod gets deployed. It's amazing how much stuff gets added by just the standard controllers, and all of that needs to be ignored.

B

You're correct.

B

B

Field by Fields, then, by concerning the ignore computation.

A

Yeah, so you could configure the policy to ignore some set of fields, but if you know that some other cluster controller will change those or you could do the dry run, if you're confident you're going to get back the same results and that's typically, the easier way to get started is with the dry run, but either scheme works or a combination.

A

All right well, thank you. Thank you.