Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2022, 12 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Trust But Verify: Bringing Supply Chain Integrity To CD GitOps - Yuji Watanabe & Hirokuni Kitahara

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Trust But Verify: Bringing Supply Chain Integrity To CD GitOps - Yuji Watanabe & Hirokuni Kitahara, IBM Research

Speakers: Yuji Watanabe, Hirokuni Kitahara
Using GitOps automation to deliver Kubernetes cloud native applications allows management of infrastructure in the same way you manage application code, but lacks the supply chain controls needed to ensure integrity and tamper-proof deployments. Whilst application source dependencies have quickly benefited from SBOMs, transparency logs, and cryptographic signatures, delivery side automation has not participated in the end to end integrity guarantees. Using CD Gitops, Kubernetes manifests are composed from multiple source assets, across several locations, each having their own potential sources of malicious or accidental tampering. Template based mutations occur throughout continuous deployment and prohibit typical signing and verification methods. This talk describes how a properly instrumented CD GitOps process can be extended to provide verification of source assets with cluster enforcement of signatures and policy permissions. By combining keyless signing via Sigstore and intersecting control points throughout GitOps, accurate cryptographic signing of source assets can be obtained and transparency of configuration provenance produced. Finally using an admission controller such as integrity shield, cluster enforcement validates pipeline integrity.

A

uh Today uh we will talk about the uh against the software supply chain topic, but the unique things about this session is: we are through Enterprises, safe and use software supply chain security Integrity to the specifically in the CD kit of scenario. So we will discuss the issue and the proposed solution to this.

A

From IBM research I'm, a senior technical staff, member and I'm leading the several software supply chain security project in the research. So and you.

B

Know yes, I am hero kitahara from idea.

B

Okay, so I okay, I'm here and from I'm in research, Tokyo and I'm.

C

B

Working on CD githubs are everywhere and I'm doing several OSS contributions like Secret store or today's main topic, rocd yep. Thank you so much.

A

Okay, so today.

A

The the first uh I will talk about the background of this topic and then the what we are trying to discuss in this story. In this talk, then we will explain the several building blocks, the Yemen manifest signature and its enforcement so then discussed how we we just. How can we apply the technology to the CD guitars and we found we just point out some similar issues and talk about the proposed solution called interest.

A

Then uh hero will do the demo okay, so the first supply chain is a product, so the protecting the product from The Source. They are sourced and it's built at the factory and go to the up. Each package then become the product so but from the end user perspectives we need more protection like it should be securely delivered and it should be securely painted and at before, actually the protect it should be protected at the time of the use.

A

So the whole this end to end is at the scope of the uh protection from the user perspective when we think about the same things in the uh application deployment in the coordinative case, the very high level right picture, the fastest source is in iterable, it comes. It goes to the CI Pi Prime. Then we use the image and push it to the registry and manifest is created at this git report. Then the when the deploy the application, the image goes to the cluster, manifest to go to the cluster by your CD pipeline.

A

Then application is stand up to the cluster, so this is a kind of the end-to-end from the source to the actual the time of the use. So what is the risk? So the maybe some case uh image on the registry, maybe the Target and manifest on the gate. Ripple is also the target, so under the even after the deployment, the Manifest is on the cluster so that it may be changed by some someone.

A

So those uh uh attack and measures activity is is a target for the protection okay, so so for preventing this, the uh actually who play for protecting the Integrity of these artifacts and the application, the user, the signature is applied, so the many Technologies, uh the English people just signing the image and how to protect how to enforce the image signing on the cluster side, but uh from the for the application.

A

The image and manifest are equally important so that we also need to protect the Integrity of the Manifest to secure the entire pipeline at the right side. So the in this talk we are mainly focusing on the uh manifest sites so how the modifies can be securely signed and deployed to the Target first, okay, so the for the for that purpose.

A

We actually contributed to the sixth project for the Manifest signing, and this is a cly2 cubicle, probably by using, for example, by using this command some annotation signature annotation is attached on the original Yahoo manifest, so the in this case the two annotation one is an angle in the message and the second one is encoded signature. So so this is so.

A

The signature is encoded and included on the Manifest, so if you can directly deploy this to the cluster, signature is also the attached on the deployed resource, manifest so the uh how to verify the signatures so the so that this is a flow of the how the signature is used to protect my device. So the sign up uh first signed a manifest, then pushing to the gate report. It will be push pushed proof.

A

Deploy, approve the Manifest to the to deploy the crust so before the project, the the project can directly verify this signature by using the cma2 and then, if the amplification is okay, so the project can deploy is Manifest to the cluster cluster sites. The admission control is is enabled so the independently this signage signature was attached on. The Manifest is verified in the cluster side uh that that specification happens in the admission time or even after that, its signature is continuously monitor.

A

So you can enable these types of the protection at the cluster side, admission control, the the is it so, the it's the.

A

Things to our library to a tool for the Yamaha manifest signing is used and in the admission control, so the one admission controller is integrity sheet. This is a project we are contributing to the uh open cluster management project. It's a extended. It works with Opa gatekeeper as a admission control. The Integrity Shield has integration with Obi gatekeeper to provide the yamuna, Manifest verification as signature verification.

A

Also, the very recent release of the key value has the Yemen manifest signature verification capability, so the several tool can be used to enable the admission control, okay, so the, uh but we then next next we we will see the more complicated situation the so in some cases you don't push the Manifest directly to the digital. Instead, you push the source material of the to produce some material as a manifest, since the actual manifest is generated. Some too right, the customized command, so customize command is already integrated in the qc10 comma.

A

Also that you can use the directory customize command, then that produced the Manifest from The Source material on the git, repo plus X. You can also include external dependent dependency material. Also that you can input some temporal template parameter, then the command this command produced a generated manifest.

A

So in this case, as a discussion is a situation is a bit different. Okay, before going to start, let me explain more uh situation of the uh this templating, so if you have so, the source material is not just one one repository, so you can have multiple reports to be like the application repository config repository. So you can use two two as a kind of the parent child relationship, but you you can also have very different, complex scenario.

A

Each organization has different application components as a kind of some configuration some module, then all the uh a source material is composed into the single Yemen set of the yellow manifest. So this is also so. This is a kind of the software supply chain issue very similar to the uh image period case. So you you sometimes this. If you build a source image, you need to prove the uh a lot of the external dependency. The same issue happens also in this manifest build.

A

So, in this case, the so from the signature amplification Viewpoint, the the situation become complicated, so so cyanide can put the signature on the social repository and the project can verify the source material, but the deployer after that verification deployed to the Manifest bit and then deploy the generated manifest. That means genetic manifest does not have the signature, and so that means the project can verify the signature before actually the point, but it cannot it it.

A

It's not delivered to the cluster size so cross side admission code is doesn't work uh in this case, so that means a cluster site needs to rely on the free check by the deployer okay. So but this is becomes a situation in the city, githubs acidic it offsets. If some some configuration configuration is management githrip. So if the Source material changed, it's automatically synced to the Target cluster.

A

It's for that. Six. During that sixth CD includes a kind of build and deploy State the build step and uh gets the source material input then generate manifest. Then each deployed to the crust automatically whenever the source material, so Source git repository, is updated.

A

So the if we use this three digits very, very useful, but if we apply this CD tops too the we, the generator management, don't have the signature, as I mentioned explained before, so that this is a kind of the mutation happened in the CCD pipeline, so no signature is, is delivered to the Target cost.

A

So this is a motivation that we uh we come up, came up with the interest project, so the in this case there are several questions, the the: how what kind of what kind of check is done in the CD side and how we will manifestability is properly done and the so the signature is not attached on the generated one generated manifest so much signature based protection does not work after the deployment, so those kind of the positive. So in this case we need to trust the CV side, the cluster side. There is no verification.

A

uh We want to improve this situation. That is a interest. Allow us to do this kind of the trust but verify approach. The interest is, as the capability attached to the audible CD then first three function. One is verify the source material signature. Then the second is, if it's okay just put the signature on the genetic manifest, then the these transform from the skit report to the.

A

So we produce the problems record to to allow to transplant to make this transparent. So the this three function is a core capability of the interest, how interest Works? Actually so this is uh interest controller attached attached into the in the Argo CD. So basically it monitors application 6 happen if it happens between the git repo and the target cluster and then it's it's deported in in the applications. Applications here is a rocd as a configuration which specifies how the meter of things should be performed.

A

So the interest controller monitors this applications here then detects the okay, so the Gate of things happen then uh trigger the three function, verify store signature, so Source material signature and produce produce attaches signature to the genetic manifest. Then the problem is they could be pushed into the prominence from the interest control.

A

So by using this, the signature is attached on the G80 manifest so Target cluster sites. The admission control can successfully protect the Manifest by using signature at which animation time, and also the even after the admission okay. So the this is a high level idea of the internet. So from now the hero will show the some demo and some deep so foreign.

A

B

So, uh okay, so let me start my demo and I will show how rocd users can protect their own CD github's Pipeline with argosity interest. Okay. So uh during my.

A

B

Let's assume that I am an argosity user and I have already configured my CD github's Pipeline, with Argo CD and the left top side. This Source repositories are synced into Target cluster. Based on my my application, CR and- and so the point point here is uh The Source repository can be multiple and nested, so this means Source repository in CDV. Talks could be uh very, could have very complicated dependency relationships.

B

So in my this demo, I have two Source repository. The first one is the root Source repository, which is configured in my application CR, and this refers to another repository as a external dependency. So this they have a parent and child relationship and now I'm thinking about how I can protect my CDV tops with digital signatures.

B

So for that I will introduced argosity interest and uh to make end-to-end signature protection. In my city, github's pipeline I have installed admission controller on target cluster, which verifies a signature on the Commander's resources and to configure entire obviously interest, and what I need to do is just three steps. The first one is signing source files so to put the signatures on the source repository.

B

My with my signing key The Source repository Firestar protected, so to do that I can sign the associate profiles with these two commands and then the uh these two signature files are generated and these signatures are used by rocd interests for social material verification and the second step is configuring, a lucid interest with custom resource called interest profile, so this profile has a selector to which application should be scoped in should be in scope of this entire profile and also it has a public key for the source, material, verification and also this has a signing key for the yamla Manifest signing.

B

So this is what I'm actually using for interest profile and it has application selector and fabric key and signing key reference, and also this has a much function. This defines pre what type of resources should be signed by rocit interests, so uh the as a final step. This rocd interests will push the assigned diameter manifest into wholesale registry as our oci image, and there are pushed image and signature will be used by automation, controller for verifying the resources so to specify the rotation of this OCR image.

B

That I will specify the OCR image in application CR, and this is the one. This is my application. Cr has a special annotation here and it is Manifest image and this is the actual location of the assigned manifest so by completing these three steps. I have enabled end-to-end signature protection in my city githubs, and let me explain how our city interest works. So first I will see the user will update the content inside GitHub repository and then it will be detected by rocd. So application sync had started so then I will see.

B

The interest is, keep losing leaders, keep monitoring the application sync. So it detects simple event and it starts verifying my source, material verification and if its verification has passed, then it signs the generator viable manifest. As on the Osteo image, and at the time of admission uh when rocd is trying to deploy the resources, the admission controller will use the pushed or CI image with signature for resource verification and all the manifestability process are recorded as a provenance data and it will be sent to provide a store.

B

This is how Roc enters works, and when the signatures are all okay, then Argo CD shows this synced status. This means the application sync has been correctly done and that our resources are deployed on the target cluster successfully. This is successful case, but let's imagine that there is an attacker on the Target crosstalk first, so this attacker is trying to Target Crystal resource without signature, and then this admission controller will block the attackers change because the attackers change is not signed that that change does not match with the sign of State in the OCR image.

B

So this is, after example, of the automation. Contour denies the attackers change using chipset Reddit, and so this means no one can change their different results on the target cluster without signing. So I was.

D

B

Rocd interest only generates the signature for the uh generated manifest by rstd, so rocd only can deploy the resource and change the resource update resource. Do that that kind of operation? Okay. So how about Source report social report side so yeah? And there is an attacker who is who has access to the GitHub repository, and this attacker is trying to change the firing inside git report, and actually this attacker can change the file inside this Repository.

B

But then I'll go see. The interest verifies The Source material signature, but signature is not valid anymore because the fire has been changed. So then, because of this does not sign the Manifest here, so there is no signature for the attackers change. That's why admission controller I can block the attackers change and application sync for it.

B

So this is actual rocd UI when blocks the attackers change for the source report and it is showing sync waves, so the application Source value change has not been you brought on the target to clusters so so far I had I have been talking about the signature. Protection on the GitHub swiping, but I can check. I can verify the different resource whenever any, whenever I want to check by using this cubic single store command.

B

This is provided by civil store and by specifying the sales kind and the public key and oci image which I have mentioned in the special annotation, and then it reports that uh resources science state, so it if it has it, is showing sign. True, it means this resource is resource, keeps the assigned State and no change. What changes were made to the resource and not only this kind of binary information, but also some I. In some cases, I want more details about manifestibility and for that I will see.

B

The interests will produce the provide provenance data which contains all the manifestable information, for example, uh the all Source variable URLs and all the version or coming to versions at the time of manifest build and also it included the application step. Application CR snapshot. So this this is used for checking the article 3D status at the time of manifest, build and also the oci image and its digest uh included in the provenance. So this provenance is provided by rxcd interest and pushed into program store, and once this data is ready on the store.

B

I can query this program's data by very simple command, and it shows what source reports are actually used and what versions were actually at that time and build timestamp or another metadators or can be checked by seeing provenance data and the actual prominent low provenance data is something like this.

B

It is uh input, Json format and it has a restore materials like URLs and commit versions and also synapse shutters are snapshot, is recorded as a build parameter, so I can check the status of our CD at the time of manifest build and also, of course, the image reference and digest are here so I can check the uh by by checking our by checking proven oscillator I can check the when my application was built and what social report and what versions were used and how was the adversity status were like that kind of informations can be checked by Providence data.

B

Okay. So let me summarize our today's overtalk, so we have addressed the issue in the supply chain. Integrity of the modern C digital engines and to serve that issue, we have introduced in argosity interest and actually I. We have confirmed that our city interests can work for in every end-to-end, signature protection and verifiable problems.

B

Yeah and overall tell us about the verify approach can make the today's application deployment more transparent and accountable and more traceable, and that's our conclusion.

B

So these are our contact and please feel free to reach out to us and uh and also please visit IBM Booth so uh or the uh any feedback and comment and questions are all welcome. Thank you. So much.

A

Thank you. So please guys have any questions. Comments. Okay,.

C

Actually, I have a very quick question: do you keep referring to interlay's profile? Is it equivalent to policy what kind.

A

Of policy yeah, what's.

C

The difference.

A

So the interest profile is uh not the rule to how the interest behaved against application. Sync events, so what kind of the application? What kind of thing should be monitored? Which resource should be signed and what is recorded? Those kind of uh coheration is included in the internet's profile. So so that is a kind of the policy for us yeah.

A

So could you could you teacher.

A

A

So the mutation in the cluster by or see in the build uh manifestability.

A

Yep, okay, it makes it very challenging.

A

The puppet: okay, yeah, that's a good, very good quiz. Thank you. So the admission controllers. Okay, so the question is uh so the sometimes. uh If you deployed the Yaman, manifest to the crosstalk some admission control, other adoration control, even the native connection, sometimes may change the uh uh yeah yeah manifest. So how we can deal with that modification in this signature checking am I, correct, yeah, okay, so that is really important question. So this is actually we are addressing the to develop the admission control for the signature, signature, verification enforcement.

A

The uh two sixth word distribute.

A

So that this is a tool.

A

Did it? Ok, so, okay? Well anyway, so the uh we have the internally uh the how the configuration for explicit explicitly ignoring some Fields. It's it's a expected mutation by sardic cycle or uh trusted admission controller or something. So that is one approach. Another approach we are enabling is driver, so internally admission controllers, any admission controller to write them create executive. Then computes. The expected changes in the Manifest then compare the which part is expected, which part is not expected, then so the the two approach.

A

So we discussed uh this taking this kind of Challenge and solution in the different talk in the six token The Joint presentation with Jim pagalia kiberna, so the we actually re-enable the this capability in the keyboard series with jimbogody. So please, please check the substation that talk if possible, yeah. Thank you! So much. Okay, yeah hi!

A

D

Works this time.

B

I'm very confident for you.

D

D

It's manifest what cosine right is there any reason you did deciding the git commit that modified the files directly with cosines, like git sign instead.

A

ah Yeah that is really important that we need to look into more, so the yeah, the okay, obviously is totally different from the git commit signing, uh because the use case is a different commit commit signing is more foreign individual committees signed, but the Avenue manifest signing is targeting the size, quantity design, but new new Tool each time. It's it's good sign from the sixth floor. Maybe uh we need to look into the more how we can apply that new approach to this scenario. Thank you.

A

Okay. Any other comment coming into a question.

A

All right, thank you so much so. Thank you for today's question.