Continuous Delivery Foundation CD Summit 2022 - Detroit, 4 Nov 2022

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: An OSS project's attempt to secure it's supply chain - Ankit Mohapatra, Berkshire Grey

Description

For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/

An OSS project's attempt to secure it's supply chain - Ankit Mohapatra, Berkshire Grey

Jenkins X is a cloud native CI/CD platform built on top of kubernetes with out of the box support for gitops, secrets management, preview environments, chatops and much more.
In order to provide all these functionalities, Jenkins X uses many open source projects as part of it's supply chain in the form of go modules, npm packages, helm charts, docker images and terraform modules to name a few.
In the light of the recent high profile supply chain exploits and attacks (solarwinds, codecov etc), securing the open source supply chain becomes critical for us and our end users.
But how do we even keep track of all the packages that make up our supply chain and then secure it?
We started by generating SBOMs (Software Bill Of Materials) for our artifacts and using vulnerability scanners to identify potential vulnerabilities.
Currently, we are in the process of integrating with tekton chains.
This talk is an attempt to summarize our supply chain security journey and what we plan to achieve in the future.
We will explore the fascinating world of SBOMs, SLSA (Supply chain Levels for Software Artifacts) levels and in-toto attestations.
More importantly, there will be some practical examples of the abstract concepts around supply chain security and how Jenkins X attempts to make a secure supply chain accessible to everyone.

A

So well a bit of intro who am I onkit I work as a devops I work as a devops at at you know, like Berkshire gray, which is you know like robotics, which does you know like robotic stuff. I am also the maintainer of uh Jenkins X. uh Why is it? You know like level two, because there are two horns? uh That's that's like just a point. um So what is you know like Jenkins sex? uh It is not.

A

um It is not. You know. Like Jenkins, uh we have a lot of people who come to our. You know exact channels and keep on. You know like asking us uh things about it, so we are uh completely. You know like different thing.

A

um We are a CI CD uh tool. I would probably say that we are, you know like a platform, so we have a lot of you know like features, so we have CI which is done using um using you know like tecton. We have, um we are kind of, like you know, like Argo, but we have our own uh jigs. You know like it Ops. uh We have. You know like preview stuff. We have, you know like Secrets.

A

uh So that's one of the reasons why it's not a tool, but instead you know like a platform. um The whole point is to make the cncf you know, like landscape appear less. You know, like scary, if you have seen it there's like uh too many things to choose, so you can either use that or you can try. You know digenc uh what is uh what's the problem that we're that we are General are trying to solve. So we produce a lot of you know like binaries.

A

We have you know like images, we have telephone um stuff, and so we are a software you know like Factory for for for for um so so the way we know like produce them, uh we use you know like GitHub and we also have our own uh our own. uh You know like gcp uh stuff, uh but the big question is: how do our end users know that all of these you know like binaries and having charts are actually you um are. Are you know like coming from us and not from say Jenkins?

A

Why or something? um Okay? Where did I go great? um Do we know what goes into building those things and are they safe to use? um uh Eventually, we want to help the end users of you know, like Jenkins sex have a secure by default chain. That's that's. You know like the North Star uh internally we use uh so we use analog gcp. We have spot because it's cheap to use them. uh We run the latest. You know, like version of our own Tool, uh every time a test you know like passes.

A

Our cluster gets the newest thing, so we do uh tests so that we can catch it before before. You know, like users of Jenkins x. uh We also have you know like an Amazon account and we are in the process of of you know like adding some tests. There.

A

We also have, we actually have you know like a Helm chart and you can use it and install it using uh on-prem clusters. So we support uh quite a bit of stuff.

A

uh Where did we start so initially we had huge Docker containers.

A

uh Some some of you know like dependencies we had were, um should have just been. You know like lines of code rather than getting an external. You know like package, which is not uh you know like maintained anymore. uh We did not have you know like a s-bomb uh which is basically um you know like a list of stuff that makes up your you know like binaries. We did not. You know like sign things and most likely we were salsa. You know like level uh zero to one.

A

We were not zero, I guess because we had you know like Version Control. We had you know like a scripted, you know like build system, but we were not exactly one even because we did not have you know like provenance.

A

um So, let's start by taking a look at the checks. You know like binary. uh You can see in in our left hand, side I, guess anyways. um It did not. Have you know like s-bombs, it's not signed but more, but if you just run the JX, you know, like version uh all you get is just this number. That means uh nothing to me uh like which branch was it made from which you know like commit Shah was it which you know like version of go. Was it um so we?

A

uh So you know the first thing we did was add. More um add more. You know like stuff to it like. If you now do a JX version, uh you know that it's built using go uh well, it's built using go one dot. You know like 19.

A

um it's using the main branch which is good and it's and in that Branch it's built on like it's it's built using this, you know like commit Shah. uh What next um we had huge uh like some of the images we had were huge, like one of them was eight uh 1.8 gigs and we have got it 204. uh You know like MB, um so doing these kind of things you know like these. Are you know like low hanging fruits that that that we first got? You know like sorted out?

A

Also, we were using tags and in some of the cases we were just using, you know like latest. uh Instead we are trying to use. You know like the digest Sha, it's so some of the uh you know some of the tax actually use. You know like digestion now, some of them at least use tags and not you know like latest, so it's still something that we are doing now.

A

um So what are uh software bill of? You know like materials? um They are, you know like trending today, they're very you know like popular uh everyone. You know like talks about them, uh and so that's where we were like well, we need to have you know like s-bomb, um so we started with sift and one of the you know. Like reasons we picked, it was because we use you know like go releaser, so Swift comes as the default. You know like option. Four. uh You know like for that tool.

A

Of course you can, you know like swap it out and that's fine. uh We also used you know, like the Json format. um I have a sample, you know like as Mom. Of course it's great for you know like machines, but if you want to you know like take a look at it, it's hard. It's it's just a huge file, a lot of things inside and I guess we can. uh You know like take a look at one of them, so this is one of you know like the s-bomb that we have.

A

uh You can see. So this is so uh most so uh it has a list of all. You know like packages. This is for one of the package uh you can see it has. You know like a spdx, uh you know like ID, so that's just a hash, so every you know like package that you have in your home will uh will have you know like a unique hash. uh It has the name which is basically you know.

A

Like name of the package, um you can see that you know like the license is none, so that was a bit of you know like concerning for us, because in the end, uh if we have, you know like a spawn, but we don't know if we are using. You know like a license which is not you know like open source. You know like friendly like then then like it doesn't really make much sense for us, um but but that is an issue with sift and we will. You know like talk about that uh soon.

A

uh There is also you know like a CPE which is the well. It's actually not. You know like package, it's uh it's. No, it's it's the common platform uh yeah the animator. uh It is basically uh unique.

A

You know like ID, um if you, if you see you know like the CPE, it starts with a which basically means that it's an app, uh it can also be Edge, which is uh which is for say, you know like a hardware, basically a device or it can be a o, which is for say you know, like Ubuntu or any other thing uh we have go, which is you know like the vendor in this case.

A

Well, in our case it's in another language, uh then we have uh compute, which is just you know like the package name and then we have. You know like the version of the package which is 1.10.0. uh The you know like Purl, is just a package. You know like URL, since we use go. It just comes from the go.mod file.

A

um If you want to look at it more, um this is basically the full file and you can scroll and it's a pretty big file. uh We are using uh spdx, you know like 2.2, which is not you know, like the latest. uh You know, like version uh sift, actually does not. You know like support it. Yet there are, uh you know, other tools that we are uh planning to. You know like switch to in case. We don't get that from safe soon,.

A

So what for you know like the issues, as I said, all of you know like license uh info, is you know like missing in sift um so I tried to uh you know like look at that code base to see? Actually why it's not there like? Is it because uh there is some? You know like issue with us and not with them and turns out that when they, when they try to get the package you know like info, they actually don't uh they.

A

They actually don't have a field for you know like license, which is why it's always, you know like an empty slice and so when they try to get the value, it's always known.

A

um So it also does not support version. You know like 2.3, so we are, um we are, you know like thinking. If we should move away from safe to something like you know like KO or you know like APCO or just the kubernetes, you know like s-bomb Tool, uh all of them.

A

You know like support it now, but I and and uh I think the most you know like important thing is um so we have you know like the s-pomb, but then what um at this point, what we did was we have gripe which, um which you know like scans them and tries to find any you know like vulnerabilities in them, but uh uh but you know like what what you know like we would like is. Maybe you know like a Cron job that runs.

A

You know like once a day and tries to you know like tell us what is uh what is wrong with uh with our bomb.

A

um So now that we have, you know, like the s-bomb, of course you know like uh what we. uh What we know like wanted to do was to you know like sign um all of the stuff um we started with. You know like binaries and, uh and you know, like the s-bombs, so if you see you know like the image, you will see that we are uh signing. You know like the tar file and we are also you know, like signing. uh You know like the bomb file um at this point.

A

We are using keys to you know like sign them and of course that's not a great thing, because uh if you have to you know like manage your keys, that's always a big issue like you might get like. You know like. Maybe that key gets leaked uh leaked and then it might become an issue um we, we actually have a branch where we are trying out. You know like keyless, uh you know exciting, and that should be out. You know like very soon um now that we have. You know like signatures.

A

What about you know like the provenance data um and what about you know like attestation, so there is, you know like a pull request which will add that to our build. uh But what is you know like provenance? It's uh in a in a in a like. Very uh you know. um So what it is is is just um uh it's just when how and where our you know like artifacts were uh were built and when you sign, you know like document about that, it then becomes.

A

You know like an attestation uh in total is probably one of the most uh popular. uh You know like format at this point. So for our you know, like GitHub stuff, we are using the salsa, uh you know like provenance, uh so there is, you know like a GitHub, uh you know action for it and for for our internal stuff, we are using. You know like tecton Chains It's. Still, you know like a work in progress. We are, um we have it uh running for our local.

A

You know like a3s clusters, but not for say you know like Amazon or you know like Google yet, but it should be done fast.

A

um Well, we don't have one, but since this is you know like open source, we can always. You know like take a look at another. um You know like project, because I I think it makes a lot of sense to actually you know like see, see one. You know, like example, before we go and add one to our own, like workflow um you, so there is a link to to uh to you know like the spec. uh There are three things that are uh that are very you know like important.

A

The first one is, you know like the envelope and then there is you know like statement, and then there is, you know like a predicate, so I have one which I downloaded from KO, and um this is basically the in total spec, and this is you know like the envelope this is you know, like a tested, you can see that you know like the payload type talks about you know like in Toto and when we look at you know like the payload that is basically based.

A

You know like 64, um so so, when you have a statement, payload is 64.

A

So so, in order, the payload is basically for you know like encoded. um That is, you know like statement, and that is what and- and we can use JQ to to. Actually you know like format it, and the important thing to check is you know like the subject, so in case of ko, what what you know like they are doing. Is there you know like subject? Is you know like the checksum file? So you see there are uh so what they do. Is they produce KO?

A

You know like binaries for different, you know like platforms, and um so so so that becomes you know like the subject and then you know like the predicate is. What is you know like generated by by you know like GitHub, and you can see that there is the Builder you know like ID, so that you know like tells you that this is like this was you know like created by the GitHub? You know like generator um and and and it has more, you know like info about how it was you know like generated.

A

So this was just a simple uh uh in total spec.

A

um So at this point our uh binaries are signed, but also our you know, like images are signed using, you know, like cosine uh s-bombs are uh generated using shift uh initially, when we tried to push and pull you know like s-bomb, we were using a new tool called um oci. You know like registry, as, uh as you know, like storage uh the, but then, since we use you know like cosine, we can just use that to to actually uh push and pull. You know like s-bomb for our uh um for our.

A

uh You know, like Docker images,.

A

um But what do we want to do? Next? um We have a lot of you know like plugins. At this point, what we have signed is just few. You know like binaries. There are still some you know like plugins that are not signed. We we would also want to sign our you know, like Helm charts, um Jenkins X today is installed using telephone. You know, like modules, we have one for you know like gke. We have one for you know like Amazon, so we support all of the three.

A

You know like Cloud providers, um but but there is no way to sign. You know like modules at this point, I think terraform doesn't you know like support it? Yet, um uh as I said, we have teched on chains working in local. You know like clusters, but it's still not uh integrated to the point where our you know like end users can just be like hey, uh uh you know like I want to have. You know like a secure chain, true and then changing sex goes and installs.

A

uh You know like tecton chains and just us you know leather yeah, so it you know like configures it so that it works with Amazon or Google or you know like azure.

A

um So that is something that we are trying to work on and if you just have you know like the provenance, it doesn't really mean much if your cluster can run. You know like anything, so we are looking at adding um adding you know like everno. You know like policies um so that it blocks any um any. You know like artifact from you know, running in the cluster um as I said uh image, you know, like sizes have uh have you know like gone down, uh but we also want to move to.

A

You know like this show less soon. um One issue we had with sift was: it does not uh generate. You know like s form at say, you know like build time. So what it does is you give it a binary it? You know like looks at it and then it you know like infers the bomb. uh Instead, what we would want is have you know like s forms at say. You know like build time, um and there are tools which can you know like help us do that.

A

uh Apco and you know, like Malang, are probably very uh popular uh at this point and also using uh APCO will help us to have. You know, like reproducible builds uh conclusion. Well, it's so uh so you know like three months back, we had almost you know like nothing at this point. I think we are in a bit of you know like better shape.

A

uh There are a lot of you know like simple things that you can do you don't have to jump into the more you know like complex stuffs, uh uh if you have so I I think um six store, for example, um has you know like helped a lot to to like make all of those? You know like complex tasks. uh You know like simpler. For us, uh there is a lot of things to learn, but you will have fun as you go.

A

uh I would also like to thank uh you know like Osama, who is our Google? You know like summer of code, um so he's you know like a contributor and he did a lot of the work and we would not be here uh if he was not our intern.

A

um We are always looking for more. You know, like contributors, you can join us there's this links to our uh slack channels um and if you have any questions, if we have time left yes,.

A

A

um Cloud bees has its product Jenkins.

A

Yes, this Jenkins X give us that Jenkins CI does and- and we also get support from product leads, which I can see. um Jenkins sex used to be a cloud-based product. It started at cloudbeast right so.

A

Yes, um I mean at this point: I would say it's, um um let's see what does it have? Well, uh it's not based on Java, I guess but but um I think it's more native to you know like kubernetes compared to Jenkins, like you like, you can have Jenkins CI, which which runs on you know like kubernetes, but we were built uh ground up for, for you know like Cloud native and, and you know like kubernetes right.

A

A

A

I mean we, uh we do have a slack Channel but again, like I said it's it's, it is, it is still. You know like an open source project and uh yes, I mean, of course we are looking for. You know like vendor support, but that will take time. I mean to be. You know like honest: it's not going to happen tomorrow, um right.

A

Right something goes wrong and it's nine o'clock at night on a Friday. Yes,.

A

Right right, yes,.

A

Yes, uh so, like I said like so uh about you know like a year back, we we did not even have any, you know like maintainers and what I mean by that is. uh Is you know like Cloud B is left and then the other people left, and nobody had. You know like access.

A

What are we yeah, of course, but but yes, I mean now we have six, you know like maintainers and it's you know like growing. So um we have plans to to to like think about. You know like commercial uh support, but of course not for this year, I mean it will take some time we. So we have. You know like a governing uh like body at this point which we did not have.

A

We did not have any of that uh three months back, so we have, you know like a Toc, and the main task of uh of you know like Toc is to keep you know like Jenkins sex going. That's that's the biggest problem, yeah yeah.

A

What we use you know like tecton yeah, so forgot.

A

A

I understand yes, because I was running uh Jenkins sex at my you know like previous work and well I. So I did it because I am the maintainer. So if something breaks of course, I can go in and fix it, but I I do you know like understand? Yeah? Yes, yes, so so um we have plans. I! Think! That's! That's! That's just what I have in mind like we are planning to go for. You know like commercial support, and um that is something that uh that the TOC is looking at.

A

So it's not dead yet I think that's! That's the important part, because uh people thought that it was dead. So.