Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2022, 12 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: How SIG Release Cooks Trustworthy... - Carlos Panato & Adolfo Veytia, Jeremy Rickard, Sascha Grunert

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

How SIG Release Cooks Trustworthy Artifacts From Raw Source Code - Carlos Panato & Adolfo García Veytia, Chainguard; Jeremy Rickard, Microsoft; Sascha Grunert, Red Hat

Speakers: Carlos Panato, Jeremy Rickard, Sascha Grunert, Adolfo García Veytia
Have you ever wondered how the Kubernetes source code is turned into artifacts for everyone to use? How do you know you can trust those artifacts? Have you heard about signing things and you're not sure how that fits in with Kubernetes? In this Kubernetes Special Interest Group (SIG) Release update, we will give a quick overview of SIG Release, highlight recent accomplishments, review our updated roadmap and discuss our continued efforts to move toward full SLSA (Supply-chain Levels for Software Artifacts) compliance. As part of this, we will deep dive into efforts to move all aspects of the build process and distribution to community controlled infrastructure and our efforts to expand artifact signing beyond just containers. Finally, we’ll talk about how attendees can become involved in SIG Release. These efforts are exciting and important, but we need your help! We’ll discuss how to contribute to SIG Release tooling, the Release Manager role, and discuss our contributor ladder.

A

Hello welcome everybody for the our Sig release, kubernetes release maintainers track talk. My name is Carlos uh and I'm going to let all my other colleagues to introduce themselves.

B

And I have one of the co-chairs of sick release, so it's the first time that I had the opportunity to be at the kubecon in person since 2019. so I'm. Here with chair from you.

C

Jeremy uh also, the first time since 2019 to be here in person, super excited to be here and share what we're doing inside release.

D

Yeah and my name is Garcia I am one of the tech leads for release engineering with kubernetes release and excited to be here.

B

And also what we and we'll do in the next couple of months. So after that, we will speak about how we turn source code into artifacts, so how we actually build the release and which user personas are involved into that.

B

And then, when we have those artifacts, we will speak about how we can trust them or even if we can trust those artifacts and how end users can verify this thrust, then we will speak about how we can own the infrastructure so what it means to have a community-owned infrastructure and also what we plan for the future. And last but not least, we will speak about how to get involved in Securities and how to help us on our efforts. So Jeremy tell us: what's new in sick release,.

C

Kubecon over the last year, the kubernetes 125 release came out. There were more than 40 enhancements, as we've seen over the last few releases. Things just keep getting bigger and bigger too many things to call out, but there's some interesting things I think in 125, one, uh the default registry location has changed from kate's.gcr.io to registry.cape.io. Everybody should use that when they're playing images now it's going to help us with infrastructure costs and just the maintenance of the project going forward.

C

Q proxy is also industrialist now, which is really cool that came out on the 23rd of August 2022., so a little bit after kubecon. Currently next slide. We are in the middle of the 126 release, so we just keep going. This is going to come out on December 6th, hopefully uh with no problems coming down the pipe. The lead for this one is Leo and the Emeritus advisor is nabaroon and I think this is super cool hit. The button. This is the first release.

C

That's been entire, really LED from outside of North America, the release lead and the Emeritus advisor you can kind of see like we've. Had this really great International presence, lots and lots of countries highlighted here over the last few years, but until now we have never had a release that was entirely LED outside of the US, so I think that's really a cool milestone for us this year, we've also been focusing on a few things in our roadmap.

C

When you get these slides, you can also find the roadmap on GitHub and we're really focusing on three areas. We want to make releases more consumable, easy to easier to pull down easier to get the artifacts for them more introspectable, so you can kind of understand, what's happening what's in each release and then also more secure, we want to make sure that the artifacts that you're getting are non-falsifiable. We want to make sure that they have signatures.

C

They have software, build materials and all those things that you have a better understanding and a little and a better way to judge the risk of consuming those those artifacts and knowing what you're deploying okay.

C

On the introspectable side, um we we wanted to look at the the Cadence and kind of follow up on our change. Obviously we used to do four releases a year, we've moved to three. We wanted to know how folks felt about this, so some some data that we gathered uh most folks seem to strongly prefer three releases a year. How many people here miss four releases?

C

Absolutely nobody who who deploys kubernetes and is like really happy that there are less things to keep up with um I know. I did uh in my last job next slide.

C

uh Going back to registered.case.io there's a lot of work going on with Sig Kate's infra to move away from uh that original GCR instance. This is a lot of work to bring community-owned infrastructure into play, but also to to really work on Distributing costs. There's an insane budget for kubernetes infrastructure and we spend a lot of it hosting container images. There's work being done right now to spread the load a lot.

C

So if you're coming from Amazon you're going to start pulling images from S3, it's going to go through a really transparent proxy, a lot of great information you can find in that cabinet and then last one we're going to turn source code into artifacts. Now so Sasha's going to take over and tell us how we do that.

B

Yeah all right, so the first thing- that's not just this one. So the first thing we would like to outline is what is necessary to actually cut a release. But the first thing really is that we need some personas who actually do the release cut and we use. We have two approaches here.

B

So the first one is that we have an actual release cycle which goes around about three to four months, and we tend to release three times per year, and then we also have the patch releases which which have the yearly support period, and we also cut patch releases mostly every month, and every second week of the month is the main is the main goal. So we actually need someone who takes over the responsibility for cutting a release right.

B

So we also have to need persons who have the time, the right time and the dedication and also are encouraged to do the many other steps for the documentation and verification of the actual release. So this can take up to multiple hours per release and also release managers tend to go and to release multiple patch releases in parallel. For example, so what who do we have on board? So the first role is that the branch managers and their shadows and those folks get selected per each release cycle.

B

And this is a highly privileged role because they have full access to domain kubernetes repository and they also have to not only cut the releases.

B

But I also have to pre-plan the releases like they offer the better and the release candidates, and this should be done together with the release team leads and, and they also have to coordinate between other other parts of the release like, for example, if they have to coordinate the release of uh before it's actually being cut with CI signal that that we can ensure that the CI, for example, screen and on the other side, we have the release managers and the release manager. Associates.

D

B

Those role is a more permanent role in the release engineering subproject and they also coordinate between cutting patch releases right. So we just have to um announce that we, when we do the next upcoming patch releases, we also have to take care of defining when a release goes end of life and things like that.

B

They also have to maintain the release branches, which means that they have to review the Cherry picks and if they got approved correctly and if we don't sneak a feature in, for example, into a release other than that they also actively develop features and maintain our code, which gets larger and larger over time. So this is also a very ongoing topic for us. They also work closely together with the security response committee.

B

So if we have any security vulnerability coming up in kubernetes, then we have to yeah, maybe schedule out of band releases or things like that, and that's also part of their job and the release managers. On top of that also Mentor, our release manager, Associates group, so which steps are involved to cut a release. So the first thing we have to do is that we have to create a GitHub issue, which is our foundation for documenting how releases will will be cut. So it contains all information.

B

We need to actually build the release and cut the release and it links any blockers or follow-up work and also it references under which conditions. For example, the CI signal has has been the release, cut so failing tests and outlines things like that.

B

It also lists which Google Cloud build jobs run and so that we can just see if we have any build issues or things like that, and we also create a after that. We can just create a corresponding select thread thread to track the release just just in time.

B

After that, we also require some help from the Google build, build admins and have to check for their availability, because they help us to build push inside Debian and rpm-based packages on app caseio and yum case Iowa. And if we have been on all of that, then we can actually State a release and I. Think it's the first time that we now will do a real world release stage live at the kubecon and Carlos can help me with that.

B

So what we do now we use our Krell release, toolbox um and Carlos will just run trail stage. Usually we would have to Define which type of release we want to cut. So it's an alpha bit our release candidate and also from which branch we want to cut the release, but per default. Everything should go real now or not.

B

Yeah I call this has to authenticate against the repository again I'm. Sorry about that live demos, yeah.

B

So we can see that we already gathered some pre uh some information. If you go a little bit more up into the logs, then we can see that we already find out which version we want to cut so there's some automate things. For example, we also have the the Cube cost version, and things like that, so those are all variables which have influence on the release cut and we can Define them or change them like we want.

B

But if call us now, there's a link to the Google Cloud build console there, and this brings us to the release and usually nobody other than the release. Managers or their Associates can see. Those looks, and you probably have to just refresh, because it's cured yeah, so we will not wait for the release to be finished now. But if you look at the execution details, then you can see yeah. Then you can see uh that the cloud build variables have been set successfully. So we have to.

B

We have created this job, this one's on a virtual machine made mainly, and the logs should now just print out anything like we. What we have to do for career stage yeah, so it puts container images, starts building the release and things like that. So if we go back to the slides.

B

Then we can see yeah, then I created an overview about what credit stage actually does. So it runs this predefined, Google Cloud build shop and it also checks the prerequisites for the for the actual release so, for example, determining the release version and the text and the repository is kind of tricky because we have to find out which release we want to cut. So we could also imagine that we wanted to create a release. Branch or another release, Branch something like this. Then it actually builds the release.

B

So it creates the binary, artifacts and container images, and it also generates the bill of materials which is kind of new to us. Then we also verify those build artifacts.

B

For example, we double check if the architecture has been read correctly and we also built a provenance attestation for this build step, and after that, we just stage everything we have been built into a Google Cloud bucket and leave it there and because now release managers have to actually publish the release, and for that we have credit release, and this build version is the unique reference to an actual release which then just will be released and pushed into the actual locations. So what does scroll release? Do it verifies the artifact provenance for Crest stage?

B

It also pushes the artifacts into their final destination like container Registries or Google Cloud buckets for binary artifacts, and it also pushes the stage. Git object objects, for example, so we use the kubernetes repository to, for example, to can be generated, change. Look that we committed to the repository and on-care release. We will actually push those git changes to yeah 2K.

B

Then we also create a release announcement, which is just a plain text: file and update GitHub release. For example, then we build the provenance attestations for career release and, after that, we archive the release on our Google Cloud packet.

B

So if everything then goes well, then we can do the same again, but now in production mode and if that's successful production mode mainly means that we just use different locations for container images and binary artifacts.

B

And if that's successful, then we can do the image promotion after our credit stage, and for that we have a different set of tools like K promo, our part of our promo tools, which is able to create a GitHub pull request to a repository and which outlines which container images we want to promote to production, and if this pull request got merged, then this the job running after this pull request will actually push the container images from the stage bucket to the production bucket.

B

And then those container images are live, and if this is done, then we can also run credit release and production mode. And that's now the point in time where we have. We need the help from the Google Cloud build admin to cut the dab and rvm packages for us. So those are, they will use the pre-built binary artifacts to create packages for it, and then we can also notify select that our release is now finally done.

B

The announcement we previously created can be then can then be used to push the announcement to the official kubernetes kubernetes mailing lists, and we have a dedicated subcommand, for example, for that, so credit is just a toolbox of use cases before you can just use credit announce and then it announces the release, but what's inside of a release, so we have binary artifacts for the kubernetes client and servers, and also for the test binaries, which is useful for integration testing and for every supported platform.

B

So we built for a huge I think it's five or six different supported platforms. We have right now, and we also do the same for the for container images which basically reuse the the binary artifacts and those container images are signed, which is new right now and we are also working on getting the binary artifact designed.

B

Then we also have the salsa provenance metadata for the stage and release steps. So those should be part of the release to actually verify what has been done in trail stage and release and also an updated. The updated kubernetes repository is part of the release, as well as well as the staged sources, and then we also have the software materials for all of them.

B

But how can we trust those artifacts Germany.

C

Everybody can talk so, as Sasha mentioned, when we do the release process, we stage things and we produce a set of images. A set of binaries things that land in the staging environment. We want to promote those things to the production, um places that they're going to go to.

C

uh Oh sorry, we sign them also, so we publish the images and signatures to a staging GCR. Those are going to be the pla. You know the places where we can kind of verify things and where they live until they go into production regions.

C

Then, as Sasha mentioned, we do a pull request. That's generated by K promo, that's going to list out all of the images that are going to come across. So the things that need to be promoted from the staging environment, to a real release are enumerated in that pull request.

C

So the first stage of trust that you get here is really that we've got a small set of people that are able to generate these releases. There are trusted set of folks in the community and then they're reviewing or generating a PR to do a review of.

C

What's going to be promoted into the release environment than other release managers so again, The Trusted subset of folks, but not the person doing the release themselves, is going to review that pull request to make sure that the things we're bringing across looked good and that we're going to bring you know, things that are are going to be valid and trustworthy inside of the the project.

C

Once that merges, we kick off some automation to do that. Actual promotion process. That's going to take the images from that staging GCR and it's going to put them into the production GCR and before that's kind of where things stopped. But now we also, since we are signing those images, we're able to verify all the signatures before we copy them across. So we get that extra level of kind of um verifiability as part of the process.

C

So you get that two-phase kind of uh security, guarantee that these images are the things that the release team has built and is ready to consume. Then. Finally, those things are pushed to the production GCR and again signed with the production uh service account so you're able to verify all of those things and see the the kind of the chain of custody. You'll see the signatures from this agent environment and then you'll also see the signatures from the actual production push.

C

All right Adolfo: do you want to tell us about how end users can verify these things.

D

D

uh Well, once we sign the images they get, they get their signatures attached in the conservatory industry. So the way you can do the verification is by using the cosign tool from sixth sort and you the command is really easy. You just have to run cosine verified and pass the attack of the image, and that should get you an actual verification of the image. Once you verify them, if you inspect the signature, that's spread up from cosine. You see those two signatures attach the damages which Jeremy was just talking about.

D

You'll, see the the one signature done by our staging account. That means that tells you that the image was actually I will release plastic release and a release manager, and then you'll see the second signature which all of the images across the kubernetes project get um then uh do you want to continue all right and then the other one? Is uh we we attached? Well, we release. We are not touching.

D

Currently, the yes one to the images, uh there's some work that has to be done yet in the image promoter to to enables us to do that. But we do also do release a complete test, one with all of the with all of the um uh with all the images, and we have a tool that was derived from all the work we did to get. The kubernetes has been going and it's called bomb which you can download and use it to generate an esbom of uh your goal project.

D

If you want- and this is, if you run, uh if you inspect the yes, one of an image you can see inside of the woman, allows you to inspect the image and see how how it is well, it allows you to look inside of the esbom and see the structure of the esbom, all of the components inside of it and the rest of it.

A

Okay, we mentioned about like having our own infrastructure and what that means. Actually, in the past, everything was managed by the googlers in the Google gcp project and in the past years we start moving back to a community one at one and what we have right now in the community side, one we have all the containers we have all the binaries and the S Bonsai provenance in the GCS packet in the registry, as Sasha and Johnny mentioned like when we cut a release. All those things goes to the community bucket and the community registry.

A

But that is one piece that is still on the Google honored infrastructure, which is the Debian and RPM packets. uh After we like in the last mile of the release, when everything is almost ready, we need some support from the googlers to build and publish the package for us, and then they build sign and push and those parts are ready and- and this uh Debian and RPM package is now like, we are working to move away from the Google honored part to be like a community one as well.

A

Then there's a current poc in progress for how we can build using open, open, build services, and- and we are working like to remove this as soon as possible from the from the Google and to be a community one. If you want to like to to participate and help to work on this, you can join the slack Channel as well.

A

D

The next step, so obviously as Sasha was mentioned, we have a team of release managers uh but we'd like to so we would like to invest our time in making the life of this managers as painful as painless as possible. Sorry, so um there's a always always have ton of work to do, and if you like to join the release, we are more than welcome. So this is a beautiful review of the efforts that we have going on and how they align with our roadmap.

D

So uh and, as Carlos said, we are working on assigning the release artifacts that that uh uh that's what the the release the system packages uh we also are in the middle of uh moving and trying to imagine that POC and trying to find the shape of it, the final shape of it.

D

um We are also trying to come up with new ways of reimagining how the image promotion should be this mainly because of the new infrastructure requirements that we have, because the the of the infrared changes that are happening from moving from Google owned things to Sig release and Community manage influence. So we we're trying to optimize how the process runs, but also maybe coming up with new ideas on how to make that process more more seamless.

D

And then we are trying to come up with um like tiers of recommendations of how to handle first, the our internal repository. So we have like huge repositories that we release like kubernetes. Then we have middle tier ones and then we'll have the smaller, which sometimes are just a library, so we're trying to set the guidelines on how we should handle and try to to how we should build the releases for those. Why are we trying to do that?

D

Because we want to one first Our Lives easier, but also we would like to be a little bit more of a Guiding Light for the the smaller projects across the org that maybe some of the tooling that we are releasing, can help them. And then this is why one of the things that we are about to to release for in the next few months before the next cycle ends is a new GitHub options.

D

Ripple capturing the tooling that we are building inside of Sig release so that anybody can consume them inside of their actions uh workflows more easily.

D

um Now uh we are also working on uh improving some of the way that provenance metadata is built inside a release process.

D

Currently we are building metadata in the wrong places, but we'll have a new utility out that will and it will enable us to query the build from the outside, try to observe it and collect the necessary information from a safer point of view- and this, of course is in aiming to finish our our salsa cap that we have still going um then I have another one one another one that we want to do is start creating other stations from the image promotion process.

D

So we have, you can verify the signatures, but we want to also let you know what happened there and who did it and um we, as such I mentioned earlier, we are working on on signing for the rest of the artifacts, so there's an effort going to sign binaries the s-bombs and the provenance metadata that one is already done, but it still needs to be put into production and some size undergoing a few changes. So the framework is getting split to be, make it easier for our organizations to to achieve compliance.

D

So we need to rework the Gap once 1.0 merges so to make it more aligned to that and we uh we need to build. We have a utility called publish release which should verify the the artifact is it's bullishing and we are working together with what used to be the spdx official s-bomb generator to bring multi-language support to one bomb.

D

Our bill of materials tool currently works with go projects, but we have organized with a couple of other S1 projects to create one single repository of language parsers, so that we can extract language dependencies in the same way. In all of them, and that that multi-language support is coming to one so there's like about 10 languages, python Ruby, roast the usual uh The Usual Suspects are there. And finally, uh one thing that I'm really happy to announce is all of that work that we are putting putting into the kubernetes releases.

D

We always try to get it out and expose it as publicly and general purpose, libraries and tools, and- uh and but we until now, we didn't have like a clear story of why those bits are important and how to use them. And how do they piece together so, together with the actions repo that we are building and we're also publishing a seek release guide to how to secure your list. And basically this is take all of our tools play. Make them play well together and you'll improve your your release, uh security instance. Quite a bit.

D

um It's uh I'm I'm. My plan is to publish now the earlier draft so that other people can come in and comment on the document. So if you're interested in that hang out in seek release and the secret is Channel kubernetes and we can chat about the document- and you can add your suggestions to it,.

B

Thank you all right now. We want to speak about how to get involved and how to help us with all of those efforts and the first thing we really would like to recommend you is to apply for release team shadowing experience for the 127 cycle. It should happen this year, maybe in December, so there should be an announcement on kdev and if you want to get to know which rows are already existing, then we linked it here as well.

B

So if, for example, you have technical rules like a CI signal or also the release notes, team is also kind of a technical team, but we also have non-technical roles, for example like croms or dogs and others.

B

So if you really, if you're, really interested in one particular area for of the whole release, then the release team shadowing program is the best way to get started with kubernetes and also provides you a good way to maintain an overview about the project at all and what is going on because you indirectly get the knowledge from the others as well.

B

So, but if you're interested in even more technical aspects of a release, then this can be done by just showing more than by becoming a release manager associate, for example, so it can show in the release team and then also later on Shadow for a more technical role. And then you can just express your interest that that you want to be a release manager associate.

B

So this also means that you can start contributing to our repositories, getting involved into our discussions as I've also already mentioned, and staying just up to date and release management or the sick release lecture.

B

um It's also. We also have a huge bunch of recurring work like updating, golang versions or keeping dependencies other dependencies up to date. We also have a bunch of work which is related to releases, for example, if a new cni release got published and we have to update the manifests for the for the packages which use it and things like that, so there's a amount of recurring work where everyone is can express their interest and start contributing to it as well.

B

Last but not least, it is always encouraged to uh connect to existing release managers to find more areas to contribute and yeah with that. I would like to say that that we're always happy to help, so they will. From my point of view, the release team shadowing program is the best onboarding experience in the community and I started with that as well, and sick release is always well coming to discuss any process or technical changes into any direction. So we have a lot of.

B

um For example, we have a retrospective for each release, which is split into two retrospectus for now and where you can, in the mid-cycle retrospective already expressed that you maybe think that something should be changed and that's always a good thing. So we totally welcome this change and our overall goal is to be most inclusive as possible, just to lower the barrier for new and also for existing contributors.

B

So we are always looking for new contributors and we really would be really cool if we uh achieve that at this cubecon and with that, I would like to thank. Thank you for listening to our talk and yeah. We are now open for questions right or do we have any time left.

D

uh No, uh no, actually, no, we have quite the big investment in the way we have done. We have been doing in in cloudville, so for now it's going to stay there, mostly because I mean, even if we were to consider a change to move things into a change to move the things into into tecton.

D

uh We are spread thinly, it's in our cities, so if so the the main bottleneck that there would be people, so it could be done, I'm, not sure if we would gain a lot from it, uh because we've built tooling around GCB to make all of this, and this happen. So since we don't have enough people, we cannot think about change like that. So far,.

D

Like a train out, there.

C

There is a lot of automation already. There are just a lot of pieces. Some of it requires manual review like waiting for PR reviews to get the image promotion. Pr emerged. Some of it takes time to get Google people to to do that last step of the mile, we're working on fixing that part, but right now, that's a pretty manual process for them. So there's just coordination that happens, but there is a lot of automation already, the the tooling um the built for Krell for the actual image promotion process.

C

A lot of that is automation um and it just kind of comes down to the the scope of how many things we're building how many places we have to push that to now and and really just kind of overseeing that process. The on the release team side. There is a lot of work too. um If you went to the keynote this morning, I think Priyanka mentioned Dr Taylor. One of the previous release leads about how much time he spent doing things, and a lot of that is people.

C

It's it's working with different sigs to coordinate which caps are going to land, make sure that PR's emerged. Things are kind of disjoint too, so there's there's just that kind of getting people to the the right place. That's a lot of the work that goes into the.

D

Yeah go for it, uh so yeah as Jeremy was saying. Stick release is really split into two. So there's there is engineering, so project and the release team and we try to and in the resting we put all of our efforts into Automation and building the tooling on all of that, but there's also the human parts of the release.

D

That cannot be automated, like writing blog posts like uh engaging uh ensuring that people get their enhancements at time and all that, so, even even if we, if we were to automate fully the release engineering part, the the human part is not, but it's also one of the most rewarding. Because that's where you get to know all the community members allow people help them grow. So that's that's the best part of the of seeing release to be, in my opinion, at.

C

Least, on the releasing side, though, there are opportunities to improve and make some automation happen and Grace, and some other folks working on enhancements, have done a fantastic job. Spinning up this new GitHub board, so we're using the the new github's projects beta to really streamline and make the whole process of tracking enhancements easier before it was a Google sheet that had the most complex automation, I've ever seen to really track things. Things are in in the KK repo things are in the K enhancements. Repo and you've got a the docs repo.

C

The website repo you gotta like keep all these things in in line and before it was a sheet. It was mostly automated through crazy scripts and then somebody on the release team going through and entering data or somebody from a Sig going through and entering data, and it was not super sustainable um and it's so much better. Now.

D

One thing that I'm interested in this image promoter: how versatile is that?

D

How versatile, in the sense of can it be reused.

D

Well, in theory, you could spin it up and use it for your own as long as you are running on either GCR or artifactory. Usually this is the technical reason behind that is that it doesn't specific Google specific calls to the uh to the registry in order to list how the images are there, I've been trying to get it to work with others and it's uh well.

D

If you take a look you'll understand, so it's uh it's not easy, but if you were running in that setup you could use it for your own organizations, and that means you can get like pull requests and free signing with six stars.

D

Okay, yeah and if you want to handset, to make it more vessels up, please I.

C

Mean you've used. Inspiration too, which is the the way that it works is a is a pretty good model. I think you could build your own for whatever environment you're in take inspiration from the code. That's there.