Cloud Native Computing Foundation SigstoreCon NA 2022, 3 Nov 2022

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: The Road to SLSA4 – Applying the Sigstore Ecosystem in a Corporate Environment- Alex Ilgayev, Cycode

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

The Road to SLSA4 – Applying the Sigstore Ecosystem in a Corporate Environment - Alex Ilgayev, Cycode

Sigstore’s ecosystem enables signing, verifying, and protecting software artifacts in a new way. By doing so, we can confirm that the software is what it claims to be. As part of the rising concerns of software supply chain attacks, we decided to adopt the Sigstore tooling, integrate it as part of our environment to increase the integrity level of our software artifacts, and share our insights of the process. In implementing Sigstore's ecosystem, we encountered several challenges that may be common to other organizations - our artifacts run on cloud-native, self-hosted, and even on-premise environments, and we use several build services, including a self-hosted K8s cluster. During the talk, we’ll explore the following concepts: - The trade-off for self-hosting rekor/fulcio instances against using public ones’. - Implementing “keyless” commit signatures with the gitsign utility instead of standard GPG. - Developing methodology and tools to verify commit signatures. - Using spiffe/spire to give our ephemeral build workloads identities. - Utilizing OIDC tokens for keyless signatures on artifacts in various build environments. - Developing methodology and tools to verify artifacts.

A

Okay, hello, everyone, so today's topic will be the road to start the floor, playing six dollar ecosystem in a corporate environment.

A

And in this talk we'll look on sixer on a bit of a different angle. We look at it from the corporate perspective in the past. In the past month me and my team in cycled we're looking extensively at six store, researching and in order to understand how he would take it and and apply it in our internal CI environments, and we wanted to achieve two goals.

A

We wanted a to sign or artifacts through the CI carelessly, and we wanted also to to use git sign to use git sign as it should and to easily sign all our commits originally also Gabrielle ahead of devops inside could work on this topic, unfortunately, couldn't come to the to the conference.

A

So, okay, let's start with the agenda, see this. This is a sixth year conference, so we will start explaining what is extra? I, don't think it's much necessary, but we will explain what it's texture and even more important, why corporate should adopt it.

A

um We explore the sixer tooling, and then we will go into a understand why what the issue could be have into uh into working with the public infrastructure of six store in terms of privacy and how to create maybe self-hosted, secure infrastructure and also why?

A

um Then, we will explore some, how to use sixer to sign artifacts in several real world CI environments and finally, we'll talk about some of the insights of this of this research.

A

um Originally when I applied for this talk. I also wanted to to talk about the verification process and maybe more thoroughly talking about git sign and the issue of currently adopting Gita incorporates. Unfortunately, I don't have well. I won't have enough time to talk about this in 20-20 minutes, so maybe in the next talks.

A

Okay, so I'm Alex, El, Geib, I'm senior security researcher at the cycle previously I led the Marvel research team at the checkpoint, where I was.

B

A

A lot of reverse engineering for complex pieces of malwares investigating attacks both for generated in cyber crime and apts, and nowadays I investigated research of vulnerabilities and mitigations for software supply chain. Part of cycle cycle is a cyber security company that provides a complete sort of supply chain solution for organizations, foreign.

A

With all the great talks we had there so far, um but basically a new standard for signing very fine, protecting aspects of software development development. It was a originally. It was built, especially a specific specifically for open source projects, but we will look at it from also a bit of a different perspective.

A

um Okay, so what would be the motivation for a for corporate, for example, our corporate, when we look at it to into adopting uh adopting a six-store tooling? So let's say: let's take an example. uh We have two main cases for such corporates. The first one is a uh using uh internal artifacts that are built in the CI environments of the corporate which are deployed into the internal production environment of the corporate.

A

um So we can, it can be used to validate artifacts before deployment by the corporate itself and I'm, focusing mainly on the keyless signature. For example. We could verify if we have like several personnels that are authorized to uh to sign artifacts because verify that these were the only ones that were signing on this artifact before deployed to production.

A

Also, it can be also used to validate the artifacts before being used by the external clients. They could be a clients that, using this the components of of the packages or they could be used also as a yeah, also our clients like open source clients. So all these packages could be can you must be signed through the CIA environment of the corporate and be verified uh that this was signed specifically by the corporate, for example, email address or someone inside the inside the the development process?

A

A sum of additional benefits could have. It's also could satisfy some of the requirements for salsa, similarly to the um to the South Providence Builder that they're using a reusable workflows to satisfy non-falsified requirements and achieve the success of free. So you could also use the cosine, for example, to sign additional pieces of information.

A

In addition, we could also adopt a git sign to sign, commit and tags and to allow tracing the origins of the code insertion.

A

This could help to remove the burden of handling gpg keys and maintaining them across a large Enterprises, and also it allows you a higher Integrity level for each and every commit inserted to the code repository and we even satisfy the some sales requirements. For example, the verified history.

A

So, let's go over quickly, some of the six or two link to understand how they work and then we'll see. How can we use them in some complex CI environment?

A

So the first one is record. There are a lot of been talking talking about a record here. So basically it's a it's. A immutable temper resistant Ledger of a metadata built on top of a trillion tool by Google. Basically, it's a it's. A public infrastructure holds the public infrastructure of record that every work can add the in the metadata to it and and query and verify that um query and verified.

A

um So in six store, it's mainly used to store signatures for artifacts and also the certificates that are the short-lived certificates that are have been used to to sign to sign that artifact.

A

Another additional important tools for sale, which is the root CA that that's in charge of signing and issuing the short-lived certificates, the poster. Basically it's a it's a you. You give it some oedc tokens so open the token that proves your identity.

A

To check whether this token was is valid through some configuration, it has it that you can check it through a several identity providers. It would be a GitHub Microsoft, Google GitHub actions could also use the Dex identity provider, which is an open source, popular identity provider, and, if polter found that the token is valid, so it will issue it will issue a short-lived certificate according to the identity given identity. From the the token, so one of the important factors about fulcio- that's it's its strength is, is it depends on the root CA.

A

So if the root CA is a, is a compromise or isn't secured enough, so we can't really trust the full sales. So it's a very important factor into whether you want or not to erase your own self-hosted version of a full C.

A

Kosan is a it's basically the signing tool for for six store. It allows you to sign artifacts and maybe additional attestation data and it stores the data on the oci registry and find the git sign. I think we can skip that, for you explain it way better than I do so. If we look up from the corporate perspective, the using public instance of record could be quite problematic.

A

um For example, I used cosine to sign some simple image of mine. It's called Simple go server in this. It was stored on the public instance of record. We can see the the index number. Everyone can just go and query this uh this. This entry in a record extract the certificate from it from the century and get my my email number.

A

It could be personal copy corporate, and this could be quite an issue for a corporates specifically when also looking on git sign, which implies on the way and a lot more entries such that, and if the artifacts are private or the project as private, they could be an issue to expose a a lot of maybe email numbers. It could be also maybe identity of machines or GitHub action, workflows or whatever.

A

Additional issues they could have with also wrote fools, even though it's not publicly accessible to everyone. It's still the demands, the trusting, the Foster server to store your emails and your certificates. You need to trust the research fit or the root certificate and and the of app that that gives you the short-lived certificate.

A

So, even though I don't recommend it to do it for any any organization, it's still I wanted to experiment what it it means to erase yourself your self-hosted infrastructure for a for Rico fulsio and the rest of the six store and the best source was a sixth or the hard way created by look and for several addition, contributors and it it wasn't so hard, actually uh yeah, it wasn't so hard, but still I will mention it in the end it uh the most problem.

A

It was maybe raising the full instance, but it demands like maintaining the uh the root CA and private Keys, which can be a very demanding for a maybe smaller companies, so I, don't recommend it for every every organization. But if you want it's possible, so let's see some hello world for a simple signature using the salesforce.in infrastructure.

A

First, we'll need to Define several environment variables. We need to get the so-called new root. Ca, there's also a public key for a certificate transparent. The part of the fulsion.

A

um So whenever we want use it to just run equal sign for our image that we want to save, we give it an some additional parameters like the the new, so-called recore and full strength of uh endpoint.

A

This leads us to some logging screen similar to the to the public sector infrastructure, using we're choosing our, for example, a Google account or identities want to use it to sign the artifact.

A

And finally, when the signature succeeds succeeds we have like entry record that we we can query whenever we want and we get our the identity, the certificate which contains the identity which you that used uh that were used to sign the artifact. It also includes the issuer, which, in our case, it's of.cycle.dev that issue this certificate.

A

Similar procedure also, we can apply for a git sign. We need also to Define several environmental variables that defines the the new self-hosted infrastructure every commit we just log in through our Google account, which is Google account, and we we have this entry, that's saved on record that we can verify and say that it contains the certificate um on the name of the one who made the commit okay.

A

So now, now that we made some introduction of the sixth row tooling and maybe how to create a self-hosted version for uh for corporates, let's talk about, let's, let's look away and several the a common use cases of CI environments in in corporate and how can we integrate the sixer tooling, including signing artifacts, using that environment totally carelessly and- and the point I want to mention- is that we don't want the developers to know about it to be transparent, as we can for the developers themselves.

A

So we will explore it to uh two scenarios and the first one is using a GitHub code management with GitHub action. Ci, we'll use GitHub hosted machines for the CI and finally, we will push the artifact to a Docker up and and for the second case, which has been quite more complex. We will manage our code in gitlab and use gitlab Runner CI, but we will use a self-manage or manage it doesn't really matter if a private kubernetes cluster that will run the sci builds in by the polling.

A

A gitlab and this bill should build the the image and push it to Docker hub.

A

So how the architecture pre-stick store will look like I'm, a user I will be pushing code into GitHub. Github. Will provision me a new VM on each on each new build, and this is why job will push will build the image and push it to Docker hub.

A

If I will be pushing the code to a gitlab I have like I managed in our case GK cluster. It could be AKs or eks. It doesn't matter. There will be a managed cluster and a port inside the cluster that will be polling gitlab for new job. Whenever the new job will received, you will provision a new cloud and, and that pod will do the CI job, including pushing the image to Docker hub.

A

Okay, so first, let's look at the GitHub action case.

A

um A simple solution to this issue into integrating six store could be through your reusable workflows, uh very similar to the salsa provenance Builder.

A

Imagine a GitHub action, a workflow when the first with the heads two jobs, the first job is like building the the image and pushing it through Docker Hub and the second job will be adding a signature to the image it could be. It would be calling a reusable workflow and an external workflow that could be reused in also in several other builds an important factor that it should be sent a permission to a with ID token to that reusable workflow.

A

This ID token is is giving the reasonable workflow a jaw token as an environment variable that could be used to give identity to the signature process. We're also sending a parameter of the the image damage should be assigning so how this erasable workflow looks like first, it should be defining several environment variables. We said we want to to try it out with a self-hosted infrastructure, so we're giving it the right recurrent fulcio.

A

We should also be giving it the um the root ca for the full cell, which is fetching from some from a gcp bucket and inside the the workflow itself. It's very plain, simple, just signing it, giving it the right parameters and giving it a IDC provider of GitHub action. uh Fortunately, a cost time is supporting GitHub action as many other identity providers. So he knows how to fetch the the job tokens from environmental variable and use it to authenticate against the.

B

A

Actions, identity, provider, and so what will be the result? Imagine by running this CI build this your usable workflow? We are winning some our sample image.

A

So when we finish running, it will be succeed, succeed in in pushing and building and pushing the image, and we receive our entry in recall when we query this entry, we will see that it contains this certificate. A fullser is aware of GitHub action. It knows how to put the right properties in the certificate it contains handful of information. First, the issuer of GitHub actions, the workflow trigger the commission and the workflow name, and more so we were able to integrate, integrate a sixth part of our SCI.

A

It wasn't so hard, but let's look at the more complex scenario. For the second scenario said we are using gitlab and kubernetes executor for gitlab. This means that every every new job is provisioned as a new pod in in the manager or in the self-hosted cluster.

A

So to in order to sign keylessly the image in the uh in that in the self-hosted OR managed kubernetes. We need some identity in order to send to fulsion and receive the self-hosted a self short-lived certificate.

A

So where can we fetch that identity from we have several options? The first will be to use the environment developer called CI job JWT, which Supply to every git lab execution.

A

Github provides every every job. This variable that contains it's signed by the by the gitlab service itself, and it gives information about the the actual job that's running. It can be used to many many use cases. Many use cases are like using it to authenticate against hashico Vault and additional cases specifically I. Don't recommend it using here because it you can. You can regenerate the token for specific audience, like you can in the other identity providers, additional solution will be to use the gcp metadata service. I said we are using gke.

A

This means that there are the nodes in the cluster are running as a compute virtual machines on gcps or every VM is a can access to some metadata service and get its identity from a gcp. It's that validly signed by gcp.

A

We can maybe use this identity in order to sign the artifact, so we will have a strong authentication that the artifact was signed in our buyer specific nodes in gcp.

A

Um This can be a decent solution, but it won't be enough because, because we won't be utilizing that we have a kubernetes cluster and we have specific parts that were doing this build, so this solution isn't good enough. uh The third solution would be to using a kubernetes identity Service. As I said, we have like a managed gke cluster, so we can. We can ask kubernetes to sign the token and use that token uh it's also a decent solution, but it demands exposing the cluster JW cast endpoint.

A

So it should be a verified somehow that that the stocking is actually valid.

A

uh So the final solution that may be the most complex one is that use the specific spiral it could give us the best of both floors, both using the kubernetes and the commercial identity, server and the gcp member server, and we can authenticate nodes using gcp and authenticate wallets. You workloads using the.

B

A

Workload, the tester, so we saw the architecture, looks like it's quite complex, but we will go over it step by step.

A

First, we need to create some specific spiral: infrastructure there's, excellent, tutorials and quick starts for for that. I personally, I suggest that to install Aspire on on some uh in some different or different server than on the on the cluster you're using to build your images on, and you also installed the Aspire agent through some Demon set inside your kubernetes. So every uh every node that will be will have this spiral API and the build that will be running on this node will have access to that API.

A

Also, the spiral agent will be authenticating with the gcp metadata service and and a test as the agent do, the being part of our gcp project, for example. So when I install these spiral, agents and I went to the Spire server to see all the agent list. I can see. I could say that I identified the three nodes I have in my cluster, and each of them has a specific spiffy ID a H this this. That means that Aspire agent was a successfully authenticated against the gcp service.

A

Next, we'll have to Define some several selectors for the workload at the station. Up to this point, we tested the nose. Now we have now not attached the the workloads that will be provisioned on every new build.

A

So for that we're creating two, a new entries in the Spire server, the first we're giving a spiffy ID of build node to every node. That is specifically in our cycle. Labs UCP project we already attested the node. So that's how he chooses the nodes and and giving them the specific ID, adding additional entry with for a workloads that every workload was that there was that has parent ID of the specific build node, which is the previously defined and also in a namespace.

A

That's called gitlab Runner will receive this specific ID of Builder, which is specifically the given to a workloads. Potentially, we could add additional selectors here, the spiffy expire. It supports a lot of selectors and the more we added the more secure the process will be.

A

Next, we need to allow all the new pods with the video provisioned in the build to have access to the Spire API, because we are using a built-in gitlab image for the for the pads which don't have the volume of the spider mounted.

A

Gitlab doesn't have support out of the box for a spire, so we use the kiverno policy to mount a the Spire API. This means the spiral unit Unix socket for every every new product is specifically in the gitla brand or namespace, which is the namespace in our cluster that all the new builds will be created in so this kivano policies just solved this issue for us and finally, we need to configure the GitHub CI workflow uh to get the SV.

A

That's with it's like a identity token that we received from Aspire that identifies the specific workload so we're getting this svid we're sending it to full Co to receive a a short-lived certificate on our on behalf on our identity and we're using this certificate to sign the artifact.

A

Finally, this entire workflow- it's like quite complex to build, but it has zero, Key, Management and and building it you, you don't need to be aware of the identities on the underlying build system to build it.

A

So this is how they. Finally CI file in gitlab, looks like we are defining a batch of environment variables for the for the infrastructure, for we defining the spiffy endpoint socket and several six store variables. All these variables could also be defined through some caverno policy to be default.

A

The finally, we used the kaniko image to build the docker to build the image and we're using a cost time with our at our variables, with ODC provider spiffy and for our server, so, of course, in nodes out of the box to work with the spiffy spiffy API and gets the S feed and use it to sign the sign, the image.

A

So when we run it in gitlab, we'll manage to assign our artifact to push the the signature to a head to record and when we query Rico, we would see the next certificate certificate was issued to this spiffy idea of of the Builder node, which we defined in the inspire.

A

And it was also issued by idc.cycle.dev, which is the spiffy Authority.

A

So this is what the the CIA use cases that's some of the insights we had through the process, um the first regarding cell phones at fulsior, as I said it may become complicated due to a management of the root CA. It could be a continuous to do it and propagating the public fruit, CA and the certificate transparency look publicly to the signers and verifies.

A

Could Implement some maintenance work additional usages for cosine? Could we also do some artifactor station, so we could use the the the architecture we build right now you can use it also to sign whether it will be provenanced, s-bomb or an ability report or any any metadata we want. It will be attached to the specific artifact and also some regards regarding the manage the kubernetes cluster or any other environment. So um it can take quite some time to build.

A

The right solution can take some time and effort, but this effort can be also beneficial for not only selling the effect artifact, but also additionally, additional stations that could bring you to a higher salsa level.

A

So some takeaways from this talk. We were experimenting six stores a lot and we're trying to bring this. This knowledge to the conference and the bottom line sector is great.

A

um So, even though it was designed with open source in mind, we wanted to deliver that. It also can fit some organization to secure their artifact with whether they are internally used or externally by by other clients um creating all the infrastructure. All the infrastructure from scratch is doable, but it's not recommended for every organization.

A

Also, you can apply some hybrid approach, for example, creating only the record instance and use the full seal.

A

um And finally, the support of identity, Solutions in sixth or cosine, specifically, is is enhancing the capabilities to support some complex CI scenarios, like we've, seen in the in the species bar and kubernetes environment,.

A

Okay, that's it! Thank you.