Cloud Native Computing Foundation Cloud Native Security Conference North America 2021, 30 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: It's Time We Start Securing Our CICD Pipelines - Shripad Nadgowda, IBM Research

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

It's Time We Start Securing Our CICD Pipelines- Shripad Nadgowda, IBM Research

Containers allowed breaking monolithic applications and business logic into modular components that can be developed independently and quickly. Such an accelerated development pattern then required a high velocity path from code to container that gave rise to innovation and automation in CICD pipelines. CICD pipelines aim to facilitate expedited DevSecOps functions like testing, security scanning and delivery of applications to cloud through automation. At the same time there is a growing open-source ecosystem around CICD technologies, where number of such functions are being made available ready-to-use, like tektoncd-catalog, GitHub Actions Marketplace for instance. As a result, our pipelines are also subjected to prevelant supply chain vulnerabilities, wherein some malicious open-source task could temper and compromise our whole pipeline. Thus, as we are building CICD DevSecOps pipelines for securing our application builds and delivery, our pipelines as-is can not be the root-of-trust. In other words, we need to make sure our CICD pipelines are “secure” at the composition to begin with and need to address security at multiple layers.

A

Okay, uh hello, everyone, uh my name is sripad nada, I'm a senior technical staff, member at ibm research and today I'm going to talk about how we secure our ci cd pipelines.

A

Now, let's revisit this office supply chain right, so it typically starts with the developer. Who is writing our code and while writing the code, we declare uh it declares various dependencies like the package dependencies, the image dependencies.

A

Then we run our ci cd pipeline in the pipeline. We we do various security and compliance functions like we generate s1. We do vulnerability analysis, we do validation of all the dependencies, whether they are coming from trusted source and not then. Finally, we build some artifacts, typically an image. We push it to some registry and deploy it onto the onto the cloud now.

A

The supply chain security applies, is applicable across the spectrum, from developer to the workload that is running on the cloud now in this talk, we'll be focusing on the security of our ci cd pipeline because, as I said right, this is the place where we are embedding and we are bringing the innovations of how we do a lot of security analytics and how we secure our code. But it's also important that we we make. We give due diligence of basically ensuring our pipelines are secure, then stuff right.

A

So that's what is going to be the focus of this talk so just like our applications, our pipelines also have their own lifecycle. It starts with the composition. Now this is the place where we are composing or we are creating our pipeline. We are defining our pipelines. Now there are number of open source uh catalogs like. If you are building a tech town pipeline, then you have tecton catalog.

A

If you are building github actions, then there is get a marketplace so we'll most probably when we are defining our pipeline, we'll be using this pipeline definitions which are ready to use right. So at this point we need to make sure that when we are bringing in these pipeline definitions that we we do the do validations of this, whether they are coming from trusted source.

A

Can we trust these definitions and also we need to ensure that when we define we configure them securely, then we set up or we configure or install our pipelines right at this point again, we need to ensure ensure that our pipeline is properly configured because our pipelines, they have access to a lot of credentials right. They have access to our registry, their access to our github credentials, their access to our keys.

A

So it's important that when we are setting up or installing a pipeline configuring pipeline, we make sure that they are securely configured.

A

We can again use some admission controller to ensure that when we are installing, we are only allowing this design and the verified artifact to be configured on the pipeline. Then. Finally, our pipeline gets triggered right it may it could be a manual event, it could be a github event and our pipeline is basically is ready to execute.

A

Now. This is the last point where we need to do the verification again, and we need to ensure that the pipeline that we are going to instantiate or execute it is safe to uh trigger, except to execute right. We are done all the verifications and now we can execute the pipeline then. Finally, once our pipeline starts executing, we need to ensure that we monitor the uh our pipelines, because, in my personal opinion, monitoring is probably the under appreciated security tool.

A

It doesn't help preventing any security uh incident, but it is probably the most mature and oldest uh tool that we can use to discover if there are any uh uh any issues with your pipelines or any issue with your applications on the cloud right. So that's why I think when the pipeline is executing, we need to ensure that we have proper monitoring established.

A

Then, once our pipeline finishes, we need to collect the automated registration of the execution state and any artifact that is produced by the pipeline right and once at any point in time. When the pipeline is in the history, we need to ensure that we have the ability to audit this pipeline and there are exists, and we don't want to start from scratch right. There are open source projects that that are there that are trying to solve some of this problem.

A

For instance, these are techno chains uh that can capture the attestation of automated registration of the task run and the images that are produced. We can sign it automatically. uh I also put in a tecton extension protocol proposal to extend this capability to the pipeline run, so we can have the end-to-end province collection and we can improve on this. Then we are existing cloud native monitors that we can use to basically monitor our pipelines.

A

Only thing we need to do is we need to do the right instrumentation of the our pipeline components so whenever, for instance, the takedown pipeline, then, when all the com, all the parts or all the containers that are executing they have the common label. So we can aggregate and view them in the through the common lens and for rest of the stuff. There is some work in progress that I'm basically working on, and this is again working with the open source community. So if you are interested, we welcome your feedback and help.

A

So one thing that I'm looking into is pipe validate right, so just like our container or our kubernetes, uh they have their own cis benchmark that provide them the guideline of how you basically configure your cluster, how you configure a workload. Similarly, we are trying to come up with basically a set of guidelines that how you can basically configure your pipelines if how you can identify any misconfiguration in your pipeline and again these are going to be codified. So you don't.

A

These are not going to be just guidelines, just ensure that you are using this latest version of this image on. These are basically actionable ones that we can codify and we can uh validate them in in our through our automations.

A

Then there is the animation controller that we are looking into, uh that that will again prevent uh execution of any providers. The ability to validate and verify the pipeline executions and pipe auditor is again some automation that we are doing to allow us, the validation of any artifact that are produced by the pipeline.

A

You can validate how it was produced which pipeline produced it and what texts were done in this one in this particular talk, I'm going to talk about the sign signing part of it, so why it is important uh is let's say, I'm building a simple takedown pipeline right. I have three tasks: git clone vulnerability, scan and build image.

A

Now, if one of the tasks and I'm bringing this task from the open catalog now, if my git clone task is compromised now in the sense that it is comp, it is tempering with the original artifact that are in the gate repository and it is as a part of the cloning now my remaining tasks. They are automatically compromised because they rely on what the first git clone task produces.

A

So if one task is compromised, then my whole pipeline essentially can be compromised. So that's why it is essential that we have when we are bringing in this open source element into our pipeline. We make sure that they are signed. We make sure they are coming from trusted source and that's what we are doing in this pipeline signing utility again. This is open source. I encourage you to basically see this, so the approach that we are taking is we are assigning this in multiple layers right.

A

So this is again example of takedown, but it applies equally to the github workflows. So first thing we do. Is we assign the pipeline definition now? The pipeline encodes, the task layout and the shared resources?

A

So we by signing it, we ensure that these are the only authorized tasks, and this is the order in which this task can be executed. Then we sign the task which encodes the execution logic, that these are the execution function that I'm going to run, and these are the runtime of the base image that I'm going to use.

A

So when we are signing, we are assigning it in this multiple layers. We sign the pipeline, we sign the task and the image separately and the uh we are basically taking two approach. One is, we are signing the uh the yaml, and now we are basically building new approach where we are converting this task, uh the techno resources into into a format, a representation, and then we are signing it with this six door right, so we are signing it with cosine and all the artifacts that are basically represented in there.

A

I just want to show you quick demo, so this is again. As I said, this is an open source project called tapest pipeline. So in here.

A

In here I have basically my pipeline definitions in this particular directory and I'm basically saying tkn for tectonic show. This will show this will pass all the definitions. It will identify. What are the pipeline? It will identify the layout uh the task that I use the images that are used that are used. Then I can go ahead and I can basically sign this.

A

I can sign this particular resources with with the my key, and I can point it to the registries where, okay, I think I'm running out of time, so I'll just show you one uh command called verify. So by verification. We can essentially say this through essentially allows us to statically uh sign the pipelines and verify it in the uh in your in your workflows right and at the same time, I wanted to show you a simple admission controller, where if we basically try to apply some pipeline or create some pipeline, that is not sign.

A

We have basically admission controller that can identify it and it can basically deny the request that signature or message annotations are formed, so we can block this execution of the pipeline.

A

And, finally, as I said this, the code is open source, so you can find it at this particular location. We have this. The talk that I give. We have article publish also when you start and if you want to basically we're looking for help right, because this is a big spectrum. So if you are uh interested contact me by email on github or on twitter yeah, thank you.

A