Cloud Native Computing Foundation SupplyChainSecurityCon hosted by CNCF + CDF 2021, 30 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Keynote: Software Supply Chains for Devops - Aysylu Greenberg, Google

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Keynote: Software Supply Chains for Devops - Aysylu Greenberg, Google

Several recent high-profile security incidents were due to compromised software supply chains. Software Supply Chain is a collective term used to describe the stages of software lifecycle from source to deployment through CI/CD pipelines, and all the static and dynamic analyses in between. In the world of microservices and cloud computing, trust in your company’s supply chain is critical, as most of the tooling and dependencies are from open source and vendor projects. When the code hits production, it’s essential to have enough observability to detect and investigate the problem and get to the root cause and mitigation as quickly as possible. With software supply chain attacks, not only is the newly deployed code under suspicion, but also all the tooling used to produce it becomes a potential attack vector, so an efficient and effective way to verify the integrity of the supply chain is paramount. This talk will discuss what information needs to be collected to allow DevOps to inspect and verify the integrity of the supply chain, the challenges of having the right level of detail to reduce mean-time-to-detection and mean-time-to-understanding, some of the existing solutions and open problems in this space.

A

All right good morning, everyone so happy to be here. Let's talk about software supply chains for devops, I'm isola greenberg, I'm senior software engineer at gogo, I'm the tech lead of gcp's, container analysis and container scanning team. I co-maintained graphics and critis open source projects, and I hosted the first software supply chain track at qcon in san francisco in 2019. So I've been in this place for a couple for a few years, and I'm excited to be here with like mine in the crowd and I can be found on twitter.

A

So what makes um challenges in software supply chains particularly unique? um The concept of software supply chains isn't really new. We've had the term software development lifecycle for a while. Now and um the way I see it up for supply chains is basically that, but in the cloud ecosystem. So what makes it especially challenging these days?

A

Is that not only are we running our production workloads in cloud, which is you know, just a fancy word for someone else's computer, but also we're now using the third-party vendor dependencies open source dependencies, so it's just increasingly more complex to vet. Every single thing that is going into building application running it the platform that's running on and all the tooling that is involved along the way.

A

um Here's the diagram from the salsa website- and you already saw that in the premiere of operation salsa. So what it shows here is the software supply chain from source, and then it gets built with the dependencies pulled in and um and then it gets packaged and shipped to the customer, whether it's a container image, binary um and so on, and as you can see here, every step of the way.

A

There are multiple attack factors that make it very problematic for us to ensure that what we're running is what we intend on be running and it has everything we need and nothing else in it. So, let's talk about devops in clouds, we'll be talking a lot about the secure as a security aspect of it. So I wanted to um cover the other side of it. The operational side of things same software supply chain, as we saw earlier, just the one that we'll be using for the rest of the talk.

A

So we have our engineer who writes source code. It gets built tested and then some static analysis tools run and then deploy checks make sure that the right thing is deployed to production, and now our code is running a production.

A

So now imagine we'll get woken up at 3am by an incident, so our pager goes off. What do we do? The first thing we do is, you know: go look at our graphs tracing some of the logs to try to figure out what's going on and now that we need to do root, cause analysis and impact estimation.

A

Everything with uh across the software supply chain is now under suspicion, and so we will look at every single stage and what happened and what changed and what might be triggering this page, and so in order to be able to query for this efficiently, we need a universal artifact metadata so that we can look at everything that happened while the software was being built and then deployed to production, and then also we can look back at it so that if we, when we roll back our software, we can make sure that we can still look at what happened in retrospect.

A

So now, let's talk about some of the existing solutions on google cloud, so for the first for the source, we have our code source cloud source repositories, and so it allows you to design, develop and securely manage your code.

A

Here's an example. What it looks like you can deploy a specific version of your code from cloud source repositories using the cloud cloud functions, and then this way it makes sure that you know which version you're deploying when you roll back to the version that was running okay last week. You also have a way to track that along the way it also integrates with cloud audit logs.

A

So I can look at my error logs and then try to see what piece of code it's coming from, what line of code it's coming from and then see if this is new code or a new traffic shape. That is triggering this page.

A

So what allows me to answer is what new code could have caused this incident so moving down the software supply chain, we have build test and cloud build, allows you to build test and deploy on the serverless cloud cicd platform. So what it allows me to answer is: how is this binary affected by the incident build tested and deployed? Did I miss any checks? Did I have the right build configuration and so on now going down further?

A

We have our scan analysis stage and that's where my team comes in, so on demand scanning continuous scanning. Other two products that my team owns on-demand scanning allows you to scan container images locally uh on your computer or in a registry. What this looks like is we have a gcloud command, which is basically a cli that allows you to scan an ubuntu image, for example ubuntu latest, which is a local, my computer, and then it will extract packages and make sure that uh all the analysis are run and then display the results.

A

As you see here, continuous scanning is similar to on-demand scanning, in that it scans images in artifact, registry and gcr, but also it allows you to monitor vulnerability information to keep it up to date. So if a new cv comes out or there's an update, there's a fix available severity has been revised.

A

Continuous scanning will give you those up-to-date results. After you push the image, so you don't have to re-push it. So here's an example of what the vulnerability results look like here. You don't have to be able to read everything. I walk you through the key parts, so you know when I'm woken up at 3am and I'm trying to figure out if this is okay with the rollback and I can go back to sleep or I should keep investigating. I see there are 74 vulnerabilities and others zero critical severity.

A

So that's good next one I will probably dig into high severity vulnerabilities. uh I see that there are some fixes um on this, so maybe I'll check to see if this is this could have prevented my incident and then maybe during business hours. If I really need to I'm gonna dig into more of these vulnerabilities at medium and low severity.

A

So the question I can answer with these products are: what vulnerabilities am I exposed to that contributed to this incident going down into deployment parts? So we have binary authorization which is deploy time security control. It ensures that only trusted images are deployed on gk and cloud run.

A

So here's an example of what the binary authorization policy looks like we'll set up uh rules. We can allow list images. So if we trust the provider of the image I'm going to say, I know they'll do a good job of making sure that the images I use from them are clean and uh can be trusted.

A

Then I can list them there and then I can create the policy um on the example here. There are two types of policies, so there is allow all images on the left hand, side and then disallow all images on the right hand, side and there's another kind you can configure, where you specify only the images that have been attested can be deployed. So now, on the left hand, side, the image is deployed successfully and on the right hand, side. The pod is blocked from deploying, and then we get an error message.

A

So the question I can answer with this tool is: what policies do I have to prevent bad code from deploying, for example, I want to make sure that nothing I deploy has critical vulnerabilities in it and now tying all the stages together. Is the container analysis api, another infrastructure, piece and products that my team owns, so it allows you to store metadata for resource and retrieve it to audit your software supply chain.

A

Here is an example from the public documents you see here, another basically view of the same software supply chain. uh With the more familiar one saying that source built test deployment checks. They can all be represented compliance checks, they can all be represented within container analysis, api and retrieved.

A

So the question a host may answer is tell me everything about this artifact software development lifecycle so tying it together. We looked at cloud source repositories, cloud, build on-demand scanning container scanning, binary authorization and container analysis api as some of the existing solutions today that you can use to monitor and use this tooling for the operating your production workloads in cloud on gcp.

A

Thank you very much.