Cloud Native Computing Foundation KubeCon + CloudNativeCon China 2018 (Shanghai), 21 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Securing the Deploy Pipeline - Felix Glaser, Shopify

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Securing the Deploy Pipeline - Felix Glaser, Shopify

Imagine taking arbitrary code, deploying it to production, and hoping everything is secure. When we don’t lock down our deployment pipelines and deploy arbitrary containers, we do exactly that. Join us to discover Shopify’s solution. After a container is built, we run checks to determine its state: Is it free from vulnerabilities and outdated software? Does it originate from the correct deploy pipeline? For every successful test, the container is signed and the signature stored in Grafeas. During deploy time, the Kritis admission controller enforces the presence of the signatures. Because the security state of a container can change, we log the metadata created during a container’s lifetime; if it becomes vulnerable, it can be recalled, fixed, and redeployed. With Grafeas and Kritis, two new tools join Kubernetes, allowing everyone to prevent privilege escalation via code deployment.

To learn more click here: https://sched.co/FuKN

A

Okay, cool: let's get started after figuring out our technical difficulties, hi I'm, Felix Glaser, my work at Shopify I'm, a production security engineer for those of you who don't have that work title. What we do is we care about everything there's below the application level, so we take care of building secure, docker images, deploying them securely, securing kubernetes for old infrastructure, building our own secure, kernels, rolling them out.

A

So everything that's below the app level is what I personally find interesting for those of you who haven't heard about choppa fine Shopify hosts more than 600,000 stores all over the world. So if you have recently bought from somewhere that wasn't one of the big market places, then there's a good chance that Shopify was behind this. The tiny online store that you bought something from so let's do a quick straw. Man Paul here, who has ever run sudo curl pipe sudo bash on their machines, yeah.

A

Basically everyone here who has done that somewhere else, then their laptops, let's say like a production environment Oh at least one and who felt good about it, no one that was to be expected.

A

Today we are going to talk about something that is very similar to running crawl, pipe Zulu bash in production. For that, let me go back about 15 years, for those of you, who've been in the industry for a bit longer deploying on VMs should be a very familiar process.

A

You installed and configured UV hooked up the betrayed, and that was your production system. It came with the advantage that it was at least virtualized, but it was really hard to roll out and to maintain, because he always needed to SSH into boxes needed to run, commands and well.

A

There's the mutability of those VMs had a lot of downsides because, like rolling like replicating a vm to like multiple VMs was a pain in the bad and mutability is also the enemy from like a security perspective, because everyone can who got access to your VM could just install the software they want it there as soon as they were in they were in for long times.

A

They could gain access to your database and well at that point, it's more or less game over for you, but then docker came around and things changed for the better rolling out applications is super easy. Now you just start a docker container wherever you want it to run in our case I guess: that's always queue Nettie's and it's super easy. You just take a docker container, bring them from bring from A to B and run it again because we got rid of mutability.

A

So this shouldn't be a surprise for you. You write some code, push it to wherever you host your code, build it with whatever you build it, and then you deploy your docker container into your cloud. Well, this still allows for manual changes. When people connect to your kubernetes cluster, they can still run a docker container from wherever can create cube, CTL kraid run edit it someone from outside the organization gets access to that. Then it's game over, so I don't want to talk about that.

A

But what can happen is that someone within your organization that has the authorization to deploy an image, deploys something that they shouldn't and here is where we come back to curl pipe sudo bash, because darker files are the new call, pipes, sudo bash because, let's say what's wrong like arguably there's nothing wrong with this darker file and I, don't want to bash Ubuntu here, but this darker file has a lot of critical vulnerabilities and even more high and severe ones.

A

So the problem here is not that you're deploying something that is by default: malicious, but you're, deploying something that is vulnerable just by aging. So, even if you are deploying your image instead of Ubuntu image, since we moved away from mutability, we now have immutable docker containers and as soon as you deploy them, you don't you can't just run up, get install unattended upgrades and just keep your darker image up-to-date. That just doesn't work. So how do we fix that.

A

We need to do is we need to find a way for us to figure out whether a docker container is OK to be run in production or not, and we could do that with a service that we just introduced into kubernetes and whenever we run a docker container. We call out to that image to that service and that service decides whether the image can be run or cannot be run. So, let's build this here together. So.

A

There's obviously, two ways this could go down: we could either make our decision at runtime, which gives us the advantage that whenever we make the decision, the decision is accurate because we'll take the docker image to all of our checks on the docker image and they decide where it's okay to be run or not. The downside of basis, obviously that the startup time increases I, don't know by like two minutes if it's a big docker image and you want to check for adjust for whether it's vulnerable or not.

A

The other thing is that we could pre commute pre-compute our checks and then just store to the result of those checks and just make sure that those results are what we wanted. What we want them to be. The downside here, obviously, is that things like vulnerabilities pop up every day, so we can never be 100% sure we're there. Our data is as accurate as we want it to be, and we might need to go back and like rhe computer or checks every now and then so how?

A

What do we attach our checks to so every every docker image is uniquely identified by its digest with a sha-256 we take that and the dock reference is where your image lives in their registry. We take those two pieces of information and just PGP sign it, and then we store the signature for every successful check that we execute so.

A

What would be the ideal place in cuber Nettie's to put this service? Well, it needs to run whenever an image gets deployed, so it needs to run on trade, edit, update and so on.

A

So the correct place for this is an admission controller, and I guess every one of you has if they've ever used, kubernetes houston admission controller under the hood, because they do you four things for us like image, Paul, always and other things where they go in during admission and decide whether they should change the weather, they should change, how they get to apply or whether they get deployed. What we're doing here is not a mutating admission controller, just a validating the admission controller and that made mission controller is called Katie's and it's open source.

A

It's yeah, it's the piece that we're going to use to base all of this on, so kritis gates are deploys now, so we now have a way. Now we have a service called fittest that we can go out to and ask where the image is okay, and if it's, okay, we'll pull it from our registry.

A

If it's not okay, because it lives, for example, outside our registry, then it won't get pulled that gates us from like someone making an accident like accidentally pulling in an image from somewhere else and also from someone maliciously trying to pull in an image from somewhere else. So where do we store those PGP signatures that we created? There's a service called Goffe us and Goff is a metadata server that stores a lot of different kind of information.

A

You can have attestations, which is the correct term, for what I called signatures up until now, but Garfias holds a lot of them, can hold a lot of information about images. It has built information like which packages were used in the build which base image was used.

A

You can store all your vulnerabilities from your vulnerability scanner in there, so this is the ideal place for us to also store our attestations in there. So now we have two critical pieces of infrastructure. Already we have our admission controller treaties, our database graph layers, but we don't create at the stations yet- and this is where voucher comes in voucher is written by my team and me at Shopify, and it's a customizable attestation creator.

A

So after an image is built, we call from our CI CD pipeline out to voucher and voucher, then reaches out to different parts to find out whether the images okay to be deployed. So, for example, it goes out to our registry and just sees where we can pull the image from our own registry, because only we can push it to there. So we know it's our image.

A

We make sure that the container doesn't run as root.

A

We check that it comes from the correct pipeline that there are no vulnerabilities in there.

A

We use clear and Google vulnerability scanners, so vouchers supports both of them, depending on whether you run on Chiqui or not so it reaches out to the vulnerability scanner sees whether their vulnerabilities you can configure, which will nobility's. You are ok with, let's say: you're: ok if they are like medium and lower, and then we can just ignore all of them.

A

Obviously you could also reach out to your CI and see where your image has like, let's say, 80% code coverage in there and only deployed things with code coverage so cool. We have now also a place that creates at a stations but I guess every one of you runs different images and different specifications for securities in different tiers. So let's say you have a production tier. Where you only want your most secure images to be running, then you obviously want them to not run its route.

A

Have no vulnerabilities, but you also want to give your developers a place where they can just try out anything and like just play around and create new software, which then later on, gets more and more tight. The more it moves, security wise, the higher it moves up in the layers. So no, we need something that decides where we want, which at a station to be required and we're not. This is where policies can come in.

A

Policies are just yamas backs where you can say what you want to be whitelisted, because there might be images that you generally trust right. You don't want to like pulling the nginx source code, build it yourself, no just like trusted developers that they do their job correctly for nginx, and then you can just specify images to be white listed. You can have different enforcement modes like if you're rolling out something new. You might only want to log.

A

Violations to policy, but not block images from being deployed, because then you can just like to bring up your infrastructure somewhere else, see that everything runs smoothly and then flip the switch to block and audit log evaluation mode is obviously require attestations because you want a stations to be there and then you can have different testers.

A

So for every check that voucher executes, we have a different, so called a tester, and under this a tester, your signatures will be grouped. So in this case we would only require the one called name: yeah.

A

So obviously, this is a little too widely scoped. If this holds true for the entire project, that's why you can have policies per cluster. The same goes there as before. The only difference here is that you pop in the Lyon cluster admission rule, and then you say, like only run it for that cluster, so that you can have a different set of rules in every cluster that you run in.

A

So we talked about vulnerabilities before there is a special one. The package vulnerability policy, where you can say which vulnerabilities, which severity of vulnerabilities you're fine with so, for example, this blocks everything high in a bob and since some times, there's no fix for CV. Yet you can whitelist them and thereby then circumvent it from being blocked. If you are fine with that vulnerability being deployed in your environment, so.

A

Now we can reach out to our different projects and different clusters and find out which at a stations need to be present where so everyone here who works as an SRA, should ask that question right like, but what about emergencies? What, if I need to go in and fix something right now and I? Don't have the time to like build a new image, fix the vulnerability before I fight a fire.

A

This is where break glass comes in. You can just on the pots back annotation, you can say, break glass, true, which will then tell kitties that, even though the required attestations are not present it'll still let you deploy something.

A

So if this is the case that you can just pop on there and it's an annotation on to everything that you deploy, how does that help in the first place because, like now, we would either need to secure that piece of information and then not tell anyone which would be security by obscurity so like if anyone can just add the break glass annotation? What is it good for well? The idea here is that whenever you break glass, it'll create an audit log that audit log gets picked up by your pipeline.

A

Whatever you use, and then you send out notifications to Pedro, Duty and Pedro duty, then pages cloud sag. So your team that is responsible for running all of this.

A

The other thing that we're thinking about is gating, who can actually add that annotation to break glass, so that the criticism that the creator's administer and Mission Control will only accept that annotation from a specific set of users, so that you can like get it. That way, um why it's look at why? This might not be super easy to roll out um in case it wasn't really clear. Those setter stations only work on containers that are specified by their shop.

A

So you think that a lot of you are still deploying with tags, but this won't work anymore. If you want to use binary authorization, because the career to submission controller only is able to look up an image in graph as if it can find it by its sha. So you need to make sure that your infrastructure is able to handle that yeah.

A

You might want to whitelist a lot of things in the beginning and then reiterate, like iterate over them and then throw them out bit by bit.

A

There's a cool thing, though: there's a plugin called resolve tags, it's open source, it's part of Goff EOS, and that allows you to deploy wave a tag and what it does is it actually reaches out to your registry or to the registry where the darker image lives then resolves the text tag to a digest, replaces it in your Yama and then ships the yama that has a digest instead of the tag that might help a lot of you when you're trying to roll this out.

A

So what have we achieved and what is left to do? Well, what we've achieved is we created a system where we rule out an entire class of vulnerabilities they're introduced by vulnerable docker images have code that is wonderful. Well, we don't defend against right now is that stuff still can get vulnerable over time. So what is really critical here to do is that you running your own infrastructure, a vulnerability scanner that scans all your images regularly, so that you can be sure that none of them are vulnerable stores.

A

Those that information in affairs like once a day once a week, I, don't know and then use the vulnerability policy to make sure you don't deploy stuff that has become vulnerability over time.

A

So, to sum this all up, this is basically the new way of securing your deploy pipeline. You push something: it gets build, voucher trades at the stations for it. Those at stations get stored in gravitas from where heaters at deploy time will pull them, make sure that they match the entire policies required in your cluster and in your project and only if they work it gets deployed and if something breaks, glass, it'll page your team about it cool.

A

Do you have any questions.

A

Do we have a microphone if not just speak up? Another repeat your question.

B

So you mentioned three components: core components; well sure, gracias and kritis. So I, just wonder if all of them were developed by Shopify, no.

A

Only vouchers developer, Shopify and the other two Affairs and creators are developed by Google, but they all open source. So if you want to roll it out in your environment, they are all open sourcing, you can all just deploy them Thanks. Do you mind passing on the mic.

C

Hi I a question about the SS stations: understand that you create the attestation based on the information available at the point of creating it. But let's say you know further down the road as you progress on on this work, you you find other things that you can attest yeah. How would that I mean? And this can be like if a new vulnerability comes out, then you realize you have to attach a specific module or.

D

C

Version or sub versions, how would you go back and then kind of we? How would that flow look like if you had to reattach certain information about the images? Okay,.

A

There are two answers to your question: right like if you're doing something completely new, like let's say you didn't check beforehand, where your images ran as rude or not, and you want to roll out that, then you just create the new. You just need to go back through all your images scan all of them again, there's actually tool. If you go and check out voucher, there's a tool which lets you do exactly that, go back and retest like a bunch of images at once.

A

The other thing is if a new vulnerability comes out- and that is unfortunately the case that that happens all the time. You just need to run your vulnerability scanning continuously, okay, otherwise otherwise you'll just miss it. Yeah. Thank.

D

A

The image the packaged vulnerability policy then takes care of this. If there's new information in Garfias it'll block it, okay,.

E

Just to try to qualify a house is different to the harbour project. Hover portray the harbour with a container scanning.

A

Well, this is this allows you to do whatever you want to scan for right, like you can create at a stations for anything that you want to use. This is not only about vulnerabilities, you could think of this as being the one piece where you can create anything that you want. For example, let's say you only want code to be deployed that was pushed by people within your organization that used that sign there commits you could make that happen with binary authorization.

F

One question on the kitties and then the graph is integration mm-hmm when, when Cadiz wants to allow or not allow something, some docker image, what is the data? It sense to graph I'll it? It has to talk to the face first and then decide or it just delays on the image metadata. So.

A

During deploy time, hideous looks at the digest, takes the digest goes to affairs, pulls all the attestations from there that it can find in graph is takes them back then, looks at your at the policy at the policies to see which are required, and then it actually makes sure that your poly that you're at the stations are signed correctly and then, if all of them are, then it gets deployed. So.

F

The image digest is a key for.

A

G

This image over nobility policies, our they maintain enough. Some new policy comes in the.

F

Image vulnerability.

G

Policies, you know, there's a new who maintains these policies. If there's a new policy publish you know there could be. You know this critical or nobody be there and the doctor images. How does this policies be mean? It's it's. It's a open source thing that you know the list of policies available. Oh.

A

Well, let's go back to well, policies are just llamo sped, you your s, eries or production engineers right and maintain, and the package vulnerability policy you just specify and per cluster and then just leave it there.

A

Cool, if there are no more questions, then thank you all for coming.