Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2021, 30 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: PSP is Dead, Long Live PodSecurity - Monis Khan, VMware; Mike Danese, Google

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

PSP is Dead, Long Live PodSecurity - Monis Khan, VMware; Mike Danese, Google

After a quick intro, this presentation will touch upon two auth features that recently went GA: bound service account tokens [1] and kubectl credential plugins [2]. The bulk of the talk will focus on the replacement for pod security policy: pod security admission control [3]. We will cover the reasons behind the replacement of PSP along with the specific technical details of PSA. [1] https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin/#bound-service-account-token-volume [2] https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins [3] https://github.com/kubernetes/enhancements/tree/master/keps/sig-auth/2579-psp-replacement

A

I'm mike- and this is mo so um I will be doing uh this- is technically the sig of deep dive, so I'll be covering um a little bit of cigar stuff. I'm gonna give you guys a highlight reel of what we've been working on over the past year or so because I know it's been a while, since we talked and then I'll hand it off to mo who will begin the deep dive into pod security.

A

So let's get started so um sigoth. What do we do? What are we responsible for? um Sig auth is primarily responsible for authentication authorizations, the kubernetes api and various other security controls um such as uh admission and uh authorization policy like rbac and extensions therein.

A

So um one of the admissions that we're gonna be talking about today is uh pod security, admission.

A

So what have we been doing for the past year or so there's actually been a lot going on, so maybe actually, probably over a year ago, we began discussing the defecation of uh pod security policy um and uh discussing what it would mean to be a successor of pod security policy, and we knew that pod security policy had a lot of users in open source, like we debated whether we could defer those users to look at out of tree um strategies for for doing the same thing like opa.

A

Ultimately, we kind of landed on pod security, which we will discuss in great depth that is going to be beta in the upcoming release of kubernetes, we're targeting beta 123, targeting beta and 123.

A

um the certificates api, which had uh was long overdue for ga finally launched ga in 119..

A

um I think it had been around in beta since, like 1-6, so it's like, maybe three years overdue, for going ga um before the ga. Some some notable changes were uh added, such as a new signer name api. So now the api can be divided between uh multiple consumers. um After ga a duration, hint was added. So now you can sign certificates or you can request uh certificates for specific durations.

A

um The exec auth cube, control, plugin and extension uh also went ga in 122. I think um which.

B

It was either 121 or 122.. I can't actually remember it. No, I think it's 120.

A

um And that paves the way for moving some of the final entry uh authentication plugins for clouds out of tree. So that's kind of part of the whole club provider extraction effort, and we also have a proposal to extend support for those plugins to um to request signing and like mutual tls protocols and the token request.

A

Api bound service account token volume and um uh provisioning uh for bound service cat tokens to replace, like the the legacy secret tokens all went, ga um the and I think the final ga was in 123. um So that's a, I think, a huge accomplishment. I think we've been working on that for a couple of years and um there uh it should benefit the entire ecosystem, because those have significantly better security properties than legacy tokens.

A

There's some more work to do there and finally, the last highlight is we have a sig sub project in uh called this secret csi driver they're, actually implementations of um so csi is the canadian storage interface. There are implementations of the secret csi driver with back-ends for aws, azure, gcp and vault, and that allows configuring volumes that project secrets in external uh um secret storage providers.

A

So um I think the kep for gaining that just uh merged- maybe this last week yeah, so we've been super busy and with that I will hand it off to mo.

C

B

I'll make some quick comments on this stuff, so, uh on the token request stuff, uh some of the stuff that's left is to stop generating the secrets, the back service accounts and really backpedal that functionality completely out of the api server or the controller manager.

B

So there's caps open for that. You know. There's plenty of work there to be done. If anyone wants to get involved, you know we're a very cautious bunch of folks. We don't want to break everyone's stuff, like we understand that you know. People run real stuff on kubernetes now, so we're trying very hard not to break things, but we also need to move the ecosystem forward. So there's a balance there and like rita and anish, have been working on secret csi driver for quite some time, so that is also a place.

B

One could get involved to try to push things forward, so the rest of the talk is mostly a code walkthrough. So tabitha and tim had a talk earlier today at 2 30 that talks about the new p, the new pod security stuff. From the perspective of like an end user, this one is more focused as the perspective as a like a perspective contributor right like if you care about how the code is implemented, and you know, maybe you want to get involved and help us get it from beta to ga and so forth.

B

So everything in the code walkthrough is based on 123 alpha 3.. So if you see something, that's not what's in head of master right now, that's just that's probably why, but in particular, uh jordan and tim really set an incredibly high bar for the code, which is, unlike basically anything that exists within tree that you know this thing has to run. You know, be able to run standalone without, like an api. Server can be run as a web hook, maybe used in ci tooling.

B

Also, all the the checks like you know, is this a para. Is this a privileged pod or whatever all of those are implemented as individual units? So there's not like some large spaghetti thing that checks all the things and the idea there is over time. You know we'll evolve these things and add new checks or iterate on the an existing check, but at a different version. So all of that stuff led to a pretty um pretty elaborate implementation.

B

For what sort of just sounds like a pretty simple thing like you know, if this field is set on a pod, say no you're not allowed to do that, because that's a privilege, part or whatever. um So as a most of this slide is j or sorry. Most of this talk is just me going through the code editor, but I did have two slides so as an example, this is from one of jordan's examples on this stuff.

B

If you wanted to run a little cli, that would, for example, do some linter checks on some pod specs that were going to be committed into your, like your prod clusters, coming up right, you're going to do some promotion, but you wanted to validate that they met some security requirements right and you.

C

B

Want to gate this all right, so this is basically the skeleton of a tool that can run just completely without an api server. Just and statically validate things right. So the pod security stuff, unlike psp, is much more constrained right. You basically say what level of the api you want and what version all right so like baseline or restricted, so restricted is best right if you, if your pod can work under that, that's what you want- and you know maybe uh maybe you're on like version 1.19 right now and you're like all right.

B

I want to make sure that this stuff is going to run on 120., so I'm going to go ahead and try to validate against 120 and see like if I need to go talk to my developers and like figure out some stuff before we actually move stuff forward.

B

So, like you know, we pass those things in we'll go over like what the evaluator and the checks are like, but basically, if you're, if you're familiar with the kubernetes code base, you're, probably familiar with the builder pattern that we have, and so all this is doing is, like you know, passing in some files locally and then it's just doing the visitor pattern on all those files, and so the actual visitor is basically like all right.

B

I was given something with a pod spec deployment, whatever I'm going to extract that spec out and then I'm just going to evaluate it against the level and the version whatever you pass me and just check if it's allowed or not right. So in roughly like 100 lines of code, you have like effectively like a production, ready, ci tool, and that's kind of the hope from this is that we want to enable people to build tooling off of this, and you know, offer value just outside of just pure kubernetes admission.

B

So the the notes um that I have kind of. Basically, I went through the code and tried to write notes on like interesting aspects. That might not be completely obvious. So all the notes are on that fork of mine. So now I'm just going to kind of walk through some stuff and hopefully it'll be helpful for.

D

B

You all read that okay, we tried to make the text nice and big early on. uh So, if you're familiar with kubernetes admission, all of the all of the stuff gets wired up in this api server package.

B

uh So admission is uh ordered in the sense that it's a deny based system right. So with a deny based system, the ordering of checks matters a lot, because if you have an early check that denies it guarantees that the rest of the stack was wouldn't run.

B

So one of the things that you know pod security does is to make sure it runs before pod security policy, because you can run pod security in audit and warning mode, so it won't deny stuff.

B

But you know what you could do with that is you know you could have your existing pod security policies, the old style go ahead and put the new pod security stuff in an audit mode, but in a very restricted mode and just let it run and generate events as things are going on right and like you, the kubernetes administrator can go check. The audit log saying. Oh, I see that mo is running this privileged pod that doesn't meet the requirements.

B

Now I can go talk to mo, go ahead and start getting this stuff rectified, but that's only possible. You know if you run pod security in front of pot security policy, and I realize this is obnoxious naming right because pot, security, admission or pot security policy like it's very hard to understand. Sometimes what you're talking about, also don't be confused by this. This this piece of code makes it sound like pot security is enabled by default. It's not it's feature gated, so the code is there but feature gated off.

B

So don't don't this. This whole function is kind of a pain to read because it's like what's off, but it's on and then the intersection. So it's a little painful. uh Oh man, it did. It did not globally change the text size. I'm sorry.

E

B

It bigger um so I'm gonna try I'm trying to assume that folks, don't know how the internals of cube work to some degree. So one of the sort of weird things within the kubernetes codebase is we have this concept of like an internal type or a hub type. uh So whenever we read your configurations or your objects, you know we read them as like pod, v1 or whatever there's an internal representation, that's completely like related, but it's meant to represent the schema across all versions.

B

um So, obviously, today, pod security doesn't necessarily have that, but you'll you'll kind of notice some of this stuff. So, like you know like in this, where it's doing the loading, if you don't provide it anything right, it's going to be like all right, I'm going to get the alpha configuration I'm going to then default. It we'll look at what the defaults look like. Then I'm going to convert it into the internal version.

B

Don't get too stressed about this, but this is like all over. Our code basis happens all the time, but the defaults for pod security policy are are kind of what you would expect, which is, if you don't tell us, we just assume you mean privileged and if you don't tell us a version, we just assume latest which, by the way privilege means don't do anything. So in order to not break all your stuff, we do nothing by default. You have to tell us to do something.

B

You know it is what it is. When you add something new that can break stuff, you can't just turn it on you'll break people's stuff.

B

So looking at the actual configuration, uh it's pretty straightforward right, it's um you know like what version do you want to enforce that? What version do you want to audit at? So? Oh I'm, sorry, I yeah I will. I will try to remember the sort of interesting bit here. Is there is a hard-coded list of exemptions, so you can imagine that you have a pod, runtime class. That is, I don't know, running vms under the hood and you're like I don't need pot security like I just have strict isolation.

B

So you could, you could tell us in this bit that like yeah, if a pod asks for this- I don't know windows, runtime class or something just ignore it completely like I have guaranteed out of band that this is safe.

B

You could similarly have some user, that's always allowed to run privilege, pods or whatever, and you know then you're saying I'm taking ownership of somehow gaining access to these pods right, because if you can let like as a for example, if you let arbitrary people execute to a running privileged pod. Well, you have broken all security for your cluster.

B

um Let's, let's walk through. uh So if we go back to the actual code, um I always like to jump through the name of the plugin to actually get to where it's wired in so it's pretty straightforward.

B

It's like a it runs on create an update of a set of resources. Primarily let me find it so it runs against basically spaces and pods and anything that has a pot spec.

B

But for the most part the enforcing side is on pods right. So the idea would be is that you don't want to break like a deployment controller with this.

B

You just want to prevent the pots from being run so because you can imagine you could have a mutating web hook that will coerce the pod into a correct shape when it's created, even if the deployment has something slightly different in it. So for the most part it doesn't enforce on the embedded stuff.

B

uh So we can. We can kind of look at how some of this code works. Real, quick, let's see like the validation logic, is so, as I just mentioned like so there's distinct logic for how name spaces are valid. Pods are validated or pod controller, pod controller being like controllers that cause the generation of pods because they have an embedded pot spec somewhere within the schema like a deployment.

B

um So the most interesting one is like pod, because that's the one that actually like does the real work right. So it's like all right, ignore sub-resources that don't matter and so forth.

A

The reason for doing that is to hoist errors that you get as early as possible because it's terrible to create a deployment and have that fail. Three steps later asynchronously yeah.

B

Yeah, um but you can see, like you know, a pod evaluation you know is strictly in force um and which makes sense right like at the end of the day.

B

The pod is what we have to prevent from running right, like everything else is a byproduct of all that, let's see, let's see, I think some of the more interesting stuff is the whole uh registry, so with the old pot security stuff right, you had this really expressive api and you would make individual pot security objects and you would try to say, like here's, my policies, that level of expression is purposely omitted from this api, because the other api is unmaintainable effectively.

B

So what we have is basically this little evaluator thing which takes a series of checks, and so the idea is that this evaluator, you know, is given the input it runs. All the checks and says is everything. Okay. Some of this code is like a little bit hard to read because it's trying to do like well, you have a check at like version 1.12, but this is version 1.20.

B

Oh, that's the like the newest check that's available, so it must also be the 120 check. It's kind of it's it's a little hard to read at times, but for the most part you don't have to worry too much about it. Jordan and tim got it right, so you don't have to worry too much.

B

The more interesting stuff is like the actual checks themselves. So there's like this global registry of checks- and you can add to them.

B

Oh, I'm sorry, ah lord, I can read it so I assume you can read it. I apologize yeah it does. It does work on my machine, um so you know what does a check happen? It's basically like a unique id. What level does it run on like baseline and restricted and then the specific checks over time right? So you could have a check the same check change its meaning over time. This is one of the core requirements of the pod security stuff, which was we can't just say like a check today.

B

Is this valid over time because the pod spec changes right? So if you have, if you add a new field somewhere within the prospect, that's privileged like equivalent of privilege, the existing privilege check needs to start accounting for that, but only at a newer version right. We don't break you in place right, no kubernetes apis are allowed to arbitrarily break in the middle.

B

So to show you what that looks like uh the privilege, tech is sort of the most canonical check right, which is like don't let these fields be true like false is good or undefined is good, but true is bad all right and we can see that this check has existed since the beginning of time effectively because these fields have existed since the beginning of time.

B

So it's you know it's a very early check and you can kind of see that it's like all right, look at all the containers, and by that I mean like all of them, like the init containers, the actual containers, the ephemeral containers and maybe one day, there'll be more containers and they'll also be in here.

B

Look at all of them and make sure that nobody sets privileged equals true in there right and if they do then fail with a nice error message all right so that one's pretty easy- and you know it's a pretty easy check. The run is non-root check, it's kind of long and kind of sprawling almost because it's got like like levels of checks in it, so it kind of just keeps going for a bit, but the nice thing is all of these things are completely distinct right.

B

So if the pod security standards change- and we add a new check- none of the existing stuff has to change. We just add a new check to the rest of thing and to make it so that you know you could imagine a more expressive version of pod security that is completely implemented out of tree as a validating admission web hook, and let's say that someone wants to not run this stuff at all. They just want to run their whole thing.

B

It would be really nice for them to be able to validate that stuff that they have built, conforms the same validation checks as the internal treatment stuff, and so one of the ways we do. That is for each of the checks. We have a test that generates a series of passing pods and generates a series of failing pods right, so those checks are obviously run uh sort of in line in our unit tests and stuff, but they also generate. So.

B

If we look at this failure case and the zeroth index, they also generate yaml matching that so all of this yaml is available. You know in a very structured way right like this is a baseline check at version 12 122, and it's a failing check for the privileged check right. So if you were implementing equivalent code out of tree, you could run all of this yaml against your outer tree implementation and validate that it conforms exactly to the entry one right.

B

Obviously you know this is not a strict requirement for having pod security, be a thing. It's just that you know. Jordan and tim were very careful to make it so that the ecosystem could build and mature on top of this stuff. So it adds pretty significant complexity to our to the work we had to do to make all this work, so I'm done with the code. Does anyone want to ask me something before I go back to the slides.

E

uh I did have a question here from online and uh this is uh this few sentences, so I promise I'm getting there a little bit longer, though. What is the recommendation for organizations that plan to use something like gatekeeper or caverno to provide pod security pieces?

E

Should the built-in pod security admission controller be disabled, or is the recommendation to have them co-exist with the built-in controller, providing some basic level of protection and made more fine-grained by a third-party admission controller.

B

um So mike, you might have a different opinion on me just because I feel like this is an opinion and less of a like a like a thing that has a concrete answer. I would personally lean towards always running like the the built-in stuff you know in whatever mode is you know? Obviously, if you can run everything in restricted latest do that that might be impractical, but and then, if you have further checks, add those checks as you're validating web hook or whatever you want opa or caverno.

B

Whatever your favorite check is because, as you upgrade, your kubernetes cluster, this stuff will get upgraded by us. This stuff is tested by sega. This stuff is maintained by segat right and we care deeply about this implementation and we validate this one. So I I would, I would sort of that's the direction. I would push people, um but it's more of an opinion. It's not a straight guideline.

B

I like the job.

A

Yeah, I agree so uh one of the initial problems uh with psp that was cited when we um you know chose not to graduate from g uh to ga was that it was uh users were incapable of turning it on in existing clusters without breaking their users. So what we did with psp, v2 or whatever you want to call it was we designed the system such that it can be turned on, and the cluster will continue to behave as normal.

A

So we would like we included kind of what we saw as the table stakes for constraining, pods, and you know we totally accept that our definitions aren't going to meet all use cases in the world and I think in those cases it makes sense to run you know opa or kievarno.

A

In that case, uh like I would say, you can use uh pod security mission for as far as it gets you and then use kiverno. I use them both at the same time really up to you. I would say, probably it's not necessary to turn off a pod security mission controller and, if you feel like you need to then there's probably a problem with pod security mission controller, and uh you should let us know why.

B

Yeah one of the core requirements of it, for it was for it to co-exist with other tools that are doing similar work and psp made that annoyingly hard. So we've tried hard on this one not to have that evan.

D

B

D

I understand it right now. um If I turn on the alpha flag in 122, all my name, spaces by default are privileged or using privileged.

D

um Are there plans to either ship something so that when I create a namespace, I can set it down to a lower level or change the default in future versions of kubernetes to steer users towards a more defended security posture.

B

So I had this discussion with jordan and tim and david they they did not believe we could do that while maintaining the compatibility requirements of the kubernetes api, which is unfortunate thing, at least like basically what they said is this. This really comes down to like a choice for your distribution and what security stance it has as this default posture.

B

So you know as a for example, I work for vmware in the tonsu portfolio as soon as this thing is beta, I'm going to go turn it on and I'm going to hit anyone with a hammer if they try to turn it off.

B

But uh you know that's sort of the opinion of the distribution. uh At least the nice thing is it's very easy to go change the default. You know it's a very simple, yaml file that just says: if you have no edit, if you have no labels, that means you're like latest restricted or something right like you, can really tighten it out very easily.

B

It is a configuration that you pass to the api server, so it's like the rest of admission. That's the rest of built-in admission yeah it it's a little unfortunate, but it's sort of yeah. We just want to break people's stuff, and this, like you know, if we were building kubernetes all over again, I'm pretty sure we'd start from the beginning with this, but.

A

Yeah, that was one of.

B

The core requirements.

C

A

If, like you are on a uh in an environment where you don't have access to that configuration file, it is also fairly easy to create an admission controller on namespace, create and just set that default. So.

E

All right, we have uh one other question, then uh we'll look to the room. If anybody else has any questions too, this question is with the built-in pod security admission implementation. Is it possible for admins to add custom profiles or are the built-in privileged baseline and restricted profiles, the only that will be available without a third-party admission controller.

B

uh Yeah, so this one, I can actually give an answer and yes, baseline and restricted and privileges. All we plan to ever implement.

A

However, the code is modular, so.

C

A

Can go ahead, um fork it and use the same structure to deploy your own admission controller um with your own profiles.

B

Yeah, I I don't even think that the 4k really just consume it. It is meant to be a library, you don't even have to forget um so yeah. You could totally do that, but basically we're trying very hard one. We don't want to become king maker here, we're not trying to like rule your policy in all the little ways, but just also like this is what prevented psp from going. Ga is because it was an ever expanding api surface and trying to make it keep up with the growth of the pot spec over time.

B

Just proved to be untenable, like being opinionated and simplistic. Here is sort of allows us to even have this at all.

A

And for the people who are willing to and have the need to go and define their own profiles, there are a lot of very good existing solutions that would allow them to do that, um such as opa gatekeeper kivarino, so that we felt like that was a problem that was already solved. Yeah.

B

Yeah at a really simple level, we want to make sure that in the future on, basically, every kubernetes distribution sort of by default, um create pod does not mean root on cubelet right like if we can get to that stage like we have made an incredible set of progress into what the default is today. In most environments,.

C

E

Come over to you.

B

I'm glad to ask for questions yeah.

C

I had a couple questions about so when it was reading and trying to decide what to react on was looking for namespaces or things that had pod specs internally.

C

So if you say, create a deployment that violates some of the rules and it's turned on to reject it like it'll, reject the deployment creation, or does it still allow the employer deployment to go through it just rejects the pods. The way like pod security policy does now.

B

So I might be misremembering because what mike said seemed to be misaligned with what I had in my head. I thought it was. We would reject the pod, but not the deployment. We would only audit and warn on the deployment.

A

But I might be misremembering, I think that's right, uh so the like uh how has worn uh how is worm mode triggered, so you have an annotation um that sets like whether it's enforced or or worn well,.

B

If I remember correctly, because it's under the pod controller path, it's always audit and worn for that piece. I think. But the idea was that your deployment could be out of the valid spec, but you could have a mutating admission controller for pods that coerces the pods into a valid spec. So the thing that gets created is allowed, so that was sort of the nuance that that it's not great right, because.

C

That was my follow-up question is like, because there are, you could mutate your pods in line and they'll they'll. They can be yeah eventually good. So rejecting the deploy out right would have been a bad idea.

B

Yeah, so that's that's the use case. We're trying to maintain is that we know that the pod itself is the thing that gets run. Everything else is implementation, details right. uh So that's where we really enforce and the namespace stuff, where it's really powerful is you know server-side dry run. Is you know a really cool feature and what you can do now? Is you know you? Could you can do a server-side dry run?

B

That says, I would like to change the label on all of my namespace says to restricted latest and just tell cubectl to do that, and what will happen is you will get back all the warning messages from the api server saying: hey: here's the like 17 pods across your entire cluster that do not meet this requirement so, but you don't have to mutate anything to do that right.

B

So now, you've gotten immediate feedback on what you need to fix without causing any disruption to your cluster and that's sort of like one of the cool things about this being label based and just the maturity of the ecosystem. Now that what didn't exist when psp was originally written, yeah.

C

E

Awesome we've uh time for about one more question: if anybody has one.

A

Let's uh hop to the slides, real, quick and all right.

E

A

um Cool, so uh let's we can so um we have some uh we're, always looking for people to join uh sigoth and get involved in the community. So as a couple good links for new contributors to check out these slides will be posted next to our talk in the schedule so come check it out. We have some good first issues.

A

We have a slack channel and we uh meet every uh two weeks on uh wednesday um at 11 a.m. Pacific standard time I'll have a lot of good conversations. A lot of ongoing initiatives discuss there and we welcome any everyone, anyone and everyone to get involved so yeah. Thank you so much for attending today, and I hope you had a good keep going.

B

As a reminder, this stuff, the psp stuff, is alpha. So if there's something wrong, please come tell us because we can fix it before we promote it to beta, because it gets a whole lot harder, because this is kubernetes and we don't try to break people's stuff, and everyone likes to depend on every implementation detail of everything we build so. But thank you all thanks. Everyone.