Kubernetes SIG Auth, 27 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Sig-Auth Bi-Weekly Meeting for 20211027

Description

Sig-Auth Bi-Weekly Meeting for 20211027

A

Hey everyone: this is the sig off meeting for october 27th 2021.. uh We have a pretty light agenda today, but we can go ahead and get started.

A

um The folks that wanted to talk about the volume security standards. Are you all here.

A

Hi uh did you I have the doc open. If you all want to talk about it and I'll give the pitch of uh where you guys are trying to solve.

B

Yes, thanks thanks uh yeah, so uh we have this issue uh a potential security problem with the volume mode conversion- uh and we discussed about this approach already in six storage, so just want to bring it up with a sick off and see what you guys are thinking ronald. You want to take it over.

C

Hey yeah sure, um so I think uh zhing has had a bit of conversation in the past with I think either author security about this, uh and basically we have a small issue um or a potential security issue with our volume snapshot feature um and we've been discussing recently about how we would we would want to sort of uh address that. um So what we want to run by, I think, without getting into too much detail about the snapshot. Issue itself uh is actually introducing something called um something like volume.

C

Security standards are basically based off of uh the pod security standards that was uh sort of uh introduced recently right.

C

So if you can sort of scroll down um below the non-goals yeah, so um so, just like how bot security standards sort of introduced like these standards and then sort of a controller to enforce that standards, but also um not very limiting in a way we were thinking of introducing something similar um but related to volumes instead of just spawns.

C

uh So we want to run that by sight and sort of get feedback not now necessarily, but definitely on the document as well and see like if the direction that we're headed in uh sort of makes sense.

C

So, um if I just had to summarize this document, um so we wanted to introduce like volume, security standards which are basically right now privileged and restrictive, uh where privileged would be the least restrictive policy and restrictive would be the most um and right now. We just have one use case right. This snapshot issue that we sort of have known about and want to solve, so it sort of made sense for now to um just have like two standards defined.

C

um In addition to that, we also wanted to introduce uh this admission controller, um the where like, where to introduce that mission controllers. It's probably a bit too early to talk about that. It's probably going to come up in the design, but the idea is to have an ambition controller which is going to go ahead and um sort of look at the standard and look at the policy. That's applied to a particular namespace and then go ahead and try and enforce that policy.

C

Right so just like pod security standards and and the whole um the way that works right now we were thinking of introducing.

C

Excuse me in three modes uh where we have enforce audit and one um and again, if enforce, is labeled onto a particular namespace. Then those uh standards applied to that namespace are enforced. When uh this admission controller sort of intercepts calls to create a volume um in that particular space. uh If it's audit, then that that's gonna be just audited and if it's one, um then uh if the mode is one, then we're just gonna want the user, but go ahead with the operation.

C

um So a lot of this proposal is again based on bot security standards. uh We looked. We also looked at board security uh policies or something like pod security policies, uh but sort of uh had the idea that you know, since that was uh like the community moved on from that in terms of pods. It doesn't make sense to go back to that for volumes, so um so yeah, that's actually, basically just a bit.

C

I didn't expect to finish it in like three or four minutes, but uh really the idea is to uh get feedback from sigoth about what they think about something like this applied to volumes, um in the sense that when you create a volume right now, our use case is just singular in terms of restoring a pvc from a volume snapshot, but we want to build something that can be extended in the future for uh other potential sort of security issues or concerns that may come about so yeah. Anything.

C

You think you would like to add to that.

B

Yeah, I think you covered pretty well.

B

C

Comments from seagate, so the main, I think the main focus would be um getting feedback on this document. This is obviously very high level at this point, um we're also drafting at cap, uh which we probably would share with sigoth as well, uh but just to get an idea of of what you guys thought when you went through broad security standards and would that be a similar mechanism be applicable to volumes as well.

D

One of the things that we uh was was probably important when we started with pod security standards was um enumerating like the things that we wanted to protect, and so this seems to be like describing a mechanism uh without a really clear sense of what it's trying to protect against.

D

So there's one example use case here which, um like the mismatched mode thing like uh getting access to a volume with different modes, um but beyond that, it's it's hard to see like what um is being protected against, and so it's hard to map that to like this three-tier, uh restricted, baseline, privileged mechanism.

D

Yeah, so I I guess the the one example use case, I'm having a hard time mapping to like how how problematic is this like running a running? A privileged pod uh seems pretty uh clearly privileged, like accessing uh volume with different modes might be okay, but might not be like.

B

I think maybe ronna can you go over there. The use case how this problem is discovered? uh Maybe we were to the beginning. I think we maybe skipped that part.

C

Yeah sure so uh I think um so. The the one use case that we do have is definitely well defined in the sense um so right now the issue is, and the reason we started thinking about this is that um a user can basically create a pvc with a volume mode of block um and then write malformed data to it and then take a snapshot of this volume.

C

And then, when they restore the snapshot as another pvc, they can actually specify the file the volume mode as file system right. Even though it was originally a block volume. And then um the potential is that when a user uses the second pv pvc and kubelet tries to mount it.

C

uh There's there could potentially be a crash right and the problem that we have is that, as you said, this is sort of a valid use case, as well as an invalid use case, uh because um there are like sort of backup uh vendors who use this mechanism uh to sort of perform their backup and restore.

C

So um so we can't uh sort of reject that all together uh and but but at the same time there can be like a malicious user that that just wants to you know, crash a system. So that's where the sort of dilemma comes in and and that's how we sort of ended up at this whole, defining um like how do you define who's, a privileged user and who's, not right who's allowed to do this operation who's? Not, um and that's where that's how we got to like this whole name, name, space level, thing.

D

Yeah, so one of the reasons pod security standards were useful was because there's like 35 different things in the pod spec that, uh like it's just a very huge surface area in terms of the api and bucketing and simplifying like what uh what a pod can do, was valuable.

D

If there's like one problematic scenario- and there aren't a lot of other clear scenarios.

D

Having like a three-tier policy mechanism seems like overkill to me,.

B

uh So jose, if you have any other alternatives that you know we are offered as well uh yeah. This is definitely just one case, but because it is a potential security issue right, so we thought we have to find out find a way to solve it, but we couldn't find out other way in a more simple way to solve this problem. That's why we thought you know this uh uh security standard seems to be a good approach that we could. We could borrow.

D

I mean so just straw, straw man, uh proposals like if, if the expectation is that, under normal circumstances, a pv won't be uh like accessed in both block and file system mode like it'll, be one or the other, um or a snapshot like uh if you required some indication on a cluster scoped object that this mixed-mode access was okay or expected.

D

That would be one way to do it in a pretty lightweight way.

B

This is a not a cluster scope. This is a user scope, so this is basically a special use case for backup and restore and backup vendors use this to take efficient backups. So they do use this convert conversion because you know normally your volume could be could be used fastest mode, but underneath it is a block, so they convert it temporarily to block during backups.

B

But then you know when they restore it. Of course it will restore back to its original mode. So it's definitely a convenient way for backup vendors to do efficient backups. uh But then there is this potential potential problem. Potential security problem.

E

D

So would that be like something on the on the volume plug-in to say, like this volume plug-in expects to restore from like block snapshots to file system, pvcs.

B

Well, this is a what most plugin would allow this one. Normally, it's not the plugin, it's the backup vendor we'll be doing this temporary conversion. So it's not like at the see the driver level actually so because backup vendor when they do that operation. It's going to it's going to create this uh a temporary temporary volume and changing that to block mode, so they can get the change blocks.

B

uh So, yes, it's not something that is controlled by the at the driver, see the driver now actually.

A

If I could ask a question on this, so I think the feedback I'm getting from jordan is, it seems, like you, have one very crisp use case and while it seems probably important to handle it somehow like there seems to be like there is a potential for quite significant harm. If it is exploitable um uh it might not warrant the complexity of something like psp or pot security standards.

A

um Have you considered a a much more targeted thing to this problem like a not a generic mechanism but a targeted like admission controller, either as a web hook or I don't know if it would be entry or not?

A

But you could imagine that when such an admission controller is enabled it would, by default, disallow change in mode unless the actor that was requesting it or or some out of band way like I don't know some extra permission, that the actor has it's allowed to request it with a different mode than it's supposed to be in so it would be. You know like expressed through some. You know admission restriction in the kubernetes api target, specifically towards this just one use case right.

A

So the idea being that the the backup, vendors, their controller or whatever is running that's doing this backup would be obviously given that permission, because it's it's doing a good thing. It's backing up your stuff, but normal users will not be granted that permission and or maybe they would be under special circumstances where the cluster admin has decided that it's safe either, because the user's trusted or the thing that they're interacting with is benign.

B

um Did we consider, I thought we looked into that earlier? Did we just uh a simple web controller? I I don't remember.

C

um So I think, uh where we got stuck and where we decided to move to like the standards uh was, where was how do you uh define who's a uh sort of privileged or allowed user and who's not, and how do we differentiate like within the same name space or something like that, um and then that's when we move to this? um If I'm not wrong.

A

D

Sorry, when storage vendors do this uh cross mode snapshot, restore um where what namespaces are they usually doing that in? Is it the ones the user's stuff is in, or is it in a separate like backup, restore controller namespace.

B

It's a username space because they're they need to backup restore the user workloads.

D

Okay, um if that's the case, then I'm not seeing how a namespace level policy to allow a cross mode, backup, restore.

D

Would be safe if, if, in order for the vendor to do this cross mode restore they have to allow like set it in a privileged mode or something to allow this cross mode operation? That also gives the user access to do cross mode operations in that namespace.

D

To me, this really seems specific to the the volume type like if certain volume storage vendors are expecting to do these cross mode backup for store. Am I miss reading that yeah.

B

It's not an attributed volume type well well, so, while back of vendor they do have to do something that is kind of a um you know privileged to to make those things happen, but uh but those are the user workloads that they have to. You know they back up. If they are, you know usernames visibility story.

B

They have to be in a username space as well in the middle, the data will be moved to someplace else in the middle right, so so there's a transition somewhere in the middle, but when they, when they convert this mode, uh that is the the the mode is specified on on the pvc itself right. So that's why I'm saying that's the it's a user mode thing um when they uh change that and then and then they get the data. Basically.

D

Yeah, I I guess.

B

I'm just not saying.

D

How like, if the privilege is, if we're calling like changing a mode, an escalation or a privileged, whatever privileged operation, if that is happening in the same name space that the user has access to, that seems like you're losing whatever protection this is providing for the duration of that operation. If we're concerned about users like maliciously, uh you know creating a pvc or snapshot in block mode and writing bad data and then doing something with it. But in order for normal backup restore operations to happen, we have to expose the username space to privileged operations.

D

So you're saying that opens a window when the user could do malicious things.

B

So you're saying user can always do this anyway, it doesn't matter like. Even if we have volume security standard, they could still do this. That's what you I.

A

Think what jordan is saying is the tenancy model of the hot security standards. Is that it's per name space? What you're saying is you need to somehow slice up a namespace, but in like a weird transient, sort of way like saying that this privileged actor can somehow change the mode and do special things, but at the same time, like normal users, can't leverage that, during the duration of the operation like like. Maybe if, however,.

B

They're saying you're saying the name space once it's applied, then that's like for everybody who has access to that namespace. Basically, that's what you.

A

Yeah, presumably the user who's running the app has access to the name space to do like happy stuff. So if you're in the middle trying to have like a a higher, really a more privileged actor, do something you can't use the name space as your tenancy boundary. You have to have some other concept that says that this actor is allowed to do something special, but not some other actor right.

A

So, like the like example, I know that I remember from openshift is the reason the pod security standards have the hard-coded list of like this user special. You know as a config flag is to support openshifts build controller which does privileged operations in shared namespaces with other untrusted tenants, but it does so by very carefully coordinating with the underlying container runtime and the kubernetes api in a very orchestrated way, but like it's only safe because they control like the whole vertical stack, and they can do that.

A

But that's it's effectively impossible to guarantee unless you control the whole stack in a very particular way. um At least in that in that particular example,.

B

Okay, so okay, so you're, saying.

F

B

It's actually not even possible to secure this one. Then there was even all right.

D

uh Not not with a namespace level control if the backup operation you want to trust and let do privilege things happens in the same namespace as the users, the users that are maybe untrusted.

D

B

D

So alternative things to consider are uh if there is um an attribute on a cluster scoped object that could be required to allow this cross mode operation. That's one possibility. Another possibility um is.

D

A like a some sort of admission check that that would actually do an authorization call. um So we uh an example of this would be uh when you sign um or approve certificates.

D

uh We actually do an authorization check to make sure that the api, requester signing or approving a certificate has access to that particular signer name. So you could do something similar. You could say if you're going to mount a restore from a snapshot in a different mode than it was produced. You need to have this authorization um so that there is prior art to that it gets a little fiddly, but it's doable.

D

um I think the question there would be uh about backwards. Compatibility like if there are people already doing this. um What would they have to do or set up in order to continue to be able to do what they had been doing without getting interrupted.

B

um uh Okay, so about this uh uh authorization check, do you have any any examples? There's something yeah.

D

I'll uh drop a link to the uh what we do for the certificates stuff, okay,.

G

Can you explain the crash?

G

Is this something that always happens or is this something happen that happens in the presence of you know, common uh kernel bugs.

B

uh This is a okay. This is actually never happened before. This is a this is just so actually young brother, a.

D

Radical crash like it, it could.

B

This this is just a potential yeah, so actually young brothers up here. I think he is reading those uh you know um kernel. You know potential colonel uh box, uh so he was basically saying that if there is such so currently there's none that's why this one is actually kind of a we're, not saying this is like.

B

We have to fix right away. We have you know, but we can take time to think about how to fix this, um but it's just this type of a bug in the kernel it come. You know from time to time it shows up. So if there is such a bug, uh then, if someone, you know uh write this, you know more formed data and then it could potentially cause this to happen. So that's yeah, that's how we come up with this. We actually send an email to. uh There was a security group.

B

This was maybe a few months ago see we actually asked about this problem. um So but then you know, uh I think we were told that yeah. Maybe this is not a.

B

Maybe it's not like a urgent or something right, but we still should look into it uh and then, when we see uh when we see the the power security standard, so I thought oh maybe this is a good way to solve this. We we were having trouble finding a solution for this problem, so.

A

um So if I understand correctly like if we're looking at the page, I have on the screen, the concern is right. So this is from 2019 so two years ago, whatever. But if another cv like this is discovered or if you're running an old kernel version that isn't patched with this. Theoretically, the kubernetes api, through these mechanisms, gives you an entry point to cause the kernel to consume a malicious uh volume which then could allow you to execute code as the kernel in the kernel.

A

And thus, uh if you know, if you had like the multi-tenant cluster or whatever, that you you've broken that tenant boundary uh and can possibly do whatever the tubeless is able to do environment, yes, um and so it's theoretical in the sense that we're unaware of any current um cves.

A

We have an avenue where untrusted data is being provided to the kernel, and we want to limit the opportunity for basically such things to happen, to actors that we have some prior trust in that summarize that okay, yes, yeah.

D

I I put some links in the agenda. Do you want me to add them to the agenda or comments on the snapshot, restore doc.

B

Yeah, if you can add to the doctor, be awesome uh yeah, so then can maybe look at this other yeah. Okay, then also, uh you also mentioned like class level control. Maybe if somehow we have some parameter in cluster scope that we can use to to check like like adding some another parameter at a classic scope using that.

D

I'm trying to remember all the objects involved in a snapshot restore so ppc and pv.

F

D

Good example: good use of attributes of the persistent volume being outside the namespace and not being under the control of the namespace user for snapshots are all of the objects in the same namespace or are there any cluster scoped objects.

B

And there is a cluster scope object, which is the one in snapshot. Content which is similar to pv, and one snapshot is the name space same at the in the same name, space as pvc.

D

B

Yeah similar so.

D

One possibility would be uh if there's a trusted component that is involved in producing these backups um it would, it could be given access to annotate or set a field on the volume snapshot content to say this is even though this was produced in block mode, uh it's okay to mount back in file system mode, so it it seems like it would make sense for the snapshots to record what mode they were produced with um and then under normal circumstances.

D

They should be mounted back in the same mode they were produced with. But if someone wants to produce one with block mode and mount it back in file system mode, that could be okay, but they need to indicate that this is expected and they are trusted, and this is not an attack.

B

Okay, so this would be for the like the backup software who, who is responsible, taking a snapshot. They can add this annotation on the content. This way they look at that they look at that hint. They would know uh at the restore time. Then it's allowed.

E

Some something.

D

Along those lines, okay, so under normal circumstances, a normal user producing a snapshot doesn't have right. Access to the cluster, scoped snapshot, content objects, but a backup controller like the privileged backup controller for the cluster, could reasonably be given access to those.

B

I see yeah. Maybe we can explore that right. um That would be relatively straightforward. Actually, if we do it that way: yeah, uh okay, so yeah. So we'll look into those two uh two suggestions.

C

Yeah, I've written out a bunch of notes from what you guys were saying. So I.

F

You definitely have some new.

C

Avenues to look at as well: okay,.

B

Yeah, I think this is yeah. This is very helpful.

C

Yeah yeah really helpful thanks a lot. Everybody.

B

Yeah thanks yeah. So if you have any other comments, please uh you know add to that dog.

B

So yeah they are uh we're hoping you guys to review when we, when we, uh you know, decide which approach to go.

A

Cool uh anything else on this guys before we do the discussion topic. You know comments.

A

uh So I just added uh this one discussion topic, um so I starting this week I switched the weekly triage meeting for a cigar to try to focus on 123 stuff, because code frieza is coming, so I've just been focusing on that, and during that time I saw a bunch of stuff with the new paw security stuff, and I just wanted to get an update from jordan kim anyone else. Who's been working on. It um is basically is there anything you all need help with.

A

Is there things that need to be reviewed, critical items for beta and 123 any just.

D

um Things are going pretty well uh we're down to about three or four pr's left before beta promotion. um All of those are all, but one are actually ready for review and in review. um So the.

D

We've got benchmarking in place the uh the default path when you upgrade an existing cluster to a version that has this feature enabled where things are privileged by default and pods are allowed, is very, very, very, very performant, which makes me happy, uh basically takes a couple hundred nanoseconds and like one extra allocation uh for every pot, that's admitted so um the impact of enabling this feature on existing clusters is essentially nil.

D

um Tim is working on getting the metric stuff in place. uh Sam was able to get the webhook version building and an image built and manifests put together so that people can install this on existing clusters and try it out. That's the first pr in that list there, so that is an emerge. Queue.

D

And then there were a couple other things listed under the beta items around um like prioritizing, which pods we check in the namespace. If there are a bunch of pods, we want to check unique pods first before we start checking like all the pods that came from the same replication controller, but generally it's on track. I would expect the code work to wrap up this week or early next week and then the docs work to be picked up to update for the docs for the step in beta.

A

So so what was there.

D

Any of these they needed an extra review or anything, or are we good right now um everything has an owner and a reviewer. uh I think probably what I would want help with once the uh web hook um once the web hook. Pr merges is people like actually trying it out like you. Can uh it's a two two line uh installation, so you make certs and then you control apply and that installs the web hook into an existing cluster and you can play around with it.

D

um So I would, I would probably anyone who's interested in engaging and helping, and uh I would probably get him to do that play around with it.

D

So I I will I'll plan to send an email to the sig off mailing list once that merges and kind of an open call for that uh and link to the docs. For that.

A

A

Does anyone have any questions or concerns on that? We don't have anything else. So if anyone has anything they want to bring up, we still have.

A

My csi driver ready.

F

uh It might be because of the issue we had with ci.

F

That was impacting a lot of different projects, a while back.

A

Okay, I'm just curious.

F

A

F

First guess, but I need to dig into it.

A

So you guys, uh if I remember correctly, though you successfully, did your one dollar release right? Yes, yes, so, if I understood correctly, you guys went ahead and just promoted the apis to v1.

F

That's right for jordan's suggestion.

A

Excellent, well, I'm glad that's all there. um I think I think a lot of folks have been waiting for that. That's exciting.

F

Yes, go. Try it if you haven't. Thank you and feedback. Welcome.

A

uh I'm trying to look, but it's the same, I'm saying there are youtube discovery things not working.

A

um This is recently jordan. Can you remind me who owns this feature? Like I know, segato is like a person.

D

Can you zoom in, I can't read what you're looking at I was just I just saw sarasota county shore. Discovery was failing, uh michael toffin was the one to set. That up is that.

A

huh Those all seem recent.

D

Upgrade that's interesting.

D

I can never remember which version stable one represents.

D

I'm wondering if this is one of those situations where, like the ede test in 122,.

D

uh Had test config that was not present in 121 and like it's running the edg test, swamp from one branch against a cluster configured from another branch.

A

uh Good idea, maybe yeah.

D

um I'll I'll ping michael and see if you can figure out what's going on with that.

A

I'm not sure I stay for citation.

A

I'm not seeing any I'm not seeing any other sort of repeat offenders other than service account discovery.

A

That's good! I guess.

A

Well, I don't think we have anything else on the agenda. uh If anyone wants to talk about something real, quick, we can't otherwise we can keep everyone back. 17 minutes.

A

um See you all in two weeks, then yeah? Okay, yes,.