Kubernetes Data Protection Working Group, 14 Jul 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Storage - Data Protection WG Bi-Weekly Meeting 20210714

Description

Kubernetes Storage Special-Interest-Group (SIG) Data Protection WG Bi-Weekly Meeting - 14 July 2021

Meeting Notes/Agenda: -

Find out more about the Data Protection WG here: https://github.com/kubernetes/community/tree/master/wg-data-protection

Moderator: Xing Yang (VMware)

A

Okay, hello, everyone today is uh uh the 14th, I think july 14th 2021. This is the kubernetes data production wing meeting.

A

So today, uh ronak will go over a design for the volume mode conversion issue that we discussed a while ago, and then we can do an update on the whitepaper progress all right rona do you want to share your screen? I think that's easier.

B

Hey yeah, I can do that. Okay, thank you. I'll! Stop! Sharing.

C

Okay, so let me just.

D

So can you just see my screen?

D

Yes, yeah so hi everyone, so um so uh so I just I was speaking to shane the other day about this issue, so I just thought I would take a stab at it and um basically we were looking to get more uh sort of input and ideas and stuff. So the document that we have here is is pretty high level, um so I probably just go through the problem and then sort of high-level solutions uh and then of course, uh any feedback now or offline later as well as is welcome.

D

So the issue uh is basically um it has to do with the volume mode uh sort of conversion issue uh where um you could take a snapshot of a pv with the volume mode of block right. So technically, there's no file system uh and then um a user could sort of maliciously um restore that that sort of tv uh sorry restore that pvc from a snapshot and change the volume mode uh to a file system and then try to mount it and then eventually that will crash.

D

um So right now, I think there's nothing that prevents a sort of a user from doing this and the sort of uh solution of just preventing that altogether is not an option either, uh because, basically it's a valid use case. um I I don't want to get into like the details.

D

Honestly, I'm not very like uh sure about how backup vendors work either, uh but basically a backup vendor uh could potentially want to, uh I think, temporarily, uh sort of restore a snapshot uh using file system mode and then sort of delete that that sort of temporary um volume that they create.

D

um So I was just looking at like uh like basically stuff like pod security policies and recently that was deprecated and replaced with bot security standards.

D

So I was just looking at that and seeing it seeing like how we can sort of apply those concepts over here um to sort of fine-grain or sort of provide permissions to what a particular user or service account can do right. So uh so this document basically um speaks of sort of both of those solutions uh where we have, um I mean.

D

Obviously, the names are just like not really thought of in detail, but basically a volume, security policy or a volume security standard uh and how we could potentially apply that to um our problem over here. um So first off like uh basically, we need to store the the pvc uh pvcs volume mode right when we, when a user takes a snapshot of a pvc.

D

Currently uh we just sort of populate that pvc name uh without any volume mode, so one change would be to sort of add that mode to the snapshot status and uh that information will be used later when a pvc is being restored from a snapshot.

D

So any questions uh up to this point.

E

I was just gonna mention that um the one thing preventing crashes is is a lack of bug in the current lack of bugs in the kernel right. We're not aware of any actual bugs that, let you do anything evil today, but there's a potential for it. Yeah.

D

Yeah exactly so, that's probably why it's not like a like a p0 at this point, and uh it's been alright till now, right cool, so um so the first sort of policy or solution that I was looking at is like volume, security policy, uh which is basically similar to um like the pod security policies that we have today, uh in the sense that um basically, we would introduce, like this uh cluster level resource um where you, where uh you create a sort of a volume security policy, and then you fill in the spec of that policy with whatever permissions you want to give to that.

D

To that particular particular policy. And then you tie that policy to either a user or service account. um So as part of this proposal, I'm only sort of suggesting the addition of a single field called allow volume modification uh which would default to true um and, of course, in the future.

D

If we come across whatever other uh sort of security nuances that we want to look at, it could easily be just added over here right so um and that volume security policy would sort of be tied to a user service account similar to what psp does right.

D

So when uh in psp, like port security policy today, uh if you need to tie a policy to a user, you sort of create the role, create the role binding and then um sort of uh mention that you know that particular service account can can use this policy like you have to specify the verb and stuff.

D

So I haven't given that in detail over here, but it's all in this, uh this link on the kubernetes docs um and then, of course, uh we would introduce an admission controller uh which basically intercepts requests um to create a pvc right. So what that admission controller is going to do. Is it's going to look at uh whether you know the source of the pvc, whether it's being restored from a snapshot, and if it is, then, is it modifying the volume mode right?

D

If it is, then it's going to look at what the volume security policy is for for the user? That's trying to create that pvc right. uh If no policy is specified, then we're going to default to true um in the sense allow it to go through because that's sort of the default behavior right now.

D

However, if that particular user is not allowed to change the volume mode, but is attempting to do so, this admission controller would reject that request. So the pvc would never would never be created on the api server.

D

D

uh Cool so so psv actually was deprecated in kubernetes 1.. I think it's 121 and I think it's going to be removed all together in 125 or something and they've sort of moved to this thing called pod security standards.

D

uh Also like a three-pier pod security proposal, um it's it's actually pretty interesting, uh and this proposal is actually very detailed. So uh I've again just tried to take concepts similar concepts from there uh and and see if we can apply it to uh sort of our use case over here in terms of uh volume creation, instead of like pod creation um and the reason they've done, that is basically they're, a bunch of like limitations with bot security policies.

D

uh Again all of that is sort of given in this proposal. So I've not I've tried not to sort of duplicate information um uh but anyway, so this sort of volume, security standard uh would support two modes which would be privileged or restricted.

D

uh Privilege is basically allow the widest level of permissions, so no restrictions on on creating pvcs or you know, modifying the volume mode and restricted would be the most restrictive policy which you know lets you fine-grain your uh sort of security model for creating volumes, and the difference here uh from pod security standards is that um you actually would label the name space. So, rather than so, one of the uh issues with tying a policy to your user um is, is I mean think of it like this?

D

I could, I could be a user that creates a pvc on my own right. So in that case I'm the user, but a pvc could also be sort of created as part of a stateful set. So then what is the user in that case right? So I think those are sort of the uh like uh nuances or sort of these smaller issues with psp that also apply here.

D

So the idea here is that you actually apply these standards to a namespace rather than a particular user, and that is in the form of labels. So um there are sort of three uh modes that we would allow again similar to bot security proposal. uh The the broad security standard, uh which is enforce audit or one so the idea behind that is that um sort of enforce is uh you know if a policy is violated.

D

It causes that pvc creation to be rejected in case of a violation but audit and one sort of lets the pvc creation goes go through, but um still you know uh sort of informs the user in in a couple of ways either by auditing or you know, on the cli and the reason we do.

D

That is because you know uh customers might be wanting or users might be, wanting to move to this model without really or would rather want to sort of see how the future works before you know enforcing these uh sort of policy constraints so um so yeah. So basically we would add it uh as a label to the namespace, uh where this prefix is obviously still undefined, but so I guess kubernetes dot, io dash, something sounds something slash enforce audit or one, and then you would specify the policy level, which is either restricted or privileged right.

D

So um so, when this pvc is created, uh the admission controller would basically read the label of that of the name space that it's created in and then decide what policy needs to be applied and in what mode.

D

So again it would compare the pvcs volume volume mode to the sort of uh source snapshots volume of volume mode and then, if they match it's all good, if they don't match, then it would look at these labels and then uh take a decision accordingly um and and yeah. Basically, that's very high level sort of uh blast of information- I guess uh so yeah.

D

So I just wanted to like sort of put this out there uh and, of course get your comments and see what what what everyone thinks about sort of these ideas and, of course, any other ideas. If you.

A

A

Rob uh so you're asking if there are any existing kernel or existing box for this. I think right now there are no no.

F

I don't want to derail on. You know just discussion on that. Okay, you know on that too much, but I'm just curious. If there's any communication with the you know, sort of the the linux proper community around hey is this the right behavior and obviously it's their ball of wax and that's a whole. That's a whole other topic as to whether or not this community can influence that, but I'm just curious if there's been anything established on that end of things, I.

A

Think young can probably answer that right, young you, there.

G

This this crashes are treated as cves on the canvas side. But, uh like you know, the fire system are. The fire systems are huge like xfs x4, uh so these cvs happen from time to time, and then it takes quite some time to have that patched in all distros.

F

Okay cool, so so it is it's it's more of a you know in pragmatic terms, deal with it here prevent the problem address it whatever, but but there has been that communication and let the real slow moving wheels turn on the linux side and maybe one day, it'll do more graceful things.

G

No well the kernel, it doesn't know that this volume has been available as block and now it's mounted it doesn't know because, like for for canal, it's just a block device. He doesn't know who modified it before.

F

Yeah understood, okay, like I said, didn't, want to derail but glad to hear there has been. You know there is some intent on the linux or in the kernel side to maybe improve things over time, or at least there's an awareness such that they could potentially have intent later anyway. Let's not derail thanks.

H

I I have this kind of stupid question that uh the the the when in the statement of the problem, is it looked like that um the couplet when we tried to mount a corrupted um tv uh it will crash and we're trying to find a way to uh address the problem that you know to avoid the situation like this happen, but uh have you ever have you ever I mean have we we ever think about? Why don't we just address the original problem?

H

That is when we mount some pv that it doesn't have uh cora does that does have corrupting file system so that it will not crash. So that is, like uh you know,.

A

To do I don't think that is that possible to check that, or are you talking about kubernetes, to check this? I I, uh the the original yeah.

B

H

A

That is actually to say if some uh some malicious user tried to do that. This is not, of course, not like normal behavior right, uh so uh I I don't think it's actually is that actually even possible to find out if the, if the file system is a corrupted or it's you're always impossible to do something.

A

The only way to prevent this is.

E

Is to not have bugs and, and the problem is, is that historically, they find bugs. You know a couple a year.

A

Right right, this is yeah.

E

So so the idea is given that there have been bugs and there will probably continue to be bugs, even if we don't know about any bugs right now we got to do something and and and just trying to scan the file system or trying to ensure that it's not corrupted is unacceptable, because that assumes that.

I

No, that's not.

H

What I meant, what.

I

I mean: is we how.

H

Can we avoid crossing uh if.

A

Well, that's probably that's a kernel.

B

A

Right, so maybe young can answer that question there are bugs, so we I think young was about actually brought this up right. uh So yeah, I'm probably uh you can answer this one better. You are familiar with the those type of bags you saw those before.

G

There are bugs in kernel some few times a year. Somebody finds how to crash your kernel using corrupted fire system, either x4 or xfs, or an ntfs or whatever these bugs happen, and there is no way how we can check before we mount that. The first system that we are going to mod is going to crash the kernel or not.

E

Yeah, you have to basically trust the user at least a little bit and that's what this security policy is about. Is it's about, establishing who you trust and who you don't trust, because it's inevitable that bugs will creep in and malicious users will exploit them.

J

And I think in this case, like a lot of these bugs manifest in ways that, like you know, a typical fsck won't detect.

G

Okay, I talked to kernel guys here in red hat and they treat them seriously like. This is a cbe. But the severity of the cv is not that high, because it requires root privileges and if somebody.

J

G

Can mount can crash the camera then like they have many other ways how to crash the kernel. They don't need to mount like they can stop the they can help the machine, basically even without mounting something wrong, so it doesn't have the highest uh cve rating, but it is fixed. It is being fixed by these throws.

G

Now in kubernetes we kind of have we allow users to remotely to attack like you crash nodes, if somebody can create snapshots and tvs pvcs, then they can crash. Basically, the kernel.

C

Okay, I had another question um so uh so the other way we can provision now pvcs. If you have a pre-existing snapshot, you can pre-provision a volume, snapshot, content and from then uh create the volume snapshot. So then, in that case, where do you get the mode from.

D

Yeah, that's actually a good question and I think that's an open issue that I've sort of listed at the bottom, which I I didn't really go through, which um basically for pre-provision volumes. I I think zheng pointed this out that uh the same thing is what you said right: the status wouldn't be populated.

D

So I think that's an open question, but uh uh again this is a high level design. So I didn't really think about like stuff like that.

A

Yeah, so I wonder uh yeah, if that's the case, maybe at least this process cannot handle that at this point, um so I think at least we want to start somewhere right. So maybe let's say, uh of course we can continue to think about how to how to handle that case. That does not look like we can cover that.

J

With this design, I think in the in the pre-provision case, somebody with admin privileges has to create that the snapshot to import right.

A

But that's so this is the volume snapshot. This is the name spaceman, so the admin will be creating the content right. uh Unless, if you're saying that we do, we add that in and also when you're creating that the con the status will not be populated right, so admin normally probably using the cli uh cubacado to create it and it so they won't be able to change the status unless they write a controller.

A

So it will be a little too much to recall them to add.

J

That but but I guess I think the main thing is since an admin has to create the content object. That's that means that content is more more trusted.

A

Are you suggesting to like we have this field in both content and the one snapshot so right now in this current proposal, this field is only in the volume snapshot status, not in content status. So I mean, I guess there are a couple of things here: do we also want that field to be in the content status? Because that's one question and the second question is: when the admin creates it: normally, they use the cube card, or at least right now.

A

You know when I try to say I don't write a controller just to create a content with this status uh populated right, so I mean how can without writing a controller? How can the status will be populated with the cut or you cannot yeah.

J

I I think my main question is is in the pre-provision case. Do are we as concerned because the content has to come from.

K

Someone with admin.

J

E

I guess you could, in theory, perform an exploit. You would just have to trick an admin into importing a snapshot that had been maliciously constructed.

A

Well, it's not the snapshot. That is, uh that has a problem is it, it is the it is. The voiding ray.

E

uh All right: well, you create a volume yeah with the raw block volume. You fill it up with a malicious, fake ext for file system or xfs file system.

J

Or whatnot you snapshot.

E

That and then you trick someone importing it as a file system snapshot. You create a volume from.

A

Yeah, so I think it's a it's probably less of a concern. Yeah, uh it's still possible. You also have like internally. You also have a malicious user. I was just reading some security documents. Internally, admin.

E

We can't stop people from creating these malicious volumes and snapshots of them, but we can at least remember that they were raw block volumes at the time that they were yeah.

A

But I mean so how what do we do? How do we do that? For this, uh this is privileged provisioned. How do we get that information? I guess.

E

I I like the idea of of amending the spec and and tracking dfs type, adding.

A

The spec, okay, um so yeah right now. This is the status because because if we add it in in the spec, then of course then the admin can add that as well is that you could.

E

Put it in the volume snapshot, content spec right, so that the controller that constructs the volume snapshot content can know what pvc it is snapshotting in. But then.

A

We're getting into the same we'll have been trying to avoid that uh so the field, so this field will be also okay, so the spat field in this case um it could be either be like a status because it changes the controller or by the user in the preparation case. Yes,.

E

For it to be, csi has to provide it right. Everything in the status is provided by the csi plugin, the the the sidecar just goes and asks the csi plugin. What's going on with the snapshot, you know tell me it's creation.

A

But I'm saying that the preparation case is not even going through the csi right, the content, we're talking about content right. It doesn't really.

E

Right everything in this in the content status comes from the csi driver. The spec is filled out by the snapshot controller, and that is the thing that could copy the the volume.

A

Mode from there, no no, but wait, but this if pre-provision case admin is the one who is creating the spec for the content.

K

E

At that point, the admin would know right. You would know what yeah.

A

What I was trying to say is when we were doing the the snap mode in samsung redesign we're trying to avoid a field in the spec that, like depending on whether it is the dynamical provision or statically provisioned, it is populated by different things, either a controller or the user in the in the preparation case.

A

So that's what we're trying to say is that's the reason we have the the source to different source if it's static and uh dynamic, so, uh but only that the id is still that it's the only thing that is there, the snapshot last id, but anyway.

E

I think we could copy the volume mode off the pvc and treat existing ones as the same way. Pvcs work if it's blank assume it's file system, but.

A

The pv may not be there anymore right. So that's the thing.

E

You know any snapshot created by the controller would just copy it, and then anyone created by an admin you'd have to just fill it in and if you left it blank, it would assume file system exactly like pvcs. Do that then didn't.

A

Yeah, all I'm saying is right. We're going back to I mean I think it's fine, I'm open to this one. Maybe we can. This is an option, uh maybe a ronald. You can write this down. So this is one option that we could uh solve this problem, which is to have this uh in volume snapshot content where we have this field in spec.

E

A

It's the only way to then.

E

At least we would have a record of if someone did create a raw block, pvc and then snapshot it. We would at least know then, when, when they tried to turn it into a faucet and pvc, that they were making the change, and then you could examine some policy to say is that okay or not and for a backup system, you'd, say sure? That's okay, because we trust the backup system before joe shmoe say no we're not going to let him do that.

E

Deny deny the policy you know at the policy level.

A

Yeah, so if it's in sweat, then we then we don't need that in the status I think, do we still need that in the status. uh Oh, we probably we still need that in the status, because that's like create create volume time we need to.

A

I guess it's okay, this controller can can see the namespace here as well. So so maybe that's something we can we can decide later if we want want it to still be in the status, sounds like it's not necessary anymore.

E

I say put it in the spec of the content: object.

A

Yeah, I was just saying like if, if it's there, it's still necessary to have that also in the status field, volume snapshot is not no longer necessary, so we just have that at one place.

E

We only need it in the one place. I think.

A

Okay, all right: okay, um okay,.

D

Let me actually look at how the api uh the api would change of the content. Then, uh let's see, if that's that's an option.

A

Okay, um so that's for okay, that's for the preparation case.

A

And the other, the other opening issue that you have there is a how to prevent malicious user from updating namespace label. I guess that we probably want to see uh how does the like the pod security standard? How how do they solve that problem? I think we'll be just choosing yeah.

D

Yeah, I need to uh probably look at that again: I'm not sure how they do it right now.

A

D

Yeah, so actually that's it for me. um I would appreciate if anyone can, if they have any other comments later, just leave it on the page, and I can start thinking about this in a little more detail as well.

A

uh Okay, michelle do you have uh other concerns with us? Should we can we move forward with this, or is there any.

J

Yeah, I I think this sounds good.

J

I guess have we: are we still deciding which approach we want to take.

A

uh Yeah, so we can actually maybe take a make a call here and we can should we also um bring this up with the security team and ask them probably. I would think they will be suggesting option two, since that's what they are doing with the pod security policy.

A

J

A

I would probably.

J

Lean towards that too, since it's more closely aligned with the model that they're going with.

A

Okay, so, okay, so should we just uh maybe, as rona, can clean this up and make that option two as the main option and the other one as alternative and then maybe before.

A

Turning this into a more formal cap, we should should we ask the security team to take a look first to see if they have some big um yeah. That sounds good to me.

L

M

Cool, um I actually have a question regarding this, so um I'm curious: why would those labels be part of uh the namespace and not the part of the pvc itself.

M

In this case, like we're, treating the entire namespace as one policy, so that would be our unit of granularity right.

D

All right so uh so I think the difference is who applies the label to the space right? um If I'm, the user, creating the pvc then- and I want the highest sort of permissions, then I would just say uh like create this pvc, with with no permission with no uh restrictions right uh versus.

D

Probably the admin would add that names at the label to the namespace.

M

um Yeah, but in that case I guess, would it be a valid use case that a user in would want uh two different set of policies within the same name space, or we are saying that that is not a valid use. Skills.

D

Yeah, so I think that's right so uh in this second sort of when you apply to the namespace, rather than having the uh thing like the policy assigned to the user. In that case, you're saying that for each namespace we just have one sort of standard um and any namespace. Sorry any pvc creator, and that namespace is going to follow that standard.

M

Okay sounds good. um I I was just curious like why. Why would that like? What are some of the pros and cons of having that versus having this um specified to a user, um but I guess we can. I mean I can add some comments to the document.

D

Yeah, let me maybe I should add that uh as a point as well, I I can. I can do that. Yeah make sense.

D

Cool, so I think that's it from from you can um let me stop sharing okay.

A

I thought there was a something else, but I couldn't remember what that was. I thought was that something else that we also oh, I think, uh can we actually can you go back to share that again, uh so the the new api object you're, adding, is that the that's like under the policy group right policy. Api group is that.

B

um No, no hey! Sorry! I couldn't hear you.

A

Can you hear me now yeah? Okay, can you move to, uh I think, just to move down that not here right there, the let's see.

A

Oh, I think this is. This is option two, so I guess maybe it's okay. I was. I was just uh so so if we oh, so this is uh yeah, so this would be um a new api object, but I think this is since this is option two, maybe it's okay, I was just trying to think like this is like an entry api object right and then because we talked about the admissions controller, uh should that be auto tree or entry.

A

So if this is the entry, but then we need to look at what is snapshot so then, for that I think there has to be actually and then, if we look at the second option, we don't have a new object. Do I we don't have a new api, it's just some labels and obviously fine yeah. Okay.

A

Okay, yeah, that's it um okay! So let me actually go back.

A

Okay, so uh I think xian chen is on vacation and he has no problem with network, so I couldn't join. uh He said he will be working on the white paper on his vacation, so uh I'll think I will see my keys back, so I just want to see if there is any update on this.

A

um So going back to I think, last time you said you are looking at the rustic approach and are you adding.

L

A

The dog or have.

L

How different your snapshot being used and we already have two examples there. I think that should be enough for the white paper. uh We can take a look more investing. You know when we dig into the test. Okay,.

A

You can, maybe just like, add a link instead of uh writing down the details. Maybe that's fine as well just say when there's another example and then just here's the link- that's maybe okay, but that's okay, I'll, take a look and and see if we can uh add this to the main dock, um and also I see that you you're trying to set a meeting to talk about.

L

Yeah, I would like to for us to continue to talk about how to design the cap for uh cbt uh for this, and hopefully that the guy can uh other engineers can help participate in that meeting, and we can.

J

L

That to a test in the next uh um I'm having trouble hearing to hear. Oh sorry, I say that, but hopefully that the community can participate in that design meeting, so we can move the job forward did okay did. Did you find the time that well I sent.

L

A

Was actually thinking do we actually have to have a separate meeting? Can we can we just uh use this meeting time we can. Actually. I think I think that would be nice too yeah.

L

I think this time I.

A

So we don't have anything on agenda for next meeting yet so why don't we just? uh uh I think that reviews I think.

L

A

The call are interested in that anyway. Okay, so let's do that? Okay, uh yeah! It's sometimes hard to find another meeting time that works for everybody; um okay, so um so about this uh quiet, ass, hooks tom. Are you still waiting for steve steve to update doc.

N

uh Yeah steve talked to me. He said he was a bit busy, I'm still very young.

A

N

Doesn't look like it um sure, uh okay,.

A

uh We can probably I look at your dog. I think that looks good. We can probably include that first and then, when when he gets time, he can update that part later.

A

And I also look at the you know the uh you know, those uh information we add in the appendix about the how various applications does choirs and backups that I think, maybe it's better to have them just in a stay in a google doc, and then we can just add a link to that google doc, I'm thinking, maybe maybe that will be better.

N

That would be good. uh I do imagine that we're going to turn this into a pdf, so we probably want to link to things a little more stable.

N

So maybe we make those pdfs or something as well.

A

uh I okay, so I'm not sure that we want to do a pdf. I was actually thinking, maybe oh yeah. We can that we can decide later yeah we can uh pdf is one way. Otherwise uh we can actually just to enter this as a just like the like the text like the uh the markups, it's fine too, I think. Well, we can decide that later.

N

Yeah, I think, you're right though I think it's it's pretty detailed and doesn't fit as well. So you can, you can push it off somewhere else. Definitely.

A

Okay, all right uh see does anyone else have any update on this? um I think ben. I think you, uh your name is also on one of the sections.

A

It's in the boiling populator.

E

Yeah the volume populated.

A

Right up uh yeah, can you write up something for that.

E

I thought that I did a while back because it might be out of date, but there should be something in there.

A

E

A

If you can okay, this is some summary. It doesn't have a detailed design right.

E

Is there an earlier version of this doc because yeah I wrote like several paragraphs at one point.

A

E

A

The same remember.

A

But definitely there are several versions, maybe confusing, maybe that you added that in one of the docs yeah it's possible. Okay, let me actually go look at other other dogs and see if it's somewhere, okay,.

A

All right um yeah other than that I don't have anything else other than that.

A

Okay does anyone else have any other issue that you want to discuss today.

A

If not, that's it for today thanks everyone.

N

A