Kubernetes Data Protection Working Group, 10 Mar 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes Data Protection Workgroup Meeting 20210310

Description

Kubernetes Data Protection Workgroup Meeting - 10 March 2021

Meeting Notes/Agenda: -

Find out more about the Data Protection WG here: https://github.com/kubernetes/community/tree/master/wg-data-protection

Moderator: Xing Yang (VMware)

A

Hello, everyone today is march 10th 2021. This is the kubernetes data protection boarding group meeting.

A

So today uh we have a few things on agenda, so we're going to look at uh this issue with a different volume mode between source and target pvcs uh just to look at what are the issues and the what we should do to solve the problem, and then we can do update of the data protection white paper, okay, so the first first one is this: uh so we actually have an issue open here.

A

So, actually, what happens is right now we actually can uh yeah. You can do this, you, a user, can create a pvc with the fastest mode, take a snapshot and then create a new pvc with block mode using the snapshot or vice versa.

A

So you basically can actually switch the volume mode between the you know, the the original pvc and the final target pvc, and then this is actually used by a lot of backup vendors already, because this allows them to um mount a originally file system mode, pvc as a block mode, and then they can efficiently copy the blocks. But then we actually discovered there is some potential security issues.

A

I see young here. Young is the one who actually discovered this potential problem. Do you want to describe what that is.

B

Okay, I can first thing disclaimer. There is no security issue right now everybody is safe. The only thing is that from time to time there is a security issue in kernel in x4 or xfs file system. So if you mount a volume from untrusted source, you can crash the kernel or do similar bad things.

C

B

C

A malicious file system in raw block mode.

B

Yeah uh exactly so, uh you people usually don't mount five um like random block volumes and people who can mount are usually somehow trustworthy, but in kubernetes case, like a random user, can mount a random volume.

B

Luckily, in kubernetes we do not allow to change the mode of the volume. So if you start with a fire system volume, you have a fire system volume, you cannot access the device and do bad things there.

B

Similarly, if you have a block volume, you either have a block volume and you can write anything there, but you can't mount it or you are trustworthy enough to mount it and then well. You have extra permission. Somebody allowed you to mount it, but in this case, in this snapshot case, you can create a blog volume, write a wrong metadata there. The bet file system that would crash a kernel and then tell kubernetes to mount it by taking a snapshot and restoring it as a file system volume.

B

This way a very ordinary user can encourage. The kernel once such cve is found in kernel. Was that understandable.

C

Yeah but yeah, that's actually that sounds like we do have an issue right now.

B

Well, uh we contacted security team kubernetes security team. These issues are usually treated as medium severity in kernel and having this possibility through kubernetes doesn't add much. Somebody did accounting and it ended up as medium as well.

B

So these issues are usually like fixed backported in the kernels, but we should know they exist and if we can, we could also we should also prevent them.

B

You mean, like.

C

Have a knob that allows an admin to decide whether this behavior is okay or not, or to limit the set of users that can convert back and forth from block to file to a subset of all the users. Is that what you're.

A

C

Yeah that makes sense.

D

C

Given the given, if we believe the only use case for doing this is like a backup style use case, and you could just have one roll that is able to do it, you could limit everyone from being able to do this, except for the backup user.

B

Yeah, if it was that easy.

C

No yeah, I'm saying like that, would be a new feature we'd have to implement. um I I just wanted to add some some extra background for other people here, so so, while kubernetes allows you to to do this, you know take a f, take a snapshot of a raw block volume and turn it into a file system volume and vice versa.

C

Not all drivers succeed when you try to do this, so it's up to the driver, whether this actually works um and some drivers allow it and some drivers don't allow it in case that wasn't clear. I think kubernetes allows it, but it doesn't require drivers to succeed when you try. It.

D

Ben when you say allow it what's the criteria, can you give an example, because? Because I was asking exactly the same question right, yeah.

C

So so I'll give I'll give a perfect example with netapp. So so netapp our driver supports a few different modes. We have an iscsi mode and we have an nfs mode. If you are in the iscsi mode. uh There is no reason why you couldn't do this. You know create a volume in block mode, then turn it into file system mode or create a volume in file system and turn into block mode.

C

um We could implement that we actually don't implement it today, for you know, because we didn't realize this was a thing that people were even interested in. So if you try it today, it will just fail, but we could implement it for iscsi when it comes to nfs like when you ask for a file system, we give you like an nfs share on our storage controller.

C

We have no way of exposing that as a raw block device, because the raw block devices is inside the storage controller, with the proprietary file system that isn't directly accessible outside the storage device. So like.

A

In that case, when you create a, if you want to create a pvc, you set it to change the mode. In this case, your driver will fail.

C

Yeah yeah so you're welcome to try it but like it will just fail in crate volume right when you it'll say you can't create block file systems on there on an nfs driver. It's like so.

A

Yeah yeah, so that will fail, even if it's like a empty, because that's basically it's not supported anyway, but for some.

C

Well, there are ways to do raw blocks on top of nfs by like creating a very large file and turning it into a loopback device like people, people play.

A

C

Right, we don't do that, but, like one could imagine an implementation that did and.

E

C

That case, you also would not be able to swap back and forth between blocks and file systems right.

A

But then so I was just wondering if that would fail in all I mean that would all draw fail actually will any java actually kind of uh create some something which has the wrong mode, but they actually do not support and then later I'll find out the data is corrupted. Would that even happen? I'm not sure. If that's a possibility, I.

C

I that seems like a place where maybe more test coverage would help uncover issues. I don't think that none of the drivers I know about have that problem like if you ask for us for a driver mode that isn't going to work. We just there. Our regular error checking will tell you that and we don't proceed.

C

E

The issue exists.

C

With empty volumes too, if you ask our nfs driver for a raw block, an empty roblox volume, we'll say sorry, we don't do that. Use iscsi.

D

The just to be clear right, the uh potential security concerns in the issue 477. um She created it's just taking a snapshot of a blocked device and restore into a file system right, not the other way around.

D

If you take a snapshot of a file system and restore into a block device uh since there's no attach, can we claim that that's a safe operation per se.

C

I think so that that does feel safe, but but there's a lot of nuances around that this whole capability of doing conversion, because- and this is what I wanted to get into aside from the security issue- was just the uh the whole concept of csi and fs types.

C

It gets extraordinarily complicated when you have volumes that can be either a file system volume or a raw block volume, because you can do it. There's a csi call called like get volume capabilities where you can ask an existing volume. Do you support this file? Type and kubernetes doesn't currently use it, but it's part of the csi spec and it's unclear what you're supposed to do.

C

If you have a volume which, like used to be an ext4 volume, but it got snapshotted and then turned into a raw block volume and then, if somebody comes and asks like, do you support ext4 like it's hard to know how to answer that question, because it's like well, it's a roadblock volume. So no, you can't mount me a zxt4, but if you take a snapshot of me and turn me back into a file system volume, it's going to be an ext4 volume, not like an xfs volume or some other file system.

C

So it's like you could argue, for it should remember the file system it used to have before it got turned into a roblox volume.

C

You could also argue that it's ridiculous to remember that, and I don't know what what makes more sense um but like, but there's this expectation of the csi layer that drivers know what file system is on the the device or on the volume and when you're converting back and forth from raw to I mean you could imagine, creating a an ext4 file system taking a snapshot turning into a raw block device, wiping all of the data on the volume, while it's a raw block device and like putting an xfs file system on that, then taking a snapshot of that and then creating a new file system volume and then expecting it to work.

C

And it's, and- and I don't know how the csi plug-in would have any idea that it has an xfs file system on it as opposed to an ext4 file system, because it was created by by a user. So we don't even have a way for people to tell the csi plug-in like what they're doing.

D

I I agree, I totally agree with that, but should should we be aware of that? It's still a question because, because the volume is now in the user's hand, right they have full access to the one they can do pretty much whatever they want in the application layer, as long as they are given the corresponding for ridges.

F

C

Well, and in this case that there is no privilege, it's just you know, if you support raw block volumes, all users get access to that.

G

uh Hello, jan, uh I have a question uh related to the security issue. uh You say that if some rope, um the software can mount the the block um device and it would potentially corrupt the kernel. You are talking the the about the kernel of the the the linux on the pod kernel or are on actually on the kubernetes um api server or node, which.

C

Was the whole node.

B

Can be compromised.

A

It could potentially crash the node right so that one of the issues that you brought up.

B

Usually it crashes, sometimes it does something more uh usually.

C

It would depend on the vulnerability. You could imagine a vulnerability that allowed you to gain control with the carefully crafted data in the file system.

B

There are usually a few per year.

A

G

So uh we are, we are preventing that happen by doing what, by not allowing to mount the block at all or what is that.

A

So we are talking about preventing this you if you look at this one right, so we know that that's the important use case for backup vendors. So we're saying we want to uh be able to support that for backups vendors, but not allow that to be um say exploited by the malicious user. So basically we will need to uh introduce some uh admissions control type of thing. So, uh let's.

D

Let's still be clear right, yeah, yes,.

A

Yeah, just a it's just. This is something that we we're. Thinking of uh you know uh how to prevent this, but they're still supporting us for um like backups over that that.

C

A pod security policy switch or something that would control that.

A

So I was so I was actually uh trying to this. Isn't really just something like uh come up really quickly. It's not really a uh thought entirely. uh So I think there are a lot of details. We seem to figure out right, so this is just some possible solution, um so we we need to have some kind of a back rule that allows this and then only like a backup software is a user uh can can allow can be allowed to do this conversion.

A

um You know if, if we are trying to create, if we're trying to create volume from a snapshot with a different mode um and but then that also means because right now in our only snapshot, we don't really know what is the existing body mode right. So that also means that we will need to add an alpha field in the volume snapshot.

A

So this is like when we are taking a snapshot and from from a pvc. So we know what is the mode on that pvc so we'll have that recorded so that and then in the csi spack we can add a new capability.

A

um This is basically this is for. This is that the second point that um ben was talking about so the c is the driver, uh well need to say whether it supports this or not. If it's supported, then we will allow uh this uh the small conversion, and so so. I think this part, I think you'll need to.

C

Hold on just just to quibble with your suggestion for the csi spec, this would be a backwards, incompatible change, because this works today and by.

A

C

A new capability it would stop working.

A

Okay, then we'll just do the other way. You know, so it's a so we will say by default. This is uh we allow conversion and then.

C

And you can declare.

A

It or something like that, I don't know the other.

C

A

C

Yeah there's an an inverted capability need.

A

To support convert by default right go back up backward.

D

Also for existing worm snapshot resources in the in current component system. This water model is not there. That needs to be neat yeah. That's.

A

A that's like a yeah, an extra field that would be alpha all right so like this will be like a new r field. Just for that, one.

D

But whatever you put there for existing ones,.

A

Existing ones will not have that, so this will be only for you. This is an interesting.

C

One to be blank, which means yeah.

A

So it's so it's going to be an option. It's like a pointer right, so.

D

Then then, for exist in order for existing ones to keep the original behavior or do should we keep the original behavior.

A

uh So the ora yeah. So if the original behavior is already there, then we need to keep those right because that's the that's already supported. So this is only for. I think we can. I don't think that we can actually block the old ones uh that will be backward incompatible right.

H

A

Yeah, so by default, this is a this is a just now by default right.

C

Because snapshots are a ga feature.

A

C

They have to continue to work exactly as they work.

A

Right so by default, this is this is now and then um then, if this is a set, then we will uh based on how that is, set and decide what to do right for existing ones. Then we don't, we don't handle those existing ones. This is this is a new new feature, new optional feature.

C

So I I want to propose a slightly different flavor and I haven't thought deeply about this, but like what, if you have a volume, that's like say it's an iscsi volume. The volume itself doesn't care, whether you try to attach to it as a file system or block device. Right like you can do that on a pod bipod basis. You can say: okay, this guy wants to mount it as a file system. This guy wants to manage his block device. The fact that kubernetes has a pvc with exactly one value is kind of limiting.

C

If the underlying device doesn't care, uh then it would be nice if kubernetes knew that, and it could just say okay for this pvc, you can mount it, however, and you wouldn't have to bother going through the rigamarole of snapshotting it and creating a new volume, um because, because the the actual state on the system is either it's either, you can mount it either way or you can't I, I would be very surprised to know of a of one where, like you could mount it only after a conversion but like not as it is right now.

C

A

You are basically proposing to even open this up even more easily, uh because currently we do not allow a pvc uh to be cloned if the mode is uh mismatched right. That's.

C

Right that doesn't make any it doesn't make any sense right. So.

A

You are actually saying that. Okay, let's open this up even more right now,.

C

A

C

Is the volume clone behavior should match the snapshot clone behavior, and if we put any controls around snapshots to prevent things we don't like, then the same should apply to volumes, but the volume should be no more limited than snapshots right. If, if I'm, because all the volume clone is, is a shorthand for take a snapshot and then clone the snapshot.

A

I think, but I think for backup cases snapshot is important. I don't think they always.

C

Know I I'm not either. We definitely need to continue supporting snapshots, but I'm saying there's no reason to limit direct cloning of volumes, given that it's reducible down to a snapshot and a clone.

C

A

I

uh Sorry so I think we started this discussion back when we were talking about the change block tracking api right design, which only considered block volumes right, and then somebody asked how does this work with file system volumes? And then we went down the path of? Can we even you know, create a.

A

That's for the check. There is a change change file list right, that's a little different. So we're saying that that is like a different design. It's not! We can't combine them it's just too so different. We can possibly combine them. It's not like.

H

H

So that will support block storage, uh irrespective of whether the pv mode is a file system or block.

A

C

And and today there are csi plugins that let you convert a file system volume into a block volume and then do the presumably, the the change block tracking could be done. And then, when you want to do a restore, you can convert the raw block volume back to a file system volume. Basically.

I

I'm trying to I'm trying to find out whether there's a legit use case for this conversion and if.

A

There is no, this conversion is yeah this, the conversion that we're talking about if it's a conversion between the block mode and the fastest mode, you're saying this one is needed.

I

No I'm asking, is it needed like what why.

A

Is it this is needed by backup vendors right? So uh maybe, uh uh let's see who who want to explain this back up.

E

Yeah, um I think, there's actually a number of use cases, um but classically like if we take a look at unix, we'll have like the raw device is not something that's accessible to everybody, and I think that's kind of what we're running into here right is that the basic bug is the user, converted a volume to a block volume twiddled with it and then was able to crash the system right, yeah and we're trying to say. Is there a legitimate use case for being able to twiddle with the blocks?

E

I think there is, but I think it's also reasonable to say that's something that we could restrict most users from doing, because we've got scenarios like backup is one uh data recovery. You know um I've. I've worked with data recovery tools that go through and read around the file system and reconstruct file systems. Beyond what ffck does we can see that as a use case in kubernetes?

E

You know you've got a thousand volumes that are trash well, you know spin up a job and run the data recovery software and all of that we probably won't be doing that for five years, but I think we may wind up there.

E

um So I think the use case of getting the raw blocks is is real, but I think we could restrict when you can do it or who can do it.

I

The the way I look at raw block, I think it's like an opaque. You know blob of bytes, you know which well, that's because you're, not a file system, I'm actually a file. I.

E

Mean but but decision I mean you, know, fsdk block based uh backup uh data recovery tools. These do not these. These understand the internal structure, databases. um These will understand the internal structure of the file system.

I

And I think the onus is on the application to take care of how they want to back up that raw block. It's not on the csi, plugins or kubernetes. I think the application should take care of roblox backups well.

E

No, this is typically what we've been doing like from the from the vsphere side right. um We typically do backups at the block level. We take a crash, consistent snapshot and then we slurp out the blocks and that's the fastest path.

I

So, in this case, a vmware will be the application because we're being very understands how these raw blocks are formatted and used, and you know no from.

E

Our point of view: it's a blog right. We provide a virtual disk to kubernetes and attach it to a vm. So, from our point of view, is a storage system? It's a blob, we don't care. What's in it, we don't look inside the backup. Application comes by and looks in from the block.

C

Level yeah, so so, art alone, there's there's multiple ways to implement backup at the application layer at this layer or down at the storage system, layer and they're all legitimate in different situations. I think the problem we're facing is this feature is ga and this works now for csi plugins that allow it. So we have to do something about that.

I

Now, I'm just separating the security problem from the use case. You know like that. We're trying to support for the backup so like I'm just separating the two.

C

And for for one flavor of backup this, this is essential. It sounds like.

I

And then that's what what I'm questioning is it really essential or not that that's my question basically.

C

I mean I, I personally don't care about this flavor of backup, but I don't deny that it's. You know it's right to exist.

H

Yeah I mean so that is the flavor right, where that's enabling um cbt and also, unless we block a backup at block level and then have this capability to get these change blocks.

H

Then it will be a limiting factor if you can't do that at block.

I

I

I

Is there a document that I would.

C

Outline why this is a was it I I wanted to. I wanted xing to finish her proposal. I wanted to see how how far this this goes towards solving it.

A

Oh, this is sagi. Okay, well,.

I

I think this proposal basically is based on the premise that this convergence is needed and essentially.

A

How we can protect against it.

I

A

Just file actually did a presentation. The other day when I talked about cbt right, yeah yeah, so that's there are backup. Vendors definitely uh rely on this. To do this.

C

And after that presentation I went off and I wrote a poc csi driver that supports this, and that's when I discovered how hard it is to get the file system type uh metadata correct when you're going through these conversions back and forth and back and forth- and you don't know what the hell is happening when someone's accessing the raw block version of the p of the volume and that that was my big concern was the csi spec is too vague about what you're supposed to do with fs types. In this situation,.

C

And there is a way to do it right, but it requires making a lot of assumptions.

C

um So so I I like I mean given that this is allowed today and it's ga. We can't just yank it back and say nope, you can't do it anymore, but but we can, I think, find a way to control it better and put access control around it so that it's not abused and that if it's not going to you know for like an nfs based driver where this is never going to work. You can just say that and then have better error handling.

A

Okay, all right, so let me continue. Actually uh I I was actually halfway through. I think uh I think I talked about adding some capability in the css back uh and then I think this is actually the difficult part. um So how do we add this rule to prevent this? um So I'm thinking that we need a validation hook and just to checks to check this rule. um If you know, if we detect that okay, this is a two, this mode does not match.

A

uh This is them, of course, assuming this is in the status, but that's still uh you know, we need a debate and see you know. I don't know if that's the right place for that, but if there's there's a mismatch, then the validation hook will check. If the our back rule is set did not set, then that's just not allowed right and then in the external relation.

A

We already have some code today um that the checks uh for the clone, um if uh at the clone time, if it was a mismatch, it actually will um block it. So we could make something.

C

Validation web hooks, don't know who is making the call right. They have no.

A

Identity yeah, so I think the well so basically, then you just have to uh set the rule. The rule will only be um assigned to the user who are allowed. So that's the same. It's not like it's not like we! um So basically we know those are uh trusted users. So we only assign those people.

C

But the the point of enforcement has to be a place where you have a context that you know which user is trying to execute. The action and validation controllers have no context of who is trying to make the change. They just do the validation of the change.

D

That is a creative field, but that's not reliable in general,.

A

Okay, so you guys have yeah. This is a. This is a uh as I said, this is a just uh I'm not really sure about this party. This is actually the hard part.

D

Like how do you prevent right another right, the uh one model cannot be in status right. How do you do important if it's in status.

A

Well, so we can't have that in spec either. That's a problem right because that's actually when you so when we take a snapshot right. So pvc is already. We already know the pvc, that's the source which we're not supposed to also uh get the the because the volume mode is not required by requested by the users already hold.

C

On you can put it in the spec and you have a validation rule that it that it can only be updated from empty to to having a value, so it can never be changed after it's set and then have a controller that sets it properly.

A

But it doesn't make no I'm saying it doesn't make sense for a volley mode to be involved in snapshot because one snapshot, you can't really request anything. You take a snapshot.

C

As pvc I'm saying, the user would leave it empty and a controller would fill it in.

C

A

Could I mean this: is I'm not saying this? Normally, if you you get something that is already there then status. That's why? But by.

D

Putting english status will, you know, just will not make the static, binding thing working right. The stack of provisioning is not going to work, you don't have the reference.

A

The static provisioning will not work you're saying so because because in the static provisioning case, we don't have the sign to start with.

D

Yeah, it has no snapshot, there's no source pvc. It's all handled.

C

Yeah, that's that's also true yeah. If you're just importing a snapshot that exists, you have to be told by the guy importing it. What is in there.

D

It has to be a field in the construct.

A

Yeah, so that's another, that's that's an another way right, so we can put that in the content yeah. This is just something that I just showed up this morning. Didn't really think through yeah.

E

A

Need somewhere in the in the snapshot somewhere, so that we can compare it.

C

Okay, but but but I I think it's going to be important to to think through the backwards compatibility case, so people are going to have a huge number of legacy, snapshots that have this value, unset and and for those snapshots we have to preserve the old behavior.

C

What we would like to do is maybe provide a recipe to an admin, who's worried about the security threat, say: hey, mr admin. You can go through and update all your snapshots with the correct value. If you know what it is and thereby protect yourself from the security threat uh and then maybe that would be sort of a path to get a legacy system updated to be secure.

D

But we we need to think through all these cases as ben was saying, and luckily we currently do not have any issue at this moment and not non-issue at this moment. So we brought this up. uh We still have time to fix this slowly in the right way.

A

Yeah, I just want to start start this discussion, and uh this is like brainstorming. This is no way I just like before the meeting so that we can get something to start. We can start discussion um yeah. So this part, um I'm not sure this is actually kind of tricky. uh I think this, how to you know uh how to block this. um Do you have any thoughts on this like if we're setting our back rule?

A

uh I think I know the security people are actually suggesting something like that, but you can only have this rule or apply to a resource or a sub resource right, so I'm actually not even sure. uh So that's why I was actually thinking about the new crd or something well.

C

It feels it feels like the equivalent of a pod security policy. That's like a volume security policy is is what's needed because it's.

A

C

It's not really tied to an individual user, it's just tied to the resource, so you know that all the security policy switches for pods exist on the pod security poly policy object, which applies to all users um and then and then I think there might be some mapping of you know how users that can request different policies.

C

But you kind of need something like that where the policy object exists, independent of who's trying to do it because again, if you just have objects in the kubernetes api server, you don't know who made them they're just there and you have to react to them as a controller.

C

And so yeah in none of the places where we want to enforce it. Do we know who is is trying to make the action right? It's always we to see objects and controllers, react to objects.

B

From what I got with the from the discussion with security, guys is that there is a object course subject: access review where anybody can ask api server if particular user can do particular action, and we were suggested to use that to check if uh this guy can restore volume as block.

A

uh Yeah, I think I was really then one thing, I'm still not quite sure is they talked about this. I think they talked about um persistent volume claims review raw. You know that thing. That's is that, like a field of the pvc, I'm not quite understanding what I mean because we're looking at a better rules because they talked about other rules. I think those are all.

D

You need a user right, but basically the fundamental problem ben was for presenting is that who is creating this pvc?

D

B

I didn't check much but from what I saw in documentation of webhooks, they pass user info with username and groups and whatever so do. Are you telling me that this information is wrong? Which information is wrong? I'm sorry! In web hook in admission controller you get a json. Is it json some object wave? How is it called admission review and the webhook sends a response?

B

Send the response and.

A

This is admission.

A

I thought that's what they say that you use.

D

D

In this document, what is that is the pvc which is not the correct thing right? If we go up a little bit.

A

No, I didn't say pvc I was I was talking here here is what I was I'm not so this is the part, I'm not quite sure like how like how does this actually look like.

F

You don't know you don't need a crd.

D

A

Okay, so this is the part that I'm not question, because when you're setting up the rows right, you would need to uh you know it's kind of you just look at this as an example. It doesn't. You know it doesn't matter what you know. uh This can be something else, but this can be pvcs, but I think the problem I'm having is we're not talking about the whole pvc resource, we're not talking they are they talking about this one field of I'm not sure what they mean by this.

D

This is a permission. This is a permission. This is a permission right. This is a permission attached to this precision volume claims. Okay, I think why don't we take uh take all this our background, stuff offline, because we have another item in the agenda.

D

We we can discuss a little bit more on that and come back with the full picture of this whole background.

G

I think so that's a good idea. Yeah, okay,.

A

We could do that.

A

All right, uh okay, so uh yeah. So we have talked about the first item and just I was uh you know. I want to start this discussion um and the second item is the white paper update uh tom? Do you have an update on this.

J

uh Yes, I still have schedule, meaning you get feedback on the motivation, so people can ping me unless we've already chatted for being added to that, I haven't made progress on the cleanup for the uh application examples, um so I'll try to do that by the meeting. Maybe we do friday or something to to go through that, um but you know we can definitely review the motivation section here, asynchronously and before that meeting. If we.

A

A

A

You want to go over this.

A

Or you want to get feedback offline? What is the? What do you? I'm sorry? I didn't get what you.

J

Yeah, I think this is not the right place to go through it. um Okay, read and get feedback make edits whatever we work best.

A

Okay, yeah, uh okay, okay, we can do that. um So uh do we have uh yeah, I think that's probably the main update uh does anyone else have an update on the white paper. I think I forgot to assign this to some people, so I need to. I need to actually uh check. Go over that and see.

C

Okay, shing. I know that you assigned me a section of the white paper about populators that I never did fill in. Oh.

A

Yes, please, uh I still have a.

C

To do for that, I'm sorry! Oh okay!.

A

Yeah, so uh if um yeah I know you're busy with other think so um yeah. If you can uh add white, fill that section it'll be great.

C

I I can report to the group here that we we didn't manage to hit the code freeze deadline for 121 with data populators, so it's going to slip to 122, but the work continues.

C

It just it just it's going to be one more release before we hit.

G

Beta well, I think, for the um the white paper is related to the data protection for the application, backup and restore. I think we have some uh rough graph of us, oh, but we need we need to to sit down and rewrite because right now, it's in like a in a very chaos format.

G

So maybe we should have a next meeting or we to uh discuss about how to who can do what to to clean it up to make it into like a white paper format, and from last time we talked about the application, backup and restore.

G

There is a lot of feedback related to one scenario is restored to original pvc, and I think that seemed to be a little bit too specific to you know as um emc. So maybe we should remove that segment um from the main white paper and and move the I'm.

A

Sorry from what section you're talking about.

G

Oh, that was uh when I present that that that scenario is this was related to restore to original pvc, where.

E

G

Where you do not create the new pvc, but you thought the existing one yeah, and that seemed to be uh too specific.

G

So maybe we should remove that segment out of the main white paper, but put it into like an appendix or something so that for most people would only concern about backup and restore application uh to a new uh pvc, then that that would be like degenerate for everyone and the the one, whoever interested in the restore to original pvc. With this, we can put it in like an appendix segment or in the white paper. Yeah.

A

Okay, yeah, okay, sure.

A

A

Okay, uh okay, so I think that's all on the agenda today. uh Does anyone have anything else you want to talk about? uh If not wait,.

C

What, where are we going to handle that off, that offline discussion about the which.

A

Oh yeah, okay, so I will yeah. I will set up something we need yeah. We need to talk about this yeah. Okay,.

G

Yeah, please look into that that discussion on the on the security uh solution, if I say whether we want to whether we want to mouse open, convert and apply systems a pvc to a block system, pvc is important to me.

A

Okay, yeah yeah. I think it's just we definitely want to you know we want the backup vendor just to be able to support that uh yeah just need to figure out how to uh control this better um yeah.

C

If there was a specific proposal like where we had a poc that kind of sort of worked, that would make it easier to have that discussion.

A

Yeah no, this is this is right now, just I'm just starting uh the discussion, so I just want people to know. uh I know that you wanted to talk about this. That was actually before.

C

A

And now this is yeah. This is a new new thing. Just came up, um yeah.

C

I'm glad you brought it up and it's a good introduction to the subject for a lot of people and uh yeah. I just I just don't know where the, how you're going to express, who is allowed to do this and who isn't allowed to do this, that that seems to be the hard part and then yeah. That's.

A

Where you're gonna.

C

A

C

Yeah but uh but I I I hope, it's solvable.

A

J

Ben was your example. What you're talking about? I mean you said you built kind of a poc for this.

C

Well, I know I I just built given that the regular netapp csi driver doesn't let you do this. I've built one that does just to see what would be involved, um and I encountered all kinds of problems that had nothing to do with the security issue. Just like the basics.

A

Allow conversion.

C

Keeping track of what the file system is so that, when you go through a file system to raw block to back to file system conversion, you you have a sense of what your what commands you're supposed to be running on that volume, because if, if somebody swapped out xff or ext, while it was in raw block mode, then all bets are off. You know you're going to have problems.

C

Yeah, definitely.

A

Are you, is this only net of storage, or are you using your tech, some tests? Sorry to do this uh right? You didn't just already handled by the story system.

C

Like iscsi storage, it's it's like the vmware case. We provide you a bag of bits and we never look inside. uh The only thing that knows what's inside is the csi, plugin and, uh and the csi plugin only knows what kubernetes tells it or what it stores by itself, and so it has to play a lot of games to sort of keep track of. You know, what's going on with this particular iscsi line,.

C

It was surprisingly hard, like I thought I thought I would sit down and do this in an afternoon, and it took like two or three days to get it to work.

A

Yeah, so I'm not, I thought I thought this just like a lot of driver could just just for this without anything additional.

C

You can but like doing it safely, is not so easy because there's all sorts of situations where the the the plugin is expected to know what the file system is like. If you need to do a resize like do you run resize e2fs or do you run xfs growfs like that, requires the plugin to know and if.

A

You're, actually, thinking about you want to change the mode you'll still be able to write to it. That's what you are trying to do.

C

Well, yeah, the pro the problem is when you, when you're, converting back and forth like you, lose control of, what's really in there and and so you need to have various safety checks that you know at mount times say: does it match what I think it is and if not, what am I going to do about that?.

A

Okay, so that's.

I

Why I want to talk about the broader use case of you know, looking yeah better.

A

Ideas case is actually not really changing back and forth. The users.

I

Exactly yeah change.

A

One direction right, I mean yeah. If you want to always do that, then I mean that may be tricky. That's why I was a little surprised. You were saying.

C

That to be clear, the backup use case involves converting one direction for the backup and converting the other direction for doing a restore like you do need both to do better.

A

Right right, so you don't have to go yeah. So I'm just saying that use case, uh it's straightforward. I don't believe we actually need to uh make some any additional changes. That's my understanding.

C

But the the basic problem is, is when you're doing imagine you have imagine you have a backup of a file system volume stored as a bunch of incremental backups of a blocked device right, which is how I imagine this happens underneath when it's time to do a restore the way you would do the restore is you would create a raw block device?

C

You would restore the base and then apply all the incrementals to it or you would do that all at once, but somehow you would start with a raw block volume, you'd schlep, all the you, the the stack of incrementals onto it, to get it to the state it needs to be in then you take a snapshot of that and turn it into a file system volume and hand that to the user and say here's your restored volume.

C

The problem is like at no point did you get to tell kubernetes what file system you thought was on there um so.

E

We've been doing that for a while and from the backup application level right, like valero, keeps track of the pvc and the pv and does the static provisioning and I'm pretty sure it slams in whatever file system. It's supposed to be yes,.

C

It doesn't expect the csi driver to know it, but but but the problem is, is if, if what the csr driver thinks it is, doesn't match what the application thinks it is, you can run into trouble you're going to run into trouble, no matter what, because it's.

E

Pretty easy to put the wrong.

C

E

C

Right right, and so so as a csi plug-in author, like I have to worry about that, I have to handle all the errors that occur when it's wrong, but that can occur any time I mean the user can simply try.

E

Mounting a volume you know they could delete a pv and then.

C

E

Right so you've got the same problem. It's.

C

Not not backup specific, I'm just saying like for doing things like resizing a volume you have to know what's in there or you have to at least have an idea. What you think is in there, uh because there is no.

A

C

A

Size, that's a different use case now right talking about the price.

E

Yeah by the time you get to resize you're, going to have converted it back to being a file right, a file system, so.

K

A

um You should already you you are you need to back up a that, your metadata in your pvc anyway, so all everything in your pvc ammo should be stored. So you actually know what you have to restore back with right. So that's something that you should not have to guess.

C

Right, the the problem is, is there's no way for the application. That knows to tell the csi plug-in that wants to know. There's no communication channel there. So so, regardless of how smart the backup application is, you can't inform the csi plugin they're like hey this raw block volume that I'm creating I'm sticking an ext4 file system in it. Just for your information, like the the csi plugin, has to figure that out empirically, and that's it's a little gross that that was my big complaint.

C

J

Work end-to-end, but you're just.

C

Pointing out the kind of robustness issue right then yeah yeah yeah it it absolutely can work. If you don't do anything hokey, but the problem is no one's preventing you from doing anything hokey, and so, if you're defensively programming, your csi plugin, you have to you, have to think like a like someone, who's trying to fuzz your api and doing all of the bad things.

A

A

Because it's so you really just you, you change it when you change mode, it's really really at that time. You're, not really. I mean I'm talking about the backup case right, so you're not really making any changes. So actually I'm not.

C

It's the responsibility.

A

I think what changes you have to make in the css driver set like that's. I guess that's what I my question is because I'm not aware that we have to do any anything in the csa driver to handle this, so maybe other drivers do so that maybe that's my the part that I didn't get.

C

Yeah it's at restored time. You have to write to the raw block volume and then convert it to a file system volume and if nobody lies to you about what the file system is you're, you're. Okay. But if someone lies to you about the file system, you will get weird errors. Yeah.

A

But that is the that's the responsibility of the upper level, the backup software right. It needs to know it needs to know what that is because otherwise I mean there is no way for snapshot to to have all the metadata that is stored in the original pvc anyway.

A

So even if right now, I know we're talking about, we need to save this one mode, but there are still tons of other information in the pvc that, if you don't save it ahead of time, then when you uh create and rolling back it's not.

C

The csi plugin, the csi plugin, has metadata and it is.

A

A plug-in is not not going to uh like, uh if you don't have the uh automatic data and you don't really know how to create your like your pvc ammo when you are when you're creating a new pvc from the source right, if you don't have all those parameters, that's like user side. It's not like you're going to read from the seaside drivers together, that's not the case.

A

So um there is.

C

The thing that gives me fits is that there's there are capabilities checks in the csi driver that are supposed to fail, if, if the capabilities that are requested, don't match the actual capabilities and you're allowed to when you create a volume say like I expect this to be xfs, and if it's not you're supposed to fail um and what you're saying is like well, you just have to trust the application that says yeah. It's xfs trust me right.

E

You can you can identify, you can fingerprint the file system as you try to mount it or before you mount it to say. Is this an xfs.

C

Volume right, but the create volume is not in the data plane. It's in the control plane, so crate volume has to go only on the metadata that it has it can't. It can't actually read the bits and see what it's dealing with, because create volume is running in a controller somewhere. Well.

E

At some point, you actually.

C

Have to initialize like an xfs file system right right, like the node. Does that and so that's why I'm saying like the node has to know it. It needs to know to do its job and then it needs to store. There needs to be metadata tracking somewhere what it, what the system thinks, what the csi plug-in thinks it is and then to implement.

E

C

The capabilities checks that csi says you're supposed to implement. You have to store that metadata in a place where the controller plug-in can always access it for both snapshots and volumes.

C

Otherwise, otherwise you can't implement those capabilities, checks correctly and you have to lie and as long as as long as as long as as long as no one's doing anything, weird those lies never get caught, but the moment someone does something strange.

C

You could end up in a bad situation. Well,.

E

I think it would be worthwhile to separate out, like dumb things, users do that cause them headaches that we would like to make better and dumb things. Users do or malicious things. Users do that actually cause security. Holes like crashing the worker nodes, or you know elevating privileges or things like that.

A

Yeah, okay, time check, okay and yeah and the top of the hour uh yeah so uh yeah. Well, we will need to think about this more and uh come up with a solution and we'll talk discuss about this again. um All right, then I think that's it for today thanks everyone.

J

Bye. Thank you.