Kubernetes Data Protection Working Group, 21 Sep 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes Data Protection WG Bi-Weekly Meeting 2022-09-21

Description

Kubernetes Data Protection WG Bi-Weekly Meeting - 21 September 2022

Meeting Notes/Agenda: -

Find out more about the Data Protection WG here: https://github.com/kubernetes/community/tree/master/wg-data-protection

Moderator: Xing Yang (VMware)

A

Hello, everyone today is September 21st 2022. This is the kubernetes data production group meeting. Today we will do some cap updates and, of course, if you have any open issues, we could also go over those at the end.

A

uh With that Yvonne, you want to go first, yeah.

B

I can uh provide.

A

It, maybe let me stop sharing it uh if you want to share anything, let me okay.

B

Yeah I can quickly just uh share my screen. um Can you give me permission to do that.

A

Yeah, yes, great.

B

um Let me just yeah um hi everybody, um just a quick update on the CBT um cap and this past week I submitted a PR to the CSI spec on repo uh to get the uh RPC specification review um so far, I got a some feedback from James diffology and also from Humble as well uh seems like um the I guess so far like the one feedback around it's around, like um you know whether so like basically the RPC specs like they are again just um identical to what we have been talking about for the past couple of weeks and um the main one main feedback from coming back from James is around like.

B

Do we really need, like this mode property, um just a quick reminder, a refresher like initially we thought of using it to distinguish between block and file. I know that during the last working group meeting Frank also touched a little bit about it. uh There seems to be some um I guess like uh work in progress in the clarity and definition of what like block file mean. Are we talking about Storage level I'll, be talking about something, that's um more in line with what the persistent volume claim or snapshot snapshot.

B

Things notice, um I think what James is saying is that you probably donated because uh they seem to think that, like um you know, the storage provider should be able to figure it out like even like uh the the I the handles and the IDS of the snapshots.

B

um So yeah I told him that I'll bring it back to the group to see like um you know, if uh we all have any thoughts and inputs on that, whether it's something that we can remove now and add it later, if needed or I guess.

B

Basically, it's just I, don't have enough, like data points around whether like All Storage providers will be able to tell just by looking at the snapshot, ID or snapshot handles, whether like uh we can infer that okay I know, this is a block, snapshot or I know that this is a file snapshot. You know.

A

So, uh let's see Frank's not here, I I, remember last time when he brought this up. uh We don't really know who can support us yet right, even for Isola I forgot what he said. He said yeah.

B

He uh I think he gave an example.

A

But does iPhone actually support this I thought it was a question of who can actually support this, but uh do you remember because.

A

C

Can I can remember um okay, Juan said uh the uh EMC right, the.

A

Iceland, that's what he.

C

A

Really he does he, but I thought I also thought there was something we also talked about like who can support this. So uh I I don't know resist, but I forgot here.

C

On top, on top might be able to support this, but.

A

Adela is here, I thought the they.

B

So on that you can support the file I.

A

Belong but but I don't know if they are interested in this API because I remember they now.

D

Otherwise, here yes, um so the question is: if you want to make a distinction between no.

A

C

About the file.

A

Where the the weather.

C

A

List do you actually have support for that? Remember, your API is not really uh open or something you don't want to support this.

D

Or something we do have a permission to see if we do have a proprietary, API called snapdiff. That would tell you.

A

um You know, but you say I remember you guys were saying you don't really need this, you just every time you take a snapshot. You automatically determine that so you're not going to implement this one. Is that that's the.

C

It's not it's a it's.

D

A it's, not a public API. Yes, a private API. That's only exposed to select Partners.

A

So you are not going to do that in your csaturated, because you have to see how Jerry is actually open source yeah, okay, I see.

B

Yeah I mean like yeah yeah. The Su all mentioned the example. That fungus was. um You know from like um one one, FS I think it's more like a NFS type of um solution um and then by this, like um James um James's point is like regardless, like uh he he's wondering like. um Oh, he thinks that, like um the storage provider should be able to insert and detect that without using users having to, and then we kind of talk about this too right like if we put this here, who's gonna provide the value like.

A

B

Know does the backups.

A

Sorry, we need a little bit more yeah, concrete cases to see who can actually support this. Otherwise we add us here: okay, you don't really know how to use this.

B

Yeah I'm happy happy to remove it just to avoid, like confusions.

C

um Because I guess it's.

B

Also, over a bit overloaded, like I, mean people see more getting tend to think of it from like the PVC and the snapshot like how it is currently being used so happy to remove it um and then, like you know, if we feel like it's truly a need for user to provide its value, then we can add it later and later.

A

Right because I was thinking even if we add this one I think it will be at most is two mode anyway, just block or file I, don't think we could ever have a third mode. So.

D

You know one thing I'd like to make clear is that even for NetApp, even though you know we are known for our fall Services offerings, we do track their deep blocks. You know, so it's not like that. As far as you know, knowing what changed we tracked, their D blocks, not the dirty files.

D

A

So you're saying your track had blocked the block. They were not at file level, so.

D

A

D

Snap, you know our snapshots or the way we do snap mirror, which is our replication engine yeah, that tracks dirty blocks. Okay,.

A

Not the files, okay got it.

D

A

You don't really need, even so, even if you open your API, you probably still don't need this thing. Okay, all right see, I.

C

D

Think it all depends on the use case. You know, like you know, like the file level tracking is useful for scenarios where you want to do backup and restore on individual files.

C

D

So say you know, restore this file to version X or Version Y.

C

D

So it's a little different use case than what we've discussed so far with CBT right, because here with CBT we're mainly talking about. Oh, these are the dirty blocks and you know, let's just back these up right for more.

A

Efficient I, wonder yeah, maybe we just uh so.

D

You know, like I, think it's a different use case. You know if you want to have Because deep down I. Imagine most of these storage vendors I mean I, can't speak for all, but I think most of them. They're snapshot, implementation or the replication engine actually is at the Block Level, not at a file level, but there may be some exceptions but I. Think many of.

A

Them are yeah, not a file. Our snapshots, yeah.

D

They are, but you know like, as far as you know, how they keep track of drd data. You know, let's say if they have to have like a log structure file system right. A log structure file system keeps track of dirty blocks.

B

Right: okay, Dave- um is this similar to how we do it for casting.

C

Like um we do like we don't track at the moment we just go searching so um there's some stuff inside of uh VMware that did file uh file based tracking between snapshots, I'm, not sure if that's public or not, um you can imagine doing it, but it really like you know it is going to be a a consequence of how you implemented the snapshotting, so yeah I I've built file systems where it would track that. But many don't so it's probably more of a future than anything at this point.

C

I don't know if anybody actually has something that would let us do this. Oh.

B

Yeah I think nugget I kind of it's starting to come back to me a little bit as well I think part of the influence there was, um you know, I think came out of the that brief conversation around. How sometimes like um you know, providers may like put a lot of features into one CSI driver. You know so I was thinking I think like um in first so, for example, with AWS right they have an EBS CSI driver and then they have an EFS CSI driver.

B

So with those kind of drivers like we don't need those.

A

They're such huge hours, those are just separate drivers right.

B

So yeah completely separate right, but sometimes I, don't I. Think, like um the initial discussion way back, was it what if they put everything into one CSI driver, then.

A

Oh netapp's travel is just one driver, I suppose all.

B

I think that was kind of what influenced like this um this property again, it sounds like we can just remove it for now. If anything.

A

Yeah I'm thinking, if we, if there are any I, mean if there are only like two modes potentially anyway, we could in the future. We could just say by default, it's a Blog and then, if you, then we add one flag for profile. Something like that right.

C

A

We want to add that in the future we can still just use the message. I was just I was just thinking out loud and like, but we don't have uh enough use case for this one. It's not clear how to do this, so I yeah. This is already pretty complicated, I'll, say just yeah. Let's start with something simple, yeah.

C

A

Also, this is still you see. This is the alpha right, so we were in CSS back. This is also aha, so.

C

A

With something concrete so.

B

Okay, yeah that sounds good, okay now and remove that um and then two other items um from the review feedback so far, I think I think, like the first one being like again, the the naming um so uh like at the CRT level.

B

um During our like last, like working group meetings, um we talked about like when a backup software or users um create or submit like um the volume snapshot, Delta CR uh we are doing gonna use, like um you know, a create slash post um request just so that we can package up the um you know that the the inputs into like one single, like payload um at the RPC level, like um after looking through like the existing specs and all the rpcs I, think like um it makes sense if we use like a list from type again.

B

This is just, for example, naming and semantics um implementation, one. It doesn't make a difference. It's just like you know, instead of say, like a great snapshot Delta, you know if I follow, like the existing pattern and style in the spec I think it makes sense to call it a list snapshot data. But just because, like you know, for example, all these fields, like Max entries and starting token they're, all existing like um fields that are like tied to like a list, RPC call.

A

B

A

So we don't really have, can you uh uh so really you just we just have this one request right. Just a list is that right, yeah.

B

I guess so yeah but like um if we um let me just uh do I have to yeah I do have to cap here, but again, I want to make sure um we are all on the same page on this to design detail. So we talk about like how we're not gonna like we try to follow the this. Our back subject access some review type of mechanism where we don't when we do a create, do something to create, but we don't actually um persist. This volume snapshot Delta resource another.

A

Reason is what we, if we don't do a post, we cannot just do a get. Is that that's a problem, because we need.

B

A

Have a full idea difference or something right remember with this discussion.

B

Yeah, so we talk about the two main reasons being like um you know. If we, if we submit, if we use a get here instead of a post or like create versus lists or get like there isn't like a way to I mean there isn't like a clean way to pass in all these parameters right right.

C

B

Append them as query parameters to the URL, and then it's like the the spec grow. Then you have to keep adding more parameter, query parameters, URL and then also at the um the code level like it requires us to use more like the lower level rest um API on the clients.

C

B

You know instead of saying you know like controller controller.create, which is the controller.res start, something something and then propose kind of thing. So um I think there's some like um um you know just some preferences to not go down to like the risk. I think so. Those are the two main reasons: okay,.

A

Yeah so so I do see yeah, maybe uh yeah. We need to uh ask other reviewers to take a look. It's definitely the naming. Definitely.

A

Mismatch here right so suppose here and then it's a get there but I do see over there. That looks like get make more sense.

B

A

Because I don't want you yeah yeah, the.

B

Main motivation here is I, don't want to say, like I, don't want it like: hey, hey, create snapshot, Delta request, but inside create, like we have all these existing Fields. You know Max entry starting token get used everywhere, but only inside lists.

A

um Right right because otherwise, normally like, if you have a crate, you would also have a list, but in that case then they are kind of identical because we're not really uh generating new resources. In this case.

C

B

A

Snapshots are already created right, so you're just getting the divs yeah.

C

B

C

B

So I think that's the one, the one main thing around the naming and semantics, but like again like when it comes to the code level. At the end of the day, it's still a grpc call, um but I just want to be more consistent with the existing CSI spec. um So there's a second or the last item. Feedback was like I. Think humble humble is assemble on the call.

A

He's not on the.

B

Call okay: he just mentioned about adding like um a secret reference, because.

A

They uh so some drivers uh need to pass this when they retrieve anything from the storage system yeah. So you should add that just I, don't ever since I call, should have that now. Okay,.

B

A

B

C

A

C

B

C

Think you're up.

B

A

You also need to have that in your um you know. Kubernetes API should also need to have like a secret reference or something. Oh.

B

It's okay yeah now, so this secret would be. What like? Is it like? This.

C

B

Okay, so if I'm on AWS, it would be AWS some secrets in there right.

A

B

A

Like that, yeah.

B

Okay, okay, sure, okay, because initially, like uh I, was under the assumption that, like CSI driver, would already know like all this credential, but.

A

I think well, it depends on houses that are handles there. Some drivers handling this list through the secrets, like our driver. We have it through some.

A

We all also saw that in a secret, but we're just you know getting a different a little bit differently, but if you want the one to pass it through, the query volume call then we'll create a snapshot. Call. Then then you need to do it this way. Yeah, you add those in the.

C

Okay, okay, yeah.

A

B

um Otherwise, like um the rest are pretty, um you know familiar, um you know like getting the response, so we talk about uh responses like the responses like you know, the volume size and the next token, if there's more, um this is where the pagination comes in uh and then like, uh and we have a repeated lists of change blocks between the change boxes offset this the size of the block uh and optionally.

B

There is this thing called token, um because one of the again one of the feedback was like: um how do we make it? So if, for someone who is nothing for users who is not familiar with, um CBT is like they would look at all this response and they'll be like okay? What can I do with all this? um You know metadata, there's no way to actually get the actual data block, so this token is I, guess uh inspired directly by the AWS uh EBS direct API.

B

So it's like it's like the CBT responds like AWS gives something returns, something called a token and then the token is something that can be used to fetch the actual data block using another API endpoint um in the tokens ephemeral, and it left for, like only like certain um number of seconds, I think um and then I, don't like not all storage provider implements this um okay, so this is added here as an optional feel like for for storage provider. That doesn't provide token based. Retrieval then like it would be.

B

We make it clear in the spec that it's up to them to decide how they want to make use of all this I think somewhere in here, I added um or maybe was it inside the okay. It's like the spec dot MD. Where is it? Okay?

B

um Just this little blurb here um saying that you know the controller plugin must not attempt to retrieve the raw data blocks, because James mentioned to me that this is that's outside of the scope of CSI um and then it's up to the storage provider to decide what to do with the metadata if they support. Like a token retrieval, a token based retrieval method, then, is there inside the response?

B

um If that makes sense,.

C

B

Maybe like just take a closer look at um this, this PR and you know at comments there and then I can address their it there, but otherwise the rest is pretty um familiar. Hopefully like it's based on what we have been talking about for the past um few weeks and months. Any questions about the CSI spec software.

B

Okay, uh if not yeah, I think the one last thing so the uh I saw like um the issue has been obtained for 1.26. um There are some updates that I need to make here. I, don't know if some funds online, but uh so I can't so right now, like I'm he's the owner of the issues, so I can really edit it Shane. Would you be able to somehow assign me or give me uh permission to change something.

A

I can change I, don't think I can give you permission. Maybe I can change it. This is investment. I would see. So we just need to change the alpha Target to 1.26 right yeah.

B

There's a list of things we need to do.

A

Yeah I think well, you can assign this to yourself a sign that you can do right. You can okay, this yeah yeah, let's see.

B

Okay, I can try to sign it and then see if let me edit the description or.

A

I can ping no I, don't think so it just oh. It.

B

A

That's not enough to yeah okay.

C

A

Me see, let me see if I can, what, if I think, I've got yeah. Okay, okay, let me change that.

C

A

A

All right, yeah I'll, add your name here. So, okay.

B

C

B

Yeah otherwise,.

A

Can you take a look? Is there anything else we need to update? Oh, we probably should add the CSS backlink here as well sure.

B

I can uh oh um yeah yeah, okay,.

C

B

So yeah and then um yeah otherwise also like I, think is it next week is the um the production Readiness review, freeze um so I again already paying like um John um Limerick and he said he's going to take a look at it. um He was. He helped us to review it last time and he was the one who asked us to revisit. Reevaluate the aggregated, API, server approach, okay,.

A

Yeah then at least yeah I think at least to get his feedback on the API server part, um the aggregated apsr department, so that yeah yeah.

C

B

I think otherwise the finger cross, hopefully like we can make. We have made it before, like uh the prr freeze happen and then get it into the enhancement pipeline before the enhancement freeze happened um so yeah. That's it from me that does anyone have any questions. Anything in particular I'd like to go through.

C

Yeah uh this is punkers I, don't know, I mean it's not a question, but uh just trying to get the information about uh how uh how this can be used uh end to end uh with both kind of problems lock, as well as file I, think block it is. It is fine, uh we'll get this change blocks and on top of that, those can be used, but for file mode volumes yeah. So.

B

Yeah um file would be another cap like this. uh This cap is is about block um I, think like fundamentally like. um Basically, we try to make like the aggregated, API server as simple as possible, like it's just like you give it a volume, snapshot base and Target IDs. You just do something like simple validation to make sure those resources exist, pull out like to see just find out where the CSI driver is, um and then, if there's any like um input parameters in it, it would perform this.

B

um The volume snapshot class and after that, you just package the whole payload and then send it over to the CSI driver like as shown in this diagram here and let the CSI driver do is magic, so whether it is um block or um file finger across, hopefully, like you know, like the aggregated API server can continue to remain as some as simple as possible and let the CSI driver do all the heavy lifting um but yeah again this at this point like there might be some I think it's like I'm just grossly over the high level oversimplify, and we have a file change file cap coming up soon.

B

C

Sure got it so that that the file mode would be handled as part of that basically yeah.

B

Yeah, we don't yeah, hopefully we you know what we do here. I think, there's no reason why it cannot be reusable file and let the CSI driver do all the again heavy lifting um so yeah cool um yeah. If um you know folks, have um have any questions or things that they want to go through, um feel free to ping me on slack um that we.

C

Can chat thanks.

C

A

uh Okay, so let me go the second cap that we're targeting uh the warning group type, also we're targeting Alpha in 1.26 release so yeah. So please uh help review it. This uh this card actually has been there for quite some time, but I did update that one for for 1.26. So that's all the comments are addressed. Just a reading to be reviewed, um let.

C

Me see instrumental.

A

So there's also a corresponding uh PR in this in csi's back repo. So this is this.

C

um Naturally, opens.

A

So in the scissors back, basically, you just introduced quite a few uh offices for create the lead volume and create delete. One group, Walling group snapshot an awesome result. There is a method for modify voting group, um there's also list and uh get methods, although I'm not sure. If we'll need both list and the get I mean I, think I will only need to call one.

C

A

Them so get is sufficient, but it's just for completeness I also added this here, but I'll see, maybe those can be removed um and also for every method. I also added the secrets known as one main concern also comes from gyms.

A

uh He basically I think he's concerned. Basically his question was like well. Why do we even need to have a group as the first class uh Concept in in CSI? We thought? Okay, can we just add group name as attribute and volume so, um but the thing is: uh if we just. If we only do that, then it will be not possible for us to um do any operations at the group level, because you know if it's just everything is just a attribute on awarding.

A

Then then we have to either have some new method allow us to pass a list of volumes. Otherwise we just operate on. One volume assume, that's like all the volumes are related or something resonance nowhere for us to to anything that really at a group level.

A

If we do not have this concept and also if we only support group snapshot, maybe possible, we do not need a warning group, but we still need to go to group snapshot, but we also want to have this concept to be generic enough, so that we could extend it to support a replication group in the future, and so for that reason voting group results are needed. So that's my response. I have not heard from him after that, but that's a.

A

It works for them.

A

Any questions from anyone on this.

B

Yeah I'm very minor questions around the naming convention. I think we talk about it too, in one of the comments. Do I need to prefix like those message name of the controller get.

A

uh I think yours, yours is lesser. I think this is okay.

B

A

Have you don't have a cat? Do you no.

B

I, don't have a get.

A

B

A

I think it's only the cat one uh had that: okay.

B

Okay, so in this case yeah, so it's like there's some a line, one once 1160. does that also match. So it's the same one as the one in the RPC, because in the RPC, like you know, there's no, it's a controller. Okay, so say: okay, um okay, because inside the RPC that's the the prefix is missing. I just want to make sure that I.

A

Don't know it is Mission because otherwise it should not be should not pass the CIA okay. Let's, let's check, because this is a you know, because I didn't make before I submit this right. Wow.

C

A

Controller controller that uh which one control get the volume.

B

If we go back to the top of the page.

A

So it's a get awarding group is the.

B

One that yeah I get volume I want to make sure to see like right here. It says: get volume I want to make sure performance up here. I'm following.

A

I'm doing the right.

B

A

No, actually, that's that's actually right why this one says it was strange: okay,.

B

So the controller prefix is only for the get from calls.

A

So, okay, let me see.

A

Oh I see I, think the I think when we do the um when we do make it's only the ones in the in the you know in the code block. That's why this one okay, yeah.

D

A

It should be the same thing so so fix this one. uh Let's see so it's not this one doesn't have it, but then we this one, has it yeah, okay, right so yeah! Okay, let me check good morning.

A

Yeah you see this yeah, so the controller guide volume is like this right, so yeah, okay, so I I'll fix the one on at top. I think that one is only in the in the dark. It's not in the yeah.

C

A

It's not even cold yeah, okay,.

B

Yeah, so super minor I just want to make sure that I follow the same convention. Yeah.

A

A

uh Okay, anyone else do you have any comments. Yeah, please help review uh so either you know the the cap or the CSS bad PR or both if you can help review those that will be great.

C

A

Oh and and another change that I just made yesterday was that um uh initially there was a there was a proposed change in the PVC spec. So when we add a PVC to a one group, uh there is a new field for the group name. So when you create a new PVC, you add a group name and that will be added to the group. So now I changed that one to a label. This way, at least in the initial proposal. We don't need any entry, API changes so.

C

Yeah, so that's just a it's just for now right, oh did.

A

I, okay, I'm, not sure if I forgot to submit this, because I forgot to submit my comments.

A

Okay, okay, do you have anything you want to talk about, or is this uh all.

C

Good, no good, fantastic, okay.

A

Laura then, even if there's a nothing else, then that's it for today thanks everyone.

C

A

C