Kubernetes Data Protection Working Group, 27 Jul 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes Data Protection WG Bi-Weekly Meeting 2022-07-27

Description

Meeting of Kubernetes Data Protection WG Bi-Weekly - 27 July 2022

Meeting Notes/Agenda: -

Find out more about the DP WG here: https://github.com/kubernetes/community/tree/master/wg-data-protection

Moderator: Xing Yang (VMware)

A

Hello, everyone today is july, 27th 2022. This is the kubernetes data protection, ohio group meeting.

A

uh Today we have a few things in the agenda: fun well given update um and on the cbt, and I will talk about the volume group cap, give an update, and after that, it's going to be open issues. What is the oh? So? Finally, you put this one under open issues: yeah.

B

A

uh Okay, all right, okay, sure, so we can get started.

A

Let me go to fonts, uh slides yeah, go ahead, yeah.

B

So uh I just want to give a quick update of the progress that we made in our design discussion. uh We will wait for ivan to come back in the next he's in the vacation right now. He will. He will talk more in detail about the the all the issues with the architecture and how we resolve the problem and so on and so forth, but basically the new design now is that we have the cbt aggregated server.

B

This is um remember back then like in about maybe one years ago morning, one year ago, I proposed our a proxy service for all the different snapshots, so this this essentially, is that so the cbt accurate api is like serving as a proxy.

B

This will be one server per cluster and when it receive a differential snapshot request, it will directing it to the csi drive that handle that particular volume snapshot, delta and- and that is the the like essential of the architecture right for each of the csi driver who support this differential snapshot or volume snapshot delta.

B

It will need to register you by create a driver, discovery cr and and provide all the information needed to contact to to communicate with it. You're going to see at the bottom of the orange box, I mean of the pink box on the right. There will be like a client, config and all that stuff. So these are information that need for the cbt aggravated api server to contact the csi driver uh to get the differential snapshot to get the volume snapshot delta detail.

B

So in a sense, so that is a the architecture. uh When uh ivan come back in the next, um at least we will talk in detail, but uh you can click on page three. I can give you like a a brief discussion. Why we still thinking this. This architecture still needs some change.

B

uh One of the thing is that um right now we're still debating between uh our our cbg. This uh team is whether we still need a volume snapshot, delta, cr right. The the green box, on the left hand side whether we need that at all uh to because when we create that, when the the backup software creating that it start the conversation right, but did we and then that cr being stored in the scd.

B

uh As a matter of fact, I'm thinking that you know we might not need that at all. Instead of putting that into a cr, we can simply put that entire content inside the uh get request. As a body on the get request on that. On that the arrow on the top of the screen, you can see that so we still need some debating. We need to need some uh discussion, uh but uh uh but that is the the the update of what uh the project right now. uh Okay,.

A

File, so you are basically saying because this is not something user heavy control anyways, it doesn't really add too much value. By presenting this as a as this type of kubernetes, api object. Is that what it is so basically yeah.

B

A

Because the backup software knows what it needs it needs to. Basically, this is like more like an internal to backup software. It will just construct this uh the thing okay and.

C

Then I have a quick. I have a quick question that cr is. The crd is actually the interface for pointing all the differences if we are not providing that basically end users or backup software, they need to switch from the kim model. To rest api is that a desire.

B

Oh, I I like I I I said we still debating it, but.

A

If you're using aggregate api server, you probably have to do this right. That's a part of that right.

A

Otherwise, it's a kubernetes, you use kubernetes way right. Otherwise um this is a company.

D

Provides rest apis.

A

I'm sorry cool.

D

This is dave. um Kubernetes provides these rest apis.

D

I think what we want to do is you want to make sure that they can be accessed through, for example, like client go, so it looks like I mean, that's normally the way you access things right is client. Go forms up a rest request.

D

Formatted like this. um I think the interesting thing we could do is like I want. I would. I would really recommend that we keep the rest api as the kubernetes api, so we don't add things that client go can't generate easily, but.

C

D

C

I agree, I think, because.

D

Yeah, you don't want to write a straight rest. Client, you want to be able to say no. I can use this with cube cuddle, because cube cuddle will generate the correct rest. It generates a particular style of rest call. It knows how to form it. You know, for example, like with a field selector or something.

C

And that's one thing: the other thing is that how do they keep track of things right? Well,.

D

Anyways the call.

C

If the, if it's the pure rest call by passing in snapshot, handle there's no easy way for the backup software to see. Oh, I have done this right. If it's the cr in the cluster, they can always query. Oh, I haven't done this before.

B

I don't know if we need to keep track of that.

D

Yeah, so the the setup so like the change block delta resource itself isn't doing a whole lot unless oops, I'm sorry, I've, gotta I'll, be back a minute. I've got a repairman here.

E

Can you ask the question? I'm sorry, I've missed this thread of conversations for a few min for every few weeks, but I thought the main problem with this crd design was that these crs can be potentially very large, there's a lot of churn for them. So it's not practical to store them in a cd, because.

B

E

We went down the path of using the rest api.

C

Arden, yes, that that's fine, we're not looking to use the str to all the change blocks. All need the request right. Only the the two snapshot handles needs to pass down to the csi driver.

E

C

When I look at this.

E

Screen rectangle, I see you know offset, I see very block specific information there or maybe no that's or is that like? What are we going to have like a different look, for example, in the status clause, we have chain block deltas and then there's offset zero offset one. Are we going to have like a section for each offset.

C

My understanding is that in the status that cr will carry the end point, the backup system needs to hit to get all these change box out. It will not contain all the actual strange blocks unless I'm wrong.

A

C

B

A

B

D

Well, so, there's I think, okay, so there's a basic misunderstanding that is going around. I think, and that is what's the relationship between kubernetes resources and rest calls and when you access, kubernetes resources from the api server you make. A rest call is that.

E

No, I think, that's what understood the question.

D

Well, no, it's not because people keep having these questions, so this rest call when we do the when we do the aggregated api. What happens? Is you retrieve resources from the api server using a rest? Call that then forwards it to the aggregated api server to satisfy that.

C

All right, zero problem with that, I I think from.

D

The user's point of view there's a resource that they're going to retrieve and they can do this. For example, cube cuddle, get uh volume, snapshot delta, but they're going to have to do like a field selector or something so that it, the right stuff, gets passed through to the back end.

D

So you can't just list all change block deltas, because that would give you every every delta between every snapshot in the system, and that makes no sense. So that's going to have to return like nothing.

C

Okay, maybe I misunderstood, because I'm I'm not able to look at the screen, but what I was asking is when we trigger the change block. Call this deal would be a cr right.

D

It's a kubernetes resource, so it's not a customer. So that's where people get get pedantic because it's not a custom resource. There's no customer resource definition involved.

C

D

C

You request when you the first time.

D

A

Look at the screen right, so, okay, it's a resource.

D

Okay, so if you say custom resource people go, oh it's not custom resource. It's a resource, though it is a kubernetes.

A

Issue this is not defined. Can you look at this? Can you see the screen yeah? Okay, because I think changing can now see the screen. uh Maybe you can explain this one, I'm also again confused with this one. So this one is the crd right this one. Whatever cn here is a crd.

D

A

D

I'm going to be pedantic for a moment, and just so we we put this to bed. So there is no custom resource definition. Custom resource definitions set up things in the api servers so that the api server understands it. There is, however, a kubernetes resource. It's just not defined through the crd mechanism, so the box in green, the volume snapshot delta, is a kubernetes resource that you would retrieve it's just not a quote-unquote custom resource.

D

You can call it a custom resource, but if you start talking to people- and you say, I'm doing custom resources, I go well wait. Crds and aggregate api servers. Those are different. They act the same from the client's point of view, except for the lack of a crd.

D

So you retrieve kubernetes resources using the the naming convention and they come back as looking like kubernetes resources.

D

So it's just the.

A

Customer you're doing a cube cutter. Are you getting this uh when you when, if you are, do you using a cube, color command, you construct the same thing you construct the same thing.

D

If so, the way I saw it was, if you the cube cuddle, you would select, for example, like name space, your namespace, slash, change, block deltas, and then you would give a field selector and the field selector would have enough information in it that the aggregated api server can talk to the right, csi driver.

D

One way you could do that is, you could say, is you could define fields in the volume snapshot delta, such as uh bass, bass and target, and so you could give a field selector where you say, bass, equals snapshot x, target equals snapshot y.

A

uh Okay, so you're saying basically, this is when we do a cube, color it's. This is not what you construct with the kind spec.

D

So you're going to get that back you're going to get that change.

A

This is only in the response. This is not in the so.

D

So there's different ways to do it right, so there's there's the concept of doing it all in like volume snapshot delta, in which case um I think you might.

D

This would be where it comes back in the status. So you would create volume, snapshot, delta, you'd, post, that you know or cuddle it and you'd create it, and then you could check the status you could retrieve the status um fong was saying that the information here in volume snapshot delta is not something you actually need ahead of time, so creating the volume snapshot delta to kind of prime the driver doesn't actually help you out very much because most of the the setups we've seen you can just say hey.

D

I want between this target between the base and the target, and you know offset into your list this much.

B

Yes, that's not exactly what I meant. We don't have to create a resource in advance. We just do it as a part of the.

A

uh Okay, so um I guess a probably it's just I'm not still not left medium is how aggregate absorber works. uh I thought so. What I see missing from this diagram is: where is the api server.

D

It should be in between backup software and cbt yeah.

A

You should actually add that phone. Can you add that? Because that's part of that field, it's missing so um so.

D

So so that's if we want to get a kubernetes standard api, which I believe is what we want to have.

A

Applied standard, that's what we are talking about right, aggregate.

B

Yeah yeah, that's that's! That's what it is.

D

Okay- and so I don't want to talk too much about constructing rest- calls, because the client should do that according to the kubernetes spec, not through something that we not do some extension we've defined.

D

I think if we can figure out how to fit things into the the standard kubernetes mechanisms we're going to be much better off so, like fetch cbd, is probably not something you want to have, because that's custom for this, using like a field selector that gives us the info that we need would let it be generated by client go or java, client or whatever. We have by saying here's a field selector or by cube cuddle by saying here's, the field selector. I want you to use or something like that.

D

So that's the challenges that I think is figuring out how to put things: how to form a kubernetes request that will go through the api server get to the aggregated api server and have all of the information that we need to pick. The right, csi driver and the right information to give to the csi driver to give back what we need, which are the change block delta records.

A

So far, I think what would happen is, if you, actually, you know, add that you know api server, they're, adding those steps yeah.

B

Definitely uh also in the beginning.

A

Can you also add like well, you take the first snapshot by you know you create the ones and then take the second step, even that one, including that. So we know we see the whole. You know all the steps, because otherwise you jump to here right or something yeah.

B

A

You are familiar with, but other folks are not very clear.

B

I understand yeah. I just want to give an update because.

A

B

Event come back, we will present it more.

A

Okay yeah, so, let's just probably yeah. Thank you for.

B

The feedback- uh I will take this feedback okay and discuss with uh ivan on modern detail uh like I said this is still just like a quick update and I don't want to get into much detail before he come back. So I think I just gave an update for now. I think I'm done with my update.

A

Okay, all right! Thank you. Let's see.

A

Okay, so basically outline you're saying this was a kind of written in there. It says it's different from customer resources, uh new kinds of objects, okay,.

D

You know, I think it's accurate to say it's a different way to add new kinds of objects, aggregation, aggregated api server and crds. Both have the same effect that you will wind up with new resources that appear to be there on the api server.

A

But it will be good to show, like you know, like, for example, in the first you create a one in snapshot then create a second volume snapshot. Those are customer resources right and then uh like when then the third step after that, the backup software, where I want to uh get cvt. So what does it do? Does it construct some just directly construct, but this one is going to go through. I thought it's, the the aps. Server will do some conversion. That's what I that's what I thought so, but that's that part, I'm not clear.

A

So the api server should.

D

Be in between otherwise you're not really doing aggregated api youtube.

A

I'm looking at this one right, so it's a this is like a different server. It's not yet kubernetes. So it's very confusing to see like it's. This line directly come here, yeah, just adding those like a backup. Software constructs what talk to the api server and the api server does what right. So that part is missing. I think, and it sounds like whatever you have here. This is the response and also like this response is going to going to where it's not like.

A

So this step is that direct. This is the returning step does not involve akso. Is that right, so this will directly go to pick up software. I.

D

Believe if it goes through the api server, because the api server is essentially.

A

Yes, all right, so then we need some details because then that if you look at the you know, what adeline was asking was why we are having all of this delta in the status again, because then we are not. We are not solving the performance, the problem at all, because all of us showing up in the api server right. So that's.

D

The question we ought to see never gets stored in the api server. It just appears from the api. Just think of the api server is acting like a proxy here. Okay,.

A

D

A

D

Which style we take? It's going to be a proxy and just pass the data back from the aggregated api server, but.

A

It's going to be saved in the aggregate data server, though, or what well well.

D

The aggregated api server can generate it on demand by calling the csi directory so.

A

It's basically not saved anymore. It's just like generated um on every call this. This will be generated basically.

D

Yeah so the idea.

A

D

To pass it through the csi driver, which actually stores all of it right, it doesn't even, but you know the storage system knows.

A

How to generate okay, so basically every call has goes goes through the short system. There's not nothing kind of stale right like when you do a in general when you do in kubernetes, when you do some kubeco get those information are kind of in api. So, but in this case every call actually will have to go call the storage system to retrieve it on demand. Basically,.

D

I mean you could build a caching, but that's an implementation detail.

A

Yeah yeah, I was just saying that, basically, it's like you have to get it. It's not like something. It's there. It's not like a jet like a cube cutter get pvc, that's already there in aps server, but this implement is not. Basically, you have to retrieve it from the from somewhere somewhere, not from ap.

D

Right so so performance on that is somewhat of a concern, but I think yeah we're talking. Second yeah.

A

D

A

Know the performance, then, when you.

D

A

Do a cache if you have performance about it, but.

D

I think it should be. You know with a with like decent. I don't know, I don't have to work on easter anymore.

A

Yeah, maybe that's secondary thing. We can consider that we can think about later. Let's you know figure out all the communication between backup software again server and aggregate the aps or first right. Let's all, let's make make sure everybody is on same page on this one, and then this uh you know additional performance issue. Maybe that's secondary.

A

Okay, I think that's uh that's good, uh I don't you want to say something or you just want to put it in there yeah.

E

I was going to ask the question: look so I shared actually a link in the.

A

E

Which kind of talks about the put look? This is more about authentication, but it kind of puts a picture. You know where the aggregated place ever sits relative to the kubernetes api.

A

Yeah this is actually yeah. This is good right, so this actually shows you where they are. Oh, uh okay, so me in the middle, this is the api server right in the middle, correct and then under yeah, so so far yeah. I think this is a good good diagram, something something you know to show all three I'll just add these comments.

D

Yeah and and some of the good things we get out of going through the api server is like the authentication step there right that all happened in the api server.

E

So sorry, one thing that it was that wasn't clear to me was this: um this um api server extension thing, that is, that going to be the csi driver, or it's going to be also another proxy to talk to the csi driver to get the change blocks.

B

Just another proxy to talk to the csi, the the thing on the left on the on the right hand, side, the green box and everything is- is still similar to what we've been talking before that we still enhance the cs. The csi driver vendors still need to enhance their driver to support these different shows the delta and the you can see those two blue box right. There is what we, the mechanism for for us to eventually call these csi vendors to support the delta to get the dell technology.

E

Okay, so we have basically two processes. You know from the backup software. It goes to the cube api server from q, api server. It goes to the aggregated api server from my apis. It goes to the csr driver.

B

Yeah, that is the true hub that we were trying to see if we can reduce uh right now. uh We need to do that too hard, because the cbt api, aggregated api server cannot talk directly to the csi plugin.

A

Yeah, that's like a requirement. The aggregate api server has to be singleton so yeah, so you have to have this one. This is like a central one, and then you have yeah.

B

A

Confidence, whatever.

B

Yeah yeah the blue block in the center. It will be one per cluster and the blue block on the right is one per csi drive right.

A

Okay, all right.

B

Again, I mean we will come back uh two more weeks and I then come back and we we I went and I still need to work to figure out all the details we haven't got that far yet.

A

Did you did you look at the uh young's comments he has some link links is that uh is that the same.

B

Yeah, I I I I I have. I think I read the link, but uh I I haven't spent much uh thought on that. Yet.

A

Okay, all right; okay, anything else on this.

A

If not, then move on to the second topic, uh which is the warning group, I just want to talk about, as I have a question that I want to ask folks.

A

uh So I think we look at this a long time ago, but I updated the cap recently um well, actually not recently, it's being a few months so so now um the the api part is more like um it's more like the pvpc, a body snapshot. One is never content that type of model. So basically we have.

A

uh We have a volume group. We have one group class, we have what in group we have one group content right. So basically we have two api objects for each api resource. So, uh similarly for the group snapshot, we have a uh one group snapshot class and there's a group snapshot and group snapshot content.

A

So um there are some some concerns that we are making this uh too complicated that we need to have two api objects per resource. So I just want to make sure that we cover the use cases. So if say, if we only have one, if we only have one group, but we don't have this one good content, does that meet your use cases? I believe that's the thing I want to figure out um now.

A

The reason that I want to go with this model is because this is a kind of a same same model as the pvpvc and volume snapshot and such content, so this supports static provisioning.

A

So if you have a let's say, if you have a volume group on your story system, uh you want to just import it right. That's a static provisioning. Then then, with this model, you can support that, because in the wording group content, you can specify the handle to the volume group on the storage system.

A

Admin creates that and then then you, uh the user, can create one group to have that bind with this. One group content it's kind of very similar to pvpc and all in snapshot on subject, content.

A

If we only have one object, then we won't be able to do that, because uh you know you're not this, because this object will be user space you're not supposed just to directly put your volume group handle, which is which is a handle on your system directly into a user object right. So if you want to import a particular only google handle, it doesn't make sense to add that directly in a user space object, so meaning that if we only have one api object, then we couldn't. We couldn't support that particular use case.

A

So I just want to know how useful is that? How important is that use case to do static, provisioning.

D

Static provisioning of what sorry I want to stress.

A

Of a voting group, so let's say if you have, if you or maybe we can think about this one like group snapshot right so let's say uh some three system: they support consistent group snapshot. They have this. They have this on their.

A

So thinking about your, you have a snapshot on your system or you have volume on your system. It's the same thing right. This is assuming that you already have this on your study system. Of course, if you don't have that in your source system, then that's that's a different story. Let's say: if you have your, you have a consistent group snapshot.

A

You have this created on your storage system. You want to import that into kubernetes.

A

uh Yeah is that clear? This is basically this.

E

A

Same concept, as uh you know, let's say you have the uh yeah you have you have your you have a snapshot created on the on your search system. You want to say, hey. Can I manage this snapshot in the in the kubernetes right? So here we are talking about group snapshot, but it's a similar concept.

A

uh So is this uh something? Is this an important use case, or is it okay that we do not support this use case? We only support dynamic provisioning was that I guess so.

D

For restore, if we need the volume group snapshot to create volumes from then in the dr scenario, you know the everything's gone, and so we have to do static provisioning in order to have the snapshots be present, so we can create volumes from them.

A

uh So you're saying in that case the the group's actual understood system, if that's something you're trying to uh basically do the restore from, is that that's what.

D

A

So you say I take a group snapshot, um it's kind of my you know my for my for my disaster recovery for my backup. Whatever uh now your queen is destroyed, I want to restore that back from my system. How do I do that? So I need to be able to somehow import. This particular uh group snapshot id into kubernetes, so I can use that again that can restore from that something like that right, yep, I see yeah. That's, I think. That's a good use case.

D

So the question would be: does volume group snapshot? Is that actually a necessary thing? So are there systems that will only do volume group snapshots or you know, could you do a snapshot with a volume? I don't know.

D

A

Let me just write down this. You scale you're, saying that uh to import.

A

Snapshot id for dr or something I'll, just add this one: that's! What do you mean? Okay, go ahead. What is yeah, I guess.

D

I mean you may not need to. It depends because you've got like a list of volume. Snapshots are.

A

Right that is, if you right, that is, if you do not, um but okay, so in that case you're saying, then you just import those into your snapshots right.

D

Actually, I guess that does the right thing, though, because if you have a volume group snapshot, you would the the logical operation is. You can create volume groups with that as the source right and then that not only creates the volumes, but it also groups them together.

A

So yeah, okay, so I think um that's the thing. Actually, I think in the in the first phase, we're actually planning just to support you to like uh do individual restore like create a warning for snapchat. So for that case, actually I'm just talking about this is like facelift uh proposal.

A

In that case, you probably don't need to import the group snapshot as long as you know your individual snapshot handle. Then you can actually restore that. However, I believe there are, some storage systems does not actually return the individual snapshots. I believe seth. I think we have some. There are some discussions on that cap so that I believe what I learned is. They couldn't really return the individual snapshots, so they only, they will only have a group snapshot in that case, that's the only way for them to restore right. So maybe I'll.

D

A

Yeah that makes sense.

D

A

D

In order to get back to a working system, so our goal is to be able to create a cluster. That's basically that's functionally identical to the original one. So I really think we need to be able to recreate the volume group, because you know if you you back up back a back up plus your you know, gets hit by me to your restore. But now, when you go to back up the restored cluster, the volume group isn't there.

D

So somebody's got to recreate the volume group in order to get correct backups after you recover.

A

I'm sorry I was typing.

B

I think it depends whether whether your system needs the volume group to function right.

A

D

Continuing backup is functioning so when you restore the cluster, if you can't keep doing backups the way you were doing them, then you've got to fix it. So you can. I mean.

A

You talking about the are you saying like: we also need to kind of import. The volume group, if that's there on the system, is that what you're saying so then we'll continue.

D

A

Let's say if we want to continue to do this, let's say: let's say periodically: we are taking a group snapshot from a one group, so when we restore our system back, we want to be able to do that again. So we want to be able to get our original one group back. Is that that's? What do you mean yeah.

D

Something yeah absolutely it needs to be grouped. Otherwise, it's not going to. Otherwise your backups as you move forward are going to be incorrect.

D

So I don't know if it needs to be the original one or a new one, but it just.

A

Yeah but this is you, get a one bag.

D

um There has to be a mechanism for that, otherwise,.

A

So, oh so this one uh is that maybe this is maybe all right now you have that because uh you can you're young of us are there that you can separately be fine? You can create that yamo file again, then, to create a new group. You probably don't need to import your original.

D

You may not, um but I think we just need the workflow, though, what what's going to happen on restore so that you get back to a working state and just work through what? What steps you need to go through and whether those are easy or hard.

A

Right so this one might be okay, I'm thinking I was thinking. This is a problem this one because they don't have individual snapshots right.

E

But pushing on systems that support both regular or individual snapshots, as well as group snapshots, is a group snapshot going to be represented by a collection of individual snapshots, or is it going to be, for example like for for a group snapshot? Is it going to point to a bunch of for let's say you have a group snapshot content? Is it going to point to a bunch of regular volume snapshot contents, or is it going to be like more self-contained? You know more independent.

A

There is a so for in every uh so for the syst for the system that that will return the individual snapshots, though, in that case the snapshots will have a reference to the group, so they are kind of you know, they're related and in group snapshot also, there will be uh in the status. It also has a should have like a list of snapshots, so it's like the information should be stored, so they they. You know so that they're not not like completely by itself, because you're not supposed to uh you.

A

You should not be able to delete the individual snapshot if they are in the group, because otherwise you know, then maybe at least what time you have some problems.

E

Okay, so the group types are a collection of individual types, whether it's group snapshot or volume. So there is a group snapshot group yeah.

A

E

Volume volume group they both.

A

Are a collection the difference is like for the for the volume group uh you are allowed to remove. You know, remove a volume from a group right, you can. You can do that, um but for snapchat doesn't make sense, because if you take snapchat, that's you know that's kind of frozen that you.

A

You need to have that one uh to do your restore that that that's why there is no there's no way for you to like just say: we are not going to provide api for you to say, remove a snap one snapshot from your groups. Remove one individual snapshot from your group snapshot. That's not going to be there! That's because that's not allowed right! It's just doesn't make sense.

E

And what's the relationship between volume group and any audience staff such groups, are they like related like when you take it.

A

Yeah, when you take a group snapshot, you you take a your source. Is a volume group.

E

Okay, then, the volume group that finishes changing may have implications too for snapshots like.

A

I understand you're saying that.

E

A

No, so well we don't so it's only so when you restore it, of course you can, you will only restore back uh whatever you know at the point of time, you take a snapshot right, so you say I have 10 volumes in my group. When I take a snapshot, then you, when you restore your media restore, then you should have 10 volumes, even though, like after taking the first snapshot you have already.

A

Let's say I deleted the two right, then that's, that's your next next snapshot right so just think about the same same way as if you make changes, you know to your voting or not. You know every every point of time. Your snapshot could be different.

A

Right so yeah, so when you restore your sway back definitely really depends on uh the. How many volumes were there when you take that snapshot.

E

Okay, so basically we're saying group snapshot or group snapshot content becomes decoupled after we take the snap just from the volume group.

A

Kind of you can still, you can still yeah. It may not have the the same. One might be ship anymore right after afterwards, because you can modify your group. Yes, then you can use your group snapshot to create a new. The reason is you you have the group snapshot. Is you could create another, a new wording group right, we're not talking about right now, because we're not supporting the in-place type of thing anyway.

E

Okay, so I guess back to dave's comment that you know it depends on how easy or how hard it is the process. I think.

E

Given that groups snapshot maps to an individual list of individual snapshots, people can individually restore their snapshots, but.

A

That's painful, that's painful! So would you allow them to yeah, so we do allow them to uh to restore individual file like individual wording that that is okay.

A

But what I'm saying is that you won't be uh won't be allowed to delete that from your uh from your group snapshot once that's uh because, because if you, because you the reason you take because that's a there's, a difference between the um range between a volume and one group and the relationship between a individual snapshot and a group snapshot because uh volume can be dynamic, a group snapshot is not dynamic.

A

E

A

E

Yeah, that's fine! um That's okay! That doesn't have any implications on restore.

A

um Yeah, your store should be fine yeah I was, I was mainly just want to see, for you know just to see if it is.

A

If, if we can just have one object right so as if we want to support this case, it looks like it's not possible.

A

What's this looks like not because they don't really have the, because the only way for them to um restore- let's say if you want, if they want to do this importing, then they have to import that group snapshot because they don't have individual snapshots to import from that's the same right. So if they want to.

B

I I have another question related to this volume group snapshot. uh It does it that's conceptually uh atomic or individual or individual lab snapshot or belong to this group snapshot can be accessed, you know gradually or.

A

So those are or depending on the backhand's uh indentation. So initially I was trying to say: hey. We have this in a consistent school, but then there are. There are some problems for like if we want to reinforce that in in kubernetes. There are you know, third systems, uh they are, they all behave differently. Some storage systems can support consistent group snapshots, but others cannot. uh So what we do is we just say: hey we sub. This is our volume group, then we'll leave it to driver to interpret.

A

You know whether they want to claim their their wanting group is consistent or not. That's that's their their thing. We just provide a way. We just provide those apis for them to um yeah.

B

So that is up to the back end to go right right, because there's no.

A

Way for us to guarantee, because you know it already, there's a lot of tricks right, you could. It could get things wrong right if they let's say they, um you have two volumes. They are not uh in the same story pool then. Maybe that you know then then maybe this three system cannot support consistency right. So that's! The type of thing we don't know is every source is different yeah. So we don't.

B

Form a backup vendor part of you right. We we, when we can say that it is safe to start using the content of this volume snapshot like uh is that, are you gonna change the status to ready or what, uh when the volume.

A

Right yeah, so there are status like that right, so there will be status like to say uh I have finished adding you know this volume and then we need to change the status the status update. So uh like the current design that right now I have two objects. So if it's just one object, when you tweet me to the design but but the api have a status field, you can take a look and see uh if this one is being up added or not, and then you can look at.

A

There is a list of the pvcs. So that's the current proposal. Of course, the you know the design still up to uh you know, subject to change, but uh yeah there will be indication there to say hey. This is ready, not ready. Something like that.

E

So back to my earlier comment about how these volume groups or volume group snapshots are implemented.

E

You know one of the arguments in the early days of kubernetes storage for having this dual object model was there may be sensitive information within a pv or you know, let's say volume snapshot content. You know that is we don't want to expose to users, regular users, that.

A

Was one argument right? I think right right, okay,.

E

I think- and I think if you know these group objects, like the volume group snap shuttle volume volume group- they are just pointers to pvs or pointers to volume snapshot contents, then I think we're okay. As far as security, because they're just pointers, you know they don't really divulge the content of those objects.

E

So we can possibly even go with a single object model and as far as basically that's why security we're okay with single object model. As long as how we refer to you know, if we construct these group objects as a collection of individual objects, you know so basically there's no security problem. If you go even with a single object model.

A

Yeah, so there are uh so I uh I've kind of seen. I know there are some saudi vendors. They have kind of already implemented something like this right. So I see some uh some of those information, pretty simple they just they just kind of used to the you know, pvcs directly. It's there right. They they have a very simple way of grouping them just using the labels. That's a that's a very simple way right um uh but of course, like you know, there's some other use case that we we talk about.

A

That may not be able to support right. So uh if we only rely on the existing uh ppvcs, then of course right we don't. Maybe we don't have additional information that we need to get exposed in the user object, then that maybe one object is not enough right yeah. So that's one argument there.

A

um So if you can just group you can just choose the same.

A

Okay, all right so just want to make sure like you know, either way, I need to make sure we can meet those requirements.

A

uh Anyone else has a any comments: uh hey tom, I see you are there. I remember you were interested in voting group a while back. I just want to see if you have any comments regarding this tom watson.

A

Mute fong, do you know, do you know this like? I know you guys are interested in this uh voting group a while back so.

D

Yeah, I was going to say: let me review it and get back to you. We do have an.

E

Implementation so.

A

That's awesome actually would be good if uh uh yeah that, then we can uh see. You know what you have learned from your implementation. Yeah yeah. We only use one object, we'll use one okay, cool.

E

B

A

Do you have both group and group snapshot or just one.

D

E

To volume group snapshot.

A

uh Okay, I think that seems probably more like uh I know. Port works has some invitations. They just they just have a group snapshot so uh yeah. So that's something we could also consider that's more simple approach.

A

um I know hpe has an implementation, I think they have to. They have both yeah yeah. So yeah, please get back to me when you get more information. Yeah good I'd like to take a look and see you know what we can support, what we cannot support with one object.

E

Yep sure will took a note to do that.

A

I'll just now add your name here, just say: action item.

A

A

Any other comments on this topic.

A

All right, if not then we'll move on to the last topic from.

B

I was I, I think we only have a few minutes left, so I'm not sure if we want to start the discussion here but yeah. The idea was that uh remember we we, when we do the cbt.

B

We start the conversation about the api very enhanced and to enhance to the csi ap for a long time before we can actually do something about. So I want to start the the conversation again for the five chain. This is similar to the the the truck the blockchain list, um in which for uh many uh vendors out there who's a sport vendor they export their pvc file system. Pvc. uh On the back end. This could be a national fire system volume on the back end. So uh for that we to do effective backing up.

B

We just need a list of files that have been changed and I've done some uh research. I found that you know like power scale, for example the our pilot scale.

B

It have a way for to get the chain list, information in which uh the the the each item in the chain list only uh specify the uh what type of chain it is and and and the the path, the five path and so on so forth and and and if we uh back the vendor can get a list of chain like that, then we can simply back up these files instead of you know, walk through the whole uh snapshot and then backing up try to check everything manually, I mean checking up individual one by one um by a g3 workers, something like that.

B

So uh we kind of uh for on a few minutes we, I may not be able to present, I think so and also I have some network as access issue. So I I rather start this conversation, maybe in the next um workgroup meeting, okay.

A

Now you want me to move this one to the next uh meeting: yeah yeah, okay, sure, good night meeting, yeah yeah I'll. Do that.

B

Now the the goal for that would be, maybe we can settle uh like a csi spec for for for that exchange for that chain list to say how it looked like and then other all the vendors who want to support the csi spec would be able to. um You know, implement that and then any vendor who want to backing up effectively. You know can do that. Csi enhancement, yeah.

A

Sure, oh, did he just got shot.

B

Basically, very similar.

A

Okay, yeah sure sure, okay, yeah, okay, yeah, oh I'll, move this one to the next meeting. Yeah we're gonna have like three minutes, yeah all right anything else.

A

Okay, that's it for today thanks everyone bye.