Kubernetes Storage SIG Volume Snapshot Workgroup, 15 Oct 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Storage - Volume Snapshot Workgroup 20181015

Description

Meeting of Kubernetes Storage Special-Interest-Group (SIG) Volume Snapshot Workgroup - 15 October 2018

Find out more about the Storage SIG here: https://github.com/kubernetes/community/tree/master/sig-storage

Moderator: Jing Xu (Google)

A

B

Again, this is six storage weekly meeting, and today we we have a few like items we want to discuss. So, let's start from topology like we discussed the last time, and so we have a PR for adding topology CSI volume snapshot.

B

So is it's very similar to volume, topology and last time, I think we have a little bit issue two comments for replication, because in the comments we mentioned, volume should be accessible from two zones, and here we mentioned okay, for example, specific volume is accessible from two zones: okay, synchronously replicated and it seems, require started provider so that if volume is accessible from two zones, that means the volume should be replicated in. Those two zones but I are thinking is, should not such a requirements.

B

We should specify here. So it doesn't matter like how volume is otherwise it can be accessed by two zones. That's fine, it doesn't, it doesn't mean it should be, must be replicated in those two zones. So is that.

B

So that's why I think here, we kind of add a little bit explanation. This requirement may be satisfied by replication. We cannot assume it must replicate.

B

But still this is kind of not very clear I think. Maybe we should remove signal a replicated at all I.

A

Think this is fine rewarding it this way and making it explicit that replication is not required. I think that's! Okay, okay,.

B

But when we say creating requires a specified volume to be accessible form to zones, aka signals a replicated. What does it mean here so.

C

I would agree that synchronously replicated is orthogonal to the access requirements, and so it's it's not an aka.

A

Yeah, so we should reward that to remove the synchronously replicated part and leave the while accessibility requirements may be satisfied by replication, blah blah blah.

D

That makes sense to me: okay,.

B

D

Synchronous is also superfluous, or getting rid of that helps.

B

Okay, sure, so if we remove this, since this is relate to both volume and snapshot and I, think if we get agreement, then we can change for both volume and snapshot.

B

And other issues related apology, I.

B

Think that's the only comments we got so far relate to topology.

B

Okay, if not so, we will make the change and then see like a whether we can be able to merge it soon and see also seen whether anyone has other acum comments relate to the topology.

B

Next item, so we mentioned, we want to add deletion policy similar to volume like PVC, so they sound workflow from volume with the PVC right we add deletion policy force natural to one thing is the lesion policy is something only system I mean we are thinking should specified, so only you can specify the addition. Paula screaming snapshot class right now. It's so we require deletion policy. Can you see it I think recurrent policy or retention policy?

B

So we have only two options: one is retain. The other is delete so by default. If you don't specify any division policy that will be delayed, that means if a user delete a volume snapshot, then the corresponding modem, structured content and also the underlying modems and shots will be deleted.

B

But if your division policy is retained, if a user in each volume snapshot the associates, volume, snapshot, content and also the physical snapshots in storage will not be deleted. Only the volume APR.

B

17 can change through the bottom snapshot. Class user cannot directly change division policy.

B

Any questions for this deletion policy.

C

Yeah, that's sort of a binary decision, but in like surface there's a time to live requirement, you might want to consider.

B

You may want to consider what.

C

Instead of saying that the replica is deleted, when it's not used directly, you give it a lifetime that a minimum lifetime that says after that lifetime.

C

D

Mean if someone wants to manage their kubernetes snapshot deletion snap solution through kubernetes, they would be, they would implement that time to live on top of these api's and then they would expect deletion to actually delete it. If someone wants to manage it underneath kubernetes, then Carnation me touching this stuff.

D

D

Yeah I think these two options are enough: flexibility that people can get done when they need to do there'll, always be people doing funny things outside of our knowledge.

B

Bottom central class, so only system mean, can change it and because, since bottom central class is used in creating snapshots right and if the data policy is defined, let's say return or meet. Then when the snapdragons credit, the deletion policy will kind of what's in the snapshot, content API object and depends on how you specify in the class right it will follow that either retain or delete and user cannot change. It afterwards only says the mean might be able to change it. So.

B

It's fine like mmm yeah, it seems fine. Okay, so next is for deletion. Finalizar pizza is to protect either volume snapshot or volume being deleted while it is being used- and we do have a more document explaining some of the protection we are trying to implement. So first one is warning snapshots content.

B

We want to prevent you delete the volume step, not content directly. Well, it is still bound to volume snapshot and this is and by a snapshot controller. So we will add finalizer right when volume snapshots it's great and we also check if the deletion times the man stamp is already set for this content, then we don't need to add feminize there anymore. Since that indicates this content will be deleted.

B

So either the controller can add the panelizer or we could have automation controller like to add randomizer.

B

Then we will also check when whether we should delete analyzer when user as they want to create a delete, write the division times then will be added to the volume snapshot, content and I in at this moment the controller will further checking whether staff chart content is currently being used.

B

That means, if it is currently bound to a snapshot, we cannot be removed finalizer, so if analyzer will prevent prevents, you delete the contents right now. Only a currently the content is not being used, or in other word it's not only to any volumes and shot, then the finalizar can be removed and the content can be deleted.

B

This is the protection for volumetric content and we also have a fertilizer for volume snapshot so because, when you're trying to delete water set shots that's possible. Currently, this bottom snapshot is used by creating a volume, since we can provision a volume from a snapshot. So that's the goal for protecting the volumes and trust so we'll add the final result of watering snapshot to and when you want to delete a shot, it will check okay, whether this volume snapshot is currently being used by any volume.

B

So the controller has to go through the list of PVCs and also check whether those PVC has a data source that match this bottom snapshot and only if the PVC already pond successfully. That means the volume is created successfully, so don't need centrally at all. If the face is still pending right, then we can now remove the panelizer.

B

Any questions so far for fertilizer for worms and content, I.

E

Have a quick question: can we delete the PVC if there is a world of snapshot created from it.

B

Yes, so our assumption is PVC the volume life cycle. It's independent of step, shock life cycle, I know, there's some storage, that's kind of have dependency so for snapshots and their volume, but that will be determined by the bottom plug in. So, if you're trying to delete, let's say a volume. Well, it is has some a snapshot, a social visit and the plug-in does not allow it. The plug-in will return like an arrow when you try to delete the volume.

B

C

B

Guy's, like modern plug-in, should return alone. Then the Canalis, as it will kind of report this arrow up to the API, so user know why it's cannot delete it, but we don't provide like protection at our lair for that, because different volume plug-in have different, very different right. Some are abandoned. Some are, is NSC between monuments and shots, so we can not make assumptions we yeah.

D

Sorry, what about protection during the the brief time, while the snapshot is getting created so after you've created a new snapshot, while it's still being.

B

Finalized err we put PVC so because we create snapshot right through bottom. We want to prevent. You deletes the volume while this snapshot is still being created, so we still plan to do this through snapshot controller and when to add panelizer. So when there is a snapshot request coming right, we can add finalizar to is associated PVC if there is no panelizer before I did it before, but there is to remove analyzer on this PVC at a bit trickier. So we are saying: ok after you die or create snapshots, the central is already cut.

B

We don't need the void. Volume exists anymore, so we can remove analyzer at that moment, but we don't know actually whether there are other snapshots are being created by this volume at that moment. So we are thinking to use kind of card account annotation. So we can check whether currently there are any snapshots being created or not.

B

So the idea is so when a snapshot request coming, we can like in increase this set charts in creation count number and after the snapshot is cut. So remember: when snapshot is cut, we will put the creation ten into the API object. So we know when the snapshot is cut. It not necessary.

B

Central is ready so as rise cut, so we can P decrease this number and the controller will check whether the count number this count is equal to zero or larger than zero and decide whether we can remove the final answer or not, and we are thinking also it might be useful. We can have also snapshot ready, stanford count so to indicate how many snapshots is created and already ready for this particular PBC.

B

B

So any questions for the this award idea for remove finalizar using this snapshot in creation con this.

D

May be a dumb question because it's more of a general question, but um what prevents race is on incrementing or decrementing the count in kubernetes. Is there a way to do a transaction that will prevent multiple increments from overwriting each other? So.

B

Update object, it will return like um version. So if, when you update I during this time, there is another update and we can check, I saw version and no and it will fail to update and you need to update again I see.

D

Okay, I'm gonna, make sense.

F

B

F

You mentioned admission plugins multiple times on this page. Is this same as admission control that is part of the API server? Is a separate plug-in dedicated for snapshots, so.

B

We'll have decades admission plugin for snapshots; no, it's not kind of it's part of it will be added as part of central controller, so we already have a PR to add animation controller, but it's kind of integrated into the snapshot controller. So as rise you as a deploy, a central controller and set up the service for the mission, it is a web cool current. So as long as you set it up, then it will be like part of the controller that makes sense. Yeah.

B

So there will be some kind of tricky thing. We have to make sure it because, let's say grace: I'm shouts call, maybe college multiple times if, like the first one is takes very long, so we should prevent you to like keep adding the count for the same snapshot.

B

Something related to that. So we will take care of that when implementing this.

B

So all this panelizer right, we might have this tomb. We cannot have like 100 present guaranteed, so there still might be some kind of risk condition, because when you check right your check, okay, it is okay to remove panelizer, but before you remove it, the condition changed, and so, but it should be rare. So basically, we cannot have a automic like operation by checking and then remove analyzer. So after you check, there is something might be like. Another event happened and it's no longer true, but we still remove analyzer, but.

B

This is the way for PVC to so we don't have like 100% guarantee, but we should have a protection for most of the cases.

B

Any questions for finalizer production, so we have visited three fundraisers one for content. One pole snapped rod and one for PVC to prevent different scenarios.

B

Okay, so so far seems it's fine, so we'll go ahead. We already have this working progress, passing so we'll update it when it's ready and for this count thing currently, we propose two counts, any other, like suggestions, wise means how many step prod is currently being created. That is increasing count the other week already. Stan short count. That means how many snapshots it's ready for this PVC.

B

B

Okay good, so we can move on to next topic, which is relate to cases in group. So seeing also have a document for this.

B

So see you want to give a like interaction like the idea of this group falling group and the snapshot cool yeah.

G

Sure so, I think in previous meetings there was some requests to provide consistency, group support. So basically, this idea is coming from cinder who implement it's fine cinder.

G

So we ultra don't call it consistence group here, because we want to make it more generic. So it's a warring group instead and then actually that can support storage systems that can provide a consistency, consistent group snapshot and also those that do not provide the consistency and then they can still use this grouping concept.

G

So this grouping concept can allow the volumes to be grouped together and then to be managed together. So there are mainly two to eighty objects. One is called volume group and the other one is called a group snapshot, so the warring group will have one group spec and one group status and in the inner spec, so we will have a the third take a source. This is the only when, if you try to create a one group for all equipped snapshot, so you need this.

G

A source they'll give you the ID of the quick snapshot, but if you just to create a brand new modern group- and this you don't need this source and we have a boolean quite a consistent group snapshot. So that can be true or false. But if what is force I am named, and then we have this list, the list of system volume create claims. So here you can define a several a PVCs that you want to be included in this group and then the next one is the status.

G

So here we have a creation, time, good creation time and then the the spool and whether the you know the group is created, is finished, creating or not so the this is a PVC list. It can either be empty or they can be a list of PVCs. That has definitions.

G

If it's empty, of course it just create empty goop. And after that you can add pieces to it. But if it's you already have a list of PVCs, then the PVCs will be created and burned. After all of them has been plugged.

G

Then we can change the group status to you ready, and this is a the arrow not sure if we can use this existing war in the bloating barrel. If you object or if we need anyone, but we just use the existent for now.

G

So if some error happens, if you, if you create a group of volumes, if you say if you have five in the list and you create it as three successful about a fourth one field, and in that case, so that's a failure, so she did return failure further for the entire group and so the so. This is the same as the or the other like create. A volume operation creates no operations where the Sun also needs to be idempotent, so which means if the driver creates a warning group.

G

If, if it let's say F, 3 is successful and and then the next one field.

G

If, if the CEO, if quanities makes the same clock in driver, needs to make sure that it doesn't create, let's say it doesn't create an should justice to be 5 and then needs to handle the failure cases so either. So, at the end, if the request is to have was 5 PVCs, then at the end it should just be 5 created to support idempotency and so.

D

So I have two questions.

D

Is the theory that the PVCs are defined externally as PVCs, and this is just a list of their names or the PBC definitions actually inside the subject.

G

So yeah you do well be it actually will be inside. So it's so it's it's similar to how you do. Let's see the staple set or you have those PVC definitions inside.

D

So it's really a template, object, ray.

G

Right yeah, so it should be yeah, so should be a template. Yeah not well yeah.

A

B

Template is awesome. Actually we are thinking we should support. Suppose I say if you create empathy, one first, when you allow, you add a policing list later right, yeah, so.

D

The reason it matters is because, when it comes to the implementation of you know, volume groups and consistent snapshots and stuff a lot of the time, the group itself is not particularly mutable, like you can't add things to the group that weren't originally in the group, you know implement.

H

On the implementation.

D

Side I mean yeah I realized in kubernetes. We can find whatever we want, but I'm just saying like if you, if you define it to be that the group is mutable, that puts a really heavy burden on implementations to be flexible enough to take something that is not in the group and then put it in the group is.

G

D

Clear involve moving data yeah.

G

So it's really it's really where you take it snapshot, then, when you take a group snapshot at that time, that's when you can really support a consistency or not right. When you are, you know when you're just accreting a group of volumes at this time, we're not talking about a consistent with snapshot right so similar is, is similar to what we did in in cinder is that you can. Actually, you can actually add a warning to the to the group you can remove on in front of so you.

D

Mean I can I can River volume from the group, but then still use that volume as its own volume.

G

Yeah, but there will be one each in the controller. We need to make sure that if it's part of the group, then you can't really just say delete that one friend was a it'll be check our checks right. If it has a group ID in that volume in a PVC, then you can't just delete it. You have you can remove it, but you cannot really just delete it when you kill it. If you have to delete a whole, the whole Boop, so there will be, there will be some some our changes there in the controller.

G

We need to make sure that you know you don't just go, didn't eat a lot and we don't knowing that it's actually in this group. Okay,.

D

I guess there's more to point out the district. This will ultimately set a very high bar for implementations to support consistent group snapshots right.

G

Yeah, so that's the reason we want to discuss right. So um you know we.

F

I guess the question is the question is whether kubernetes or previously sender gaining force, consistency or authenticity across all the volumes, or is it the job of the plugins to enforce consistency, because based on why push we take.

G

All right, it's it's actually so I'll just talk about instant, the cinder. That's the way we implement this one, it's the clogging, it's the responsibility of the plugin, because not all the systems can can support this type of consistence. But right now we're talking about this consistency group at the storage storage system level, right so the plug-in. Only if the plug-in can support of this, then it can do this. Otherwise it can't just say: inconsistencies force so that you.

F

The reason yeah, the main complication- is that mm-hmm, for example, let's say somebody's using 10 storage backends for NFS.

F

They don't necessarily know whether all these bombs are coming from the same back end and they cannot rely on the tables of the back end to have consistency. They.

G

Are saying that they have they want to create, like group with different recordings, are from different back-end? Is that what your Chinese say well,.

F

It would not necessarily know where their volumes are coming from right.

G

F

G

We have the so yeah, so this one doesn't have a I wonder if I just should have the like a class like we actually talked about this one. So in send our example, we do have a group type right so that you, basically you just add- need to set up ahead of time. Yeah.

D

G

I

Point out that there's nothing here that enforces.

D

G

Use the same storage.

D

G

So, basically in and then I was thinking in cinder. The only scene that we added in the group type is whether it's consistent or not. So that's why we, you know G and I. We had to discuss about that. So that's why we kind of move it out, but maybe it makes sense. You still have that have a type so that user knows. Okay, it's there are different types: these poor different functionalities, even.

F

That's sufficient, even if it's the same storage mass, they can't constant. The bottoms can still come from different vacuums and the plugin cannot necessarily enforce question to say cross I.

G

Will say whether supporters are not right, the type in the type you don't define, whether the widely supported in that and then well.

D

But but here but I think what art alone is getting at is you you could define a group that has the consistent type true and then you can have two different PVCs that are on totally different pieces of storage, and you could put them both in the group now. The Cooper days will not stop you from doing that, but now the plug-in has no way to actually.

G

I think we was because this is the same as the other. You know when you create rank rewarding you do you have this? The probationer or you know in the class you have that specified right. So here is something that we need to that like as well right. So you need to specify what is your profession, so you know, okay, this clunky.

D

But they have just having the same provision or isn't even using enough most of the time, because the limitations.

G

D

G

That's that's a driver. It needs to make sure right.

D

But but but that making sure could involve like moving the data as as as part of putting something into the group like the act of adding somebody, the group might mean that the backend, in order to make the guarantee that it can take inconsistent snapshot. It may have to lift up the data from one place and put in another place, which could be an enormous copy operation. Just add something to a group and, and you you can, you can do it, but it's it's not just you know like flipping a switch.

B

So we can I say adds according to a group, that's, although the user request is but it might fail, then we need to have a way to show okay. You cannot do this way and more.

D

Importantly, if, if I want to have like 10, my volumes in a group and I want to be able to take a consistent snapshot and there has to be way to up front. Tell the system that please put all 10 of these things in the same place, so that they can be consistently snapshotted. So.

I

D

I

So maybe I'm incorrect, but I thought the storage class, with the exception of you, know something like Trident. That does another obstruction of multiple backends behind it and there's other cases. That does that. That might do that as well, but that that should work and then it should be up to the plug-in to determine the scheduling issues from there shouldn't it. I mean.

D

Yeah that the plugin is is capable of addressing the issue. It's just adding things or removing things from groups couldn't of moving data around and I. Don't know if and that's.

I

D

Instantaneous operation for.

I

A plugin that has multiple backends yeah, that's true, doesn't.

F

Send their support multiple backends Fox Center also supports multiple actions, but.

I

It's different because in Center, when you specify the consistency group you're specifying against the driver, so it's all going to the same driver, which is the equivalent of its storage class. But.

D

In this design, you can create the volume first and then add it to the consistency group later and cinder would have no way of enforcing that it went to the right place.

I

I'm not sure what we're talking about I think.

G

I think your concerns more about whether we allow I wanting to be a do or not right.

D

When you make the group itself mutable than the act of putting things in the group and taking things out of the group, has it has a meaning as a semantic meaning.

G

As far as where the data is supposed to be okay, because I know that we have some we'd be running into some boxing insulin or twenty on your add, I need to make sure that's from the same poor things like that. The scheduler actually got involved right. You add some special checks that yeah, so definitely it can be tricky. So the storage.

C

Close over tall so.

G

This is something that we are discussing right so right now in this file right now we don't have because the chena discussed you just say if the.

C

Group has a storage class, then you can check the class of the individual volumes against that class as criteria or similarly, if I'm going to create a group with a class and I specify how many volumes like I know what to create.

C

So I think you need to add that concept.

G

Right, yes, please.

C

G

To add some checks to make sure that, and you can't add to that clogging to that group. Definitely it's just yeah. It's definitely our prawn that I they.

A

Agree so I agree with the concerns about whether this should be immutable or not beautiful, but before we discuss that I wanted to take a step back and put on my API reviewer hat and question: do we need another API object? What is the end purpose of doing something like this and I understand the idea that we want to be able to snapshot multiple volumes and end up with a consistent, a snapshot across all of them, but the way I was imagining.

A

This would work was through the grouping that we already have for volumes, which is the workload the pod. When you create a pod, you specify the volumes that you're concerned about, for that particular application and in general, when you want to take snapshots of multiple volumes, the natural grouping is the consumer of those volumes. So if we already have the pod, as a group, natural grouping may be having some way to trigger a snapshot against.

A

You know a pod object to say: please take a consistent snapshot of all the volumes that are available to this particular pod. That seems more natural to me and it can also take advantage of the consistency or the lifecycle hooks that we could inject into the pod object right. So for cue, essing and things like that, and what we can. If we have those, we could potentially end up building a system that would work across more storage systems, not just storage systems that enforce consistency. Well,.

D

And sorry, if you have quiescing at the pod level, you don't need consistent snapshots because in theory there's there's no I/o happening. You know you could just take individual snapshots while everything's aquarist and have something that ends up consistent right.

A

And if there's an application that doesn't consume the cue essing hook for some reason- and there is an underlying storage system that does support consistency groups, potentially you could take advantage of that as well. I think it opens up a lot more opportunities, but.

D

You're gonna have the same problem where you can attach lots of PVCs to a pod that don't have the same storage class aren't in the same place right.

A

A variety of reasons it may.

D

Be impossible to actually.

A

Take that consistent.

D

Snapshot right.

A

The biggest concern I had was the immutability mutability aspect. I, don't like that. A nice thing is that once a pod is created, it is essentially immutable in terms of the number of volumes that it has. So that's good, but you what you're? Probably your concern is right. You could have a pod that references three different bonds from three different backends I, think that would be okay. We need to think about how to handle that.

A

One way is that we, you know, let kubernetes handle p snapshotting and q essaying of the workload and then it just triggers three. Can you know snapshots against three backends at the same time or if it recognizes that all three of them belong to the same or two out of the three belong to the same storage system? It issues some, you know, group snapshot, call to say: please create a snapshot of volume, a and volume, B and I think I. Think.

D

What's really needed is a hint to to the thing. That's scheduling, PVC is to say, like I want I want to create three PVCs and I want them to end up in the same place, and so I need a way to basically when I create those three PVCs add a little bit of metadata. That says, you know, go I, don't care where it is as long as all three are in the same place and then, if I use those three PVCs in a pod and I can be guaranteed.

D

That I will have the ability to take inconsistent snapshot because they will all three have gone to the same place.

F

As if and he saying they don't have to, the PVCs, don't have to end up on the same place if.

D

You want to be able to do a higher level, I'm saying like the. If you want to rely on the storage system to provide like extra consistency, guarantees you need a way to before you create the PVCs. Tell the storage system. I want these to be in the same place. So.

I

I really like the approach that Zod mentioned, because that makes the most logical sense for the use case and also it makes the most. You know easy, workflow and use case for the for the end user. What I'd like to see you? The awesome, is addressed that first and then figure out. If you want to do some special scheduling enhancements for volume to a back-end for your special use case bin, then we should look at that as a separate, separate one.

B

Thing, I wonder is apartment. You can't have many different kinds of volumes so, like some opposites, enzyme are like a tomorrow and so wait which, like volumes, do it group and it's not like easy way to specify right. There may be ten volumes for that part, but two of them belong to one group, the other to put onto a second group I'm, not sure whether it's a huge case for that or.

I

Associated with that pod, hmm wouldn't you just do all PVCs associated with that part.

B

So you mean but I mean like you might have different kinds of volumes right he's.

D

Not saying all the volumes he's saying just the persistent volumes yeah ignore the number system volumes.

B

On to the same group, you know.

I

Well, the group is the pod right, so.

I

B

About you have maybe two pods should belong to the same group. Nice fan dad.

I

I'm not sure what that context would be, but what you could do, then, is just issue the snapshot on both of them. The thing is, is the only things unless you get into multi, that's the only thing is actually writing data to those up to those PVCs be in the pot in most cases right. Yes, it's not always true. So.

B

You said you have three pause and three paws have their bottoms and three volumes might be a good for this group you're trying to have a nice day replicas. So in.

A

That case, what I imagine is basically building an equivalent operation for these different classes of deployment objects that we have so initially we start with some mechanism that will allow us to create a snapshot of a pod. I, don't know call it. You know pod snapshot. Once we have that primitive ironed out. We have a ability to take a consistent snapshot of all the persistent volumes for a given pod. Then we can use that to leverage it to build higher level functionality.

A

Stateful sets builds on top of paused replica sets builds on top of pods. You could introduce higher level primitives. That say, you know, snapshot stateful set and it can do the custom logic that is required to handle that higher-level snapshotting, but then use the underlying primitive that we exposed, which was not pod, snapshot that we.

G

So gianna talk about the stateful settled it. So, let's say: if you have a the replica number is three and then in your template you have let's say five volumes were in the in your templates.

G

So when you let's say when we talk about this consistent group saying does that mean we need to create three three consistent groups? Each group will be associated with with a part or with a one replica of the stay for set, because we're not really if there are multiple replicas we're not putting them all together in this okay.

A

It would become an option at the stateful set snapshot controller level so for the stateful set snapshot, you have one of two options. One is that you get consistency or per shard or you get consistency across all the shards, and if you choose to do consistency per shard, then essentially what we do is go out and create a bunch of snapshot. Pod snapshot, objects in the pod snapshot controller handles that if they want consistency across all the shards, meaning all the volumes across all the instances, that's a little bit more challenging.

A

But you know you can imagine doing something similar to what we did with the the pod snapshot as an extra option. At that level.

D

Shouldn't we have a use case in mind before we. Yes,.

A

D

A

From the bottom up so Rina, we introduced the basic, very basic, primitive, of creating a snapshot against a single volume. I think the natural next extension of that is, let's focus on the pod. Once we have something at the pod level, we can start going up the stack and see what additional things we need to add. In addition to the primitives that we already have.

B

So I think that's also a good kind of way to design, but one thing I want to make sure is: if we kept our level right, then, when you create volume, you kind of already indicated okay, this volume should be in a group like I mentioned right, so we need to say: okay, these wanting to create a peer on to this group and the driver might do something place them into the same place or right now we don't have that wait. What.

H

I

B

Say when you create a part and specify volume, and we don't have a way to specify okay I see.

A

That, as an opt, the way I see it is the pod is the natural grouping. The easy first implementation of this should be. The pod snapshot. Controller looks at all the PVCs and triggers you know. First, it calls the qsn hook against the application to make sure it pauses the application. Then it calls snapshot against each of the volumes and once that's completed on qsr as the workload.

A

If we want to get fancier and say, okay, you know two out of these: three volumes are off the same type, and the storage system should be able to do fancy things to make sure, there's consistency between them. We could leave that logic to that pod snapshot or to figure out when it when it's uh you know issuing those snapshot calls as long as it has some way to tell the storage system. Instead of taking a snapshot of one volume.

A

I want to take a snapshot of a list of volumes, then the underlying storage system can handle that. However, it wants to.

B

So, in order to take cases and like group shots, we have to first have those volumes like created or put in the lake house consistent group.

B

A

Is a volume group kubernetes.

D

Doesn't necessarily need to know what that is, you can do that at a band, so.

G

I think I think if we do this a pot snapshot, then it's going to be so we'll achieve like application level consistency, but it doesn't really guarantee there's a certain level consistency, because so the snapshots will be taking quite a time right. So we are, we I think we we will not see to that at the first step. I think that's so yeah.

A

I think application level. Consistency should be the higher level priority and whatever mechanism you come up with triggering this, you could have additional options on it. So, for example, if we model it's similar to the way that we modeled the the volume snapshot, we have something like a pod snapshot and then you can have options that say: I also want storage level consistency by default. That option is off, so you only get application level. Consistency kubernetes can handle that. If your.

D

Application level consistency is better than storage level. Consistency. If you have application level consistency. You know when you're.

H

D

Yeah storage level. Consistency is only interesting when you don't have application level. Consistency.

F

Is quiet and you have stupid snapshots each one of them together? They're gonna be consistent because there's not gonna be any.

F

We don't have to worry about storage locations.

B

Do you think, okay.

B

I think we actually need a different direction. We need to first to see how request application, because we don't have that hook yet. So it's.

D

Probably a better place to spend effort right.

B

So so that's a kind of a separate feature we want to go so here is more or less for storage consistency shot because some provider, sorry provider like senior, have their consistent group they caught in the morning group.

B

So that's for that feature so application consistency, so we'll have a kind of a different discussion. We first we need to see how to add code for application.

B

A

Consistency and pod level consistency or application level, consistency, I, think application. Consistency should be the higher priority here.

B

Okay, okay, then so we we can have a separate design and relate to part right. So probably next time we can have some idea about that and discuss that. So that's.

A

B

Can leave it for now, then we can kind of go back to this. How we address this after that discussion, so it's almost the time and any other comments or questions or what we have discussed so far.

B

Yeah we encourage our when you start, using only like a future snapshot and you, let us know, like your experience and what's.

B

Okay- oh that's for today, if no other questions comments.

G

Thank you thank.

A