Kubernetes Storage SIG Volume Snapshot Workgroup, 2 Jul 2018

Previous Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Storage - Volume Snapshot Workgroup 20180702

Description

Meeting of Kubernetes Storage Special-Interest-Group (SIG) Workgroup for Volume Snapshot Workgroup - 02 July 2018

Meeting Notes/Agenda: https://docs.google.com/document/d/1qdfvAj5O-tTAZzqJyz3B-yczLLxOiQd-XKpJmTEMazs/edit#heading=h.2p9otr19xbfx

Find out more about the Storage SIG here: https://github.com/kubernetes/community/tree/master/sig-storage

Moderator: Jing Xu (Google)

A

I think we can start now. Yes, I just started recording do you? Can you see the recording text in the upper left corner yep.

B

A

C

We'll see my screen.

D

Yes, I can see it. Okay,.

C

Thank you. So today's agenda is first, oh, we quickly go over the yet CSI snapshot proposal and then, as.

C

Operation, design proposal to you so see and I have this snapshot. Is that proposal submitted to the community before we already.

D

C

A snapshot and like a design proposal, but since right now we are more focusing on CSI support and there are some changes so I think we, we better start a new one for easy, like a review. So this is the proposal we submitted yesterday actually, and there are some background objectives, as we mentioned last time.

C

The the main goal for the first phase is just to provide the basic API for snapshotting and also we want to provide a way to create new volumes from snapshot and maybe also restore, but for this restore we have special called in-place restore we'll talk about later and in the next phase.

C

We want to like more offer, such as the application cook to freeze application and freeze application or amounts months file systems before taking up 13 snapshots and also we want to provide higher level management for backing up and the restore, maybe for part or even superset, and so first the snapshot API design. We have the basic two API to represent snapshot, volume snapshot and the other is called volume snapshot data and that their relationship is very similar, PV, C and P me and so as well.

C

We have spec and status for snapshots you and in the bottom snapshots back. We have that raw data name which it means. That's this, the pointer to the bottom snapshot data output object and we also have a PVC name as a single field to indicate okay, which PVC want to snap shots. To here. I have a little question about is so currently we only say we once take section for volume which represented by PVC, but thinking about in the future.

C

We may want to, let's say at capability of taking snapshots for with a pv name, I'm thinking that system me my wants to take that shot for their volume it prepared, but at that time there's no PBC or socially beybattle pv. Yet so forces of me if they need to take snapshots for their volume. Currently, they have to say, create a PVC bonded and then use PVC name in the bottom snapshot, a spec which is may not smell, convenient and also thinking about. Let's say clone operation force, not Cologne.

C

Maybe we called cocky son shot across different zones regions, and then we have a different source of snapshot. So basically the source of business rod is another snapshot, so is it should we make it more like a general like source for snapshots, struct here or just currently just have a single string like a single killed any like comments about this.

C

E

You so what's the point of taking a snapshot of a PV, because a PV that is theirs doesn't have any PVCs associated with it. It's not gonna change. Unless somebody out-of-band is modifying that volume.

C

I'm thinking, like maybe let's say we have a PV- is empty, but it's possible, like cousin II might prepare some volume that have pre-populated data right. We talked about like law or those like before operations and we might have a volume that is property data, and at that moment there's no PVCs associative is it so seems like when we talk about PvE. It doesn't mean it's just like empty volume.

E

So I think maybe it's better to discuss your other volume operation proposal. First, maybe before we discuss this because I think a lot of a lot of what you have in that volume operation. Football is more generic and it can support just snapshots, but also other types of operations and also other types of yeah sure all your sources, data sources, yeah.

C

So right now, yes thinking about like the future operations and we right now we talk about a lot of different kinds of data sources and that's why I'm thinking about a single field here is like hard to extend right in case we needle and then we can leave this question here and then move on and see the volume operation proposal and then come back is so like a source struct. It was definitely like Xfinity to extend in the future and then the status we have 10 cm and conditions.

C

Currently we have many four conditions: creating uploading, ready and also annual condition, and please, like.

C

Let us know if you have any sauce or feedback about those conditions.

C

And now move to volume, snapshot data object so.

E

I want question gene if you go back to the volumes marshal condition definition. There is the status here and it can be one of two false or unknown.

C

E

Does that mean.

C

Is or condition we always have three like video, true force and unknown and in our volume, sexual conditions, because it's very beginning there is no condition, and once your object is created, we will move to creating condition and then creating creating condition will become true.

D

Is anyone else having trouble hearing, Jing, er, Jing I.

E

Can't hear fine, okay,.

D

It must be something on my end.

C

Close to the speaker and speak louder, is that better it.

F

Seems to be a little far yeah.

C

F

C

F

C

C

So after the snapshot is AP objects created and when the controller is like goddess request and trying to trigger creating snapshot, we move to creating condition and the creating Commission becomes true and then, after, like this snapshots, call comes back. That means the snapshot is cut and then the condition uploading condition if there is a bloating face for that volume. Plucking that bloating phase becomes true.

E

Okay, so it's basically just a way to I guess one way to look at it would be instead of having three conditions. We're simulating six conditions. One is like creating started, creating done uploading started, uploading done, ready, started, ready or kind.

C

Of like that, right, yes.

E

Yes, does it make sense to use the same field for both or do we need extra status field.

C

Your your question is: does it make sense to what use.

E

Because effectively status is a it's just you're, doubling the scenarios we're trying to cover with conditions right.

C

There is a like a kind of discussion relate to use status like just and I say announce that status or condition. So what's the I think Oh or we prefer to use conditions because it gives us like flexibility and because status or faces after you defined is hard to change and you don't. You cannot add new faces because application, if you see a new face like face or status, they cannot recognize what does it mean so, overall, we prefer to use conditions in our like architecture.

E

Does it make sense to, for example, have two conditions one is like uploading started, one is like uploading done or status, then.

C

I, don't think it is neither is, none is necessary, so you mean you have to uploading start uploading finish and the creating start. Grading finish so for each one I think always becomes Sherrod. It means it's.

C

It goes into that condition and the way you see another condition next day after creating right then uploading becomes true. That means it entered uploading face. You don't need additional, creating done to represent that that's sufficient.

B

C

Yeah I don't see like any benefit of adding more. Like finish,.

C

You can leave okay, okay, okay, so.

C

This is for volume, snapshot data and here the main information for all except Dana spec is the volume snapshot data source? We have this name because a PC on PB EPB has a like volume source.

C

The source current listings will only support CSI volume, so we are going to have a single field to represent CSI volume, central source and in the CSI bottom section sauce. We have driver, name and snapshot handle. That is the snapshot. Id returned from the society columns, bobbies.

C

E

B

We've got this break, I wanted to say something earlier, but I couldn't find the boy quick enough on a status for snapshots, rather than looking at it from the point of view of sort of the maintenance activity for the snapshot. If you think about it from the state of the snapshot, whether it's consistent or not, whether it is synchronized when it was synchronized those kinds of things you may wind up in better shape.

C

So you're talking about the snapshots, consistency not.

B

Just sync: consistency, and whether or not it is synchronized.

C

B

Synchronized with its source, it isn't necessarily up-to-date I.

C

C

So if I say application pause it and wash your and they take a snapshot, we shot the synchronized I'm, not sure. What do you mean? That means like when you make a request and when they snapshot is taken, because there's normally there's delay right and we have a 10-step to represent the actual time that the snapshot is taken. Okay, yeah, but yeah mmm or did I can soon see.

C

I, don't think we is that to like show we put an information in the volume sent shot, so user know like when they take it, whether they do prepare application or they just take snapshot at a random time.

C

It might consistent or not, and in the future, we'll add a hook to help user to I could take consistency. I'll take consistent same shot.

G

Can you hear me this is n? um Could you generalize that, instead of a having it be a volume snapshot source, could it just be a volume source that could point to a snapshot, but in terms of cloning, since these overlap, I could also point to the source of a clone and then that struct could be consistent between the two things for proposing.

C

The source for the volume snapshots my questions here right now: it's only PVC name, that is the source for the snapshot.

C

There is some confusion, I think kind of confusing, because here volume snapshot data aspec, the name called volume snapshot in our source. Actually, this is another.

F

We have another one yeah, we have another one called data source later, it's very confusing Hey.

G

Yeah, it's confusing and I think it's I think for a lot of people. I've talked to like it's confusing because we base it off of the binding of the PVC, but we're actually snapping the physical volume, so I think. That's also why it's confusing so maybe between the two we can come up us three could maybe come up with something that makes it consistent and clear what is actually being after clone gene. Can you move it down a little bit? There was another.

F

Structure that you show yeah this one.

C

G

George, would you mind going on mute, real, quick I can hear you. Sorry, hey no worries, no worries so.

C

Right now, we want to propose like source all those like prepare your volume and they can pre-populate data, that is the data source, so the term sauce has kind of different meanings here and this data source in PVC we want to introduce. This is first four steps on purpose right after you take that shot, you have snapshot, API checked represented, and you want to create volume from the snapshot before we always create volume like empty volume.

G

Yes and cloning follows the same thing: I've updated are designed to be the same things like I, really like the idea, the data source object and the PVC I think it flows through both designs. Well, so maybe we can come up with something different above, so it's less confusing yeah yeah.

C

Right now we are kind of proposed you state, a source with this name, for this purpose represents when you create volume, and what is, if you have you, have some pre popular data, either from snapshot or from another volume, and it is the place you point to that source and in a PVC claim is back. We have destruct and that inside of this tract in because this is snapshot proposal. So currently we only put snaps on sauce.

C

That is, would it be the API name for your snapshot, and here we only consider the same namespace, operation, R and all of these objects should be missing. Namespace and four-cross namespace will have a different proposal for that purpose and so think about other type of sources.

C

As airings proposal like clone there, another source would be PVC source. You represent that source with zucchinis name and if it is like gate, repo and the source would be a git repository name or some other related information. You can add in the future.

C

And any questions or feedback about this, they have sauce.

F

Gee I wonder if we should rename the volume snapshot, data source to warden snapshot source, maybe a little bit of commuting.

C

B

C

Meaning so it's a bit confusing?

C

Yes, so that's the idea right. We want to have a general like data source in put into the PVC, so that when a provision are created in the volume that it can recognize it and know how to provision volume either like created from snapshot or or clone from different volume neck, it can do the correct operations based on this information, and we have this struct. So it could be like added any new sources in the future.

C

C

Any ID feedback or comments.

C

Or we all agree, this approach.

G

I agree with this approach: I think it I think add other flavors. Besides get in snapshot, I mean PVC. It could be a clone, you know potentially source. It could also be you know a couple you could have it be.

G

You can point it to like a feet or something to bring that in to create it, but I think for beginners. This is a good start. Okay,.

C

Okay, thank you.

C

And the next object we have, we add, will be called a snapshot class, so it's kind of similar to storage class for QVC right so that we have this central class. So you can specify what is the blocking the driver name for this snapshot and also user can put their parameters in the snapshot class, and this is opaque parameters and were like when provisioner provision is snapshot, it can carry the crack meters and we also plan to have a default snapshot, a class for each volume plug-in for each says. A driver. For example.

C

Tc will have this Disney default snapshot class and either s can have their default snapshot a class. So if user does not specify a special class, the default will be used.

C

And this is the.

C

Storage class- oh this is just another place. We talked about before that whether we also want to put data source into the storage class so that for this, for any volume which are using this large class, they will always pre-populate data from this office. Yep.

G

I, like that, and that's what I had in mine as well, is that when you, for instance, if you had a storage class for cloning, you had a Windows image or you had a MySQL database backup, and you wanted to recreate that pre populated volume in a namespace. A storage class would be a really good way for an admin to be able to have that cloneable Peavey. You know notated in the storage class and that's what would be cloned or even snapshotted from for each of the dynamically provisioned volumes.

C

Okay, yes, so that's like doesn't mean a way to specify this information.

C

You so any other feedback or thoughts.

C

Okay, well, we can move on seems like we kind of all greatest, adding this information, that's good and for CSI snapshot a controller right. We have this diagram showing the the simple I capture right, Desiree, it's very similar to external provisioner, external or internal attach. Are we already had ez assign and also we have a CSI one driver container and communicate? Is this external step, shorter and Eastern, except charterer, will responsible to cracking the beatings, a natural and binding snapshot and sensor data objects?

C

So it has the function like kind of similar to the entry, a PD controller, so for creating the meeting and binding progress and it will watch the API server for those autumn, snapshot, API objects, and so it will run a set card along with and I say, like sterno attached or provisioner in each CSI volume driver.

G

So would the user, when you say there would be an external snapshot or provisioner in each of the CSI volume, as every CSI volume are we just need to include that I, don't understand what you mean by that as a default. It would be there or you're just invoking this controller separately. So.

C

It depends on how you deploy it so right now right if you need to have let's say: DC bottom driver and you deploy your external, attach your external provisioner and, at the same time you can just add external SEP, shorter and deployed there. The oil P point in the same place in the container you send a car container.

C

So if you don't deploy right, you can deploy separately or otherwise you you don't have these functions. You don't have this controller running. Okay, that makes sense.

C

So, since yes, I right now, user need to set it up rights to deploy. So this is just part of it. We don't and like additional work for users too, so it can just put them shorter in the deployments M file. So that's it.

C

And there are some explanation about the CSI design we had before so I think we don't need to talk much about it.

C

So those changes I think we already talked about in the CIA design meeting.

C

So this is the or the proposal so welcome any feedback comments, and I put the link, however, like Potenza, so you can easily find it.

C

Now can we move on to another proposal called volume operation.

C

So the the background, this proposal is when we're working on snapshots feature we see after snapshot is available right. There is a very common use cases replacing the volume. Please a snapshot so suppose you have your volume or social, with the PVC PV right represent by CDC PV and for different reasons like you have some data loss or failure, or you just want to upgrade your volume with new data. You want you replace the volume like in this PVC, with a new volume created from a snapshot. So actually this involves like two steps.

C

The first one is, you have a snapshot and then you create a provision, a new volume from the snapshot and when this new volume is available, you want you on point your PVC with the old PV and then point it is to this new volume which you should have another pin each represented, and this kind of have several steps and they'll be complicated and for user to do manually and also.

C

Like have all kinds of problems, if user do it manually and before we propose cuts in place, restore a restore API object, but now, when we think about in future right, we have probably many different kinds of this kind of little bit. More advanced volume operations available like migration, clone, etc and copy. So I think it might be a good time to propose, instead of just an individual like restoring objects, to represent this job. We want to have a general volume operation.

C

Api object to represent kind of this job, for operating storage, different operations, and here in this proposal, I just list a few use cases when thinking about related to volume operation in the object. So we are going to have this egg object and external controller to control it to operate on it and the first one here we caught restore is it is the in-place restore we've been talking about and in a spec we are going to have when I say the operation mónica type right.

C

What kind of operation you want to work and we have a source, we have a destination, and the in this restore from such shots case. The source would be a modern snapshot. Name and the destination would be a PVC name that it would be something you already have exist in your work mode and.

C

Where's this morning, operation, the the controller will first create a new volume from the snapshot, provision volume and then have a new pair of PVC prime PV pram. To represent this volume. And now when this volume is available, the controller will try to unbind the the previous binding like the original PVC. One points to the like o TV and then.

C

When it's ready, it will binds this PVC one to the new PVC PV Prime and the operation will tell you like what's going on and what's the condition whether this job is still pending, you want faces or the job is fail or succeed.

D

What would happen any attached case.

C

So here we about before in the first phase, we want to make sure the volume is unattached before taking like unbind unbind. So.

D

With the with this operation, just be pending until you detach the volume and then it would proceed or would immediately fail, you found an attached volume.

C

When, for example, user can I say kyouda, pod or restart.

B

D

So so so, if there was some sort of a race between this and attaching it to a new pod, somebody would win and either the revert would wait for the pod to die or the pod would wait for the revert to complete.

C

D

The other thing that I find interesting about this ISM here, the implication is you can restore an arbitrary snapshot on top of an arbitrary volume. When you know in many storage system, designs, snapshots are actually tied to the volume from which they originally came and restoring the volume to a snapshot, that's associated with tends to be very fast, whereas restoring a volume to a different snapshot that it's not related to can be more expensive operation.

D

So this sort of this sort of hides that complexity from the end-user, but I wonder I, wonder if you can get into trouble, because the user may only want to do this if it's fast, whereas you know, if it's going to be slow, they might prefer a different approach.

C

You can't kind of volume from these snapshots just on the same volume on the same based yeah.

D

I mean if you start from a PPC and you get a P V, and then you take a snapshot of that and then later you want to revert it a lot of storage systems where the snapshot is actually tied to the volume that I came from. You could do that workflow very efficiently. It would be almost instantaneous, whereas if, if you took a snapshot of one TVC and then you took a different PVC and tried to restore the snapshot to that one, it would, it could be an expensive operation.

C

B

It the case then you're going to know ahead of time.

C

B

Not clear to me that there is really fundamentally the difference between returning synchronously versus asynchronously, so.

C

Here, the meaning of your snapshot is taken from from this disk, and you want to just revert your data from the yeah.

D

I, actually I don't see any difference between these two operations other than the name of the operation. Take the PVC name and the snapshot name and I understand that the semantics are supposed to be different, but it seems like it seems, like you would use really just one operation with two flavors.

B

At the differences between revert and restore I think is the question.

D

Because the end user can't see the PV, they can only see their PVC and there's their snapshot, and so they don't care whether a new PV gets created and rebound in the background or not all they care about is, did it happen quickly or to take a while.

C

C

From and the second one is shots from the disk and then you have.

D

The and then user only sees the TVC and in both cases, you're reusing, an existing TVC. So.

C

G

Think, if you're, if you're restoring, if we use the words, the that you have here the operation to restore if you're, keeping that original snapshot I, would hope that you'd create a new volume and a new binding, because that implies you want a you, don't want to!

G

You want to base it on what you had, but you want to go forward and change that, wherever with have your Golden's not shot, you would revert back to that and continue to use that getting rid of anything that was moved forward past the snapshot shot position I'm just trying to think in terms of what we looked at with cloning. It was important to have maintain a golden image, as we called it of to be the same with the snapshot that point in time that you want to keep safe.

G

So to me, these Reed restore would be I want to keep that away and I don't want people to change it. So create me. A new volume and a new PVC for the restore revert would be get rid of anything I've done past. The point of the snapshot and go back to this original position is that the intention of these two operations.

D

So maybe maybe this is a part of ministers in the restore a new PDC is created, and so the the persistent volume claim name is expected to not exist at the.

C

D

C

D

Not at all clear from the way this was written, whether there's one PVC or two PVCs.

C

C

Controller will create a new volume and also.

D

Iii understand the difference under the covers, but from the user's perspective, they can't tell the difference between what's going on in the restore case in nerver case, it looks exactly the same to them and is.

F

That the name, maybe it isn't to explain much swing, revert sound sense. That's part, and you describe this in.

B

Terms of what is done to the underlying volume, because at the end of the day, it's important to know whether whether the underlying volume is destroyed or changed, I.

D

Just agree: I think that at the end of the day, women what the user sees and the user cannot tell whether a new volume was created or an existing volume was overwritten from them. It looks identical your.

B

B

From the point of view of the maintainability of the system that the robustness of the system, if the original volume is destroyed changed, then that's a different use case from if it is preserved for the.

C

First case, what is the original volume will be, not destroyed, so it's still there, but it is like represented by a different PB scene like PVC prime name, so users still have a handle of that all.

D

Sounds like there's two PVCs in the first case, there's the.

C

Bbc for the original volume and.

D

There's a new PVC that has.

C

F

She knows in the first kiss in the truth, there was a new PB created right, so the underlying.

C

C

So users still have a handle of that already volume. Now you have, you are going too heavy and the end you are going to have two PVCs, that's right and then the first one right you have before we will point to the new volume you provisioned and then the other one will be used to point to your original volume. So you are going to have to re amande to please represent them, so user can check and have a handle of a post volumes and they can decide what to do. Is the older one.

C

Sorry yeah there's some confusion. I didn't explain clearly so I already know I didn't think to have a new PVC to represent audio volume. But yes, after I talked to you on this come on, we are thinking. Since you have to re ohms. You should have two PVCs to represent them, so you Zurich.

B

It depends right, PVCs the volume handle that the client seems right. Yes, volume itself a bit stored, that's a that works, persistent, don't know. So that's both combinations are all four combinations of those things are interesting for different use cases.

C

So for the first restaurent he sees you create a new volume image. After that you are going to have two volumes, so we have two sets of PVC PVC and the for the second one. There's always one volume you just reword data back from some snapshots, so there's those PVC yeah, there's no. That.

D

Makes much more sense to me, but when you describe it that way then becomes identical to the volume.

C

Part they can not easily change. So that's why and if you create a new PVC to represent this volume rights, you have to maybe modify your sample up to the.

D

So you're saying well: well, there's two PVCs and two PVS they're actually getting swizzles with each other as part of this operation and that the new PV attaches to the old PVC and the old PV attaches to the new PVC.

C

D

That needs to be written up and explained because, because that doesn't make it a unique and interesting news case.

C

So here yeah, but I can expand that. So it's actually here you see I have another operation. Just for this purpose cause replace all. You can call swap PVC to PV pointers, so just some cases not from snapshot, but sometimes user have a volume already PVC resin tip, but somehow they want to have another volume right with different configurations, different type or size, and that they just want to replace the underlying model. I still have this PVC to represent it.

C

Have a new volume and user can have a PVC neo to represent it, but they want to swap the pointers between these two sets of PVC PBS. Okay,.

D

Well it, if you have this, then you could combine the cream volume from the snapshot with this, and then you don't need the restorer anymore.

C

We want to offer for this general volume operation, either some user many one just once you do one operation instead of two or some user may prefer like to know exactly what's going on. That's we can just offer this kind of swap the pointers, so user need to do the first phase manually yeah.

C

So this is a third operation. We propose right today you have two volumes right and the original one represent by PVC one and the new one represent a PVC new, and so since user, once you change the volume or the existing BBC one, so they can use this operation. I can swap pointers.

C

And another operation we use is for some water plugin. They offer some like modify existing morning. Even when it is attached like in a blast. You can change the size, the volume pipe and even like cups, and so with some disk also kind of volume operation we can offer so the operations that modify a lot easy tomorrow, night's you specify.

C

What's the prime meters, you want to change and.

E

C

So not ordering plugins cancel can provide, is right for some I say the rest. You can change our autumn time from I. Want you like a different type. Yeah I believe is.

G

A storage class to be able to clone the volume and our suggestion is, you- would have the provisioner- would support or not support us. So in your PBC, you would specify the storage class in the storage class itself. You would have the provisioner with an annotation of we call it smart phone. So if you're doing a true clone, where you're you know using the storage array to create that clone, it would support it or not support it, um and then that way it generalizes it to the user.

G

The users just asking for you know the storage class to dynamically, create a volume through doing that. It's pre-populating the volume that it's returned to right.

C

These all these are some thoughts about possible different kinds of volume operation. Some make Minox like appropriate for using this on an operation like but I. Think overall, the the general idea is that it's more like a job right.

D

C

In storage and the user can easily track what's going on where there is funding, succeed, finish and you can it's very flexible. You can add your own different type of job CRD, okay,.

G

Yeah I like that idea: I guess it just since it would be a CRD though, then how would it even if it was dynamically creating the volume on the backend? How does it tie into the provisioner, then the provisioner would talk, would execute this volume operation and provide status. Is that the idea.

C

So it's some basic building block provided by the lower level. So, for example, the restore I proposed here right. So once you have this, if you object a controller, we all need to create I, say a PV on your PVC and this Rab object and with the snapshots as a source, and then when this API barrage is created, then the snapshot controller will do their work to actually PV controller.

C

Do their work, supervision, the new volume from the snapshot, and then when that is available operator controller can go ahead to do the next step, so it will utilize some functionality.

C

Nor and I will music. Oh.

D

Well, one of the one of the tricks here with all these different operations is going to be. This has to be some sort of a status or state field on the PVC that gets changed that will lock out any other offers. So the two operations can't proceed in parallel if they conflict with each other.

D

So as the controller, that's just looking at these operations we'll have to try to update the PVC to some new state before doing anything and then set it back when it's done and you'll have to have a different States for every one of these operations.

C

And so that was you can check the annotation to see whether is allowed to something or if there's some operations pending. It is in some physicists, so operations on that. Well,.

D

The status of the operation could just be on the this operation CRT itself, but there needs to be some some some field that can get atomically updated on the PVC to to just show that it's it's being used by something. So no one else can grab it.

C

Right now, I can use annotation because that's the place.

C

Then, when this annotation may may has some special being to represent this, is handing during like you are restoring, is a pending phase.

C

B

C

Some of the operations, at least so this is very early like design and we probably need to go through Iran's.

C

We have some prototyping for restore, but it's very not complete, so it involves some changes in PV controller. Definitely, and also we have some external controller to handle something since we're already like at the end time- and we I think we have very good feedback and this proposal is public I. Think very. Not everyone came, they comments.

C

Yeah, it's always your feedback comments. Welcome.

C

Think about this proposal or concerns.

C

Sorry, maybe so any other like issues.

E

Shammi comment on this dog Jing: can you open it? So we can comment on it.

C

E

I wasn't able to calm a time, maybe.

C

Let me know if you cannot, because I changed to share to public last night. So just be me if you cannot thank you yeah. So now we have the PR for snapshots proposal. Please like comment it and this volume operation proposal and then based on the feedback we can set up. We will have another meeting next week as euro and to talk about.

C

Okay, I think that's the time to finish it. Thank you very much. Everyone and the see you next week, a nice result in this to 9 o'clock. If nothing changed, sorry about the delay notice, yeah, that's! Okay! Thank you. Thank.

B