Kubernetes Data Protection Working Group, 28 Jul 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes Data Protection WG - Bi-Weekly Meeting 20210728

Description

Kubernetes Data Protection WG - Bi-Weekly Meeting - 28 July 2021

Meeting Notes/Agenda: -

Find out more about the DP WG here: https://github.com/kubernetes/community/tree/master/wg-data-protection

Moderator: Xing Yang (VMware)

A

All right, hello, everyone today is july, 28th 2021. This is the kubernetes data protection meeting.

A

So I think today fong will go over the cpt design dock and after that we can see how the data protection white paper is going and after that we can have open issues. So if you have any topic you want to discuss, please add it to the agenda.

A

Okay. So now our phone is all yours. Let me make you a co-host, so you can share.

B

Okay, thank you.

B

Yeah we we, um it's actually been a while, since we talked about this cbt design, so let's just uh quickly walk through a little bit and then we can start sticking to the feedback that we got from the design from the community. And then we will see what we can talk more about.

B

So basically, from the beginning, we come up with the. uh It seems to be: there's a need in the community that we should have a way to back up more efficiently by instead of backup the entire value we just want to see. What's the difference between the current data in the volume and the previous time that we back up and the way we do, that is that we compare the this the difference between two snapshots, the current one and the one that we use to do, the last backup.

B

That's a differential snapshot between two snapshots and either it can be a file system or it can be a block volume, and we can just compare the change in these begin to end mark up the specific chain. Only if it is a block uh device, then we just you know back up these blocks that have changed now. This is similar to many vendors that offer out there like vmware.

B

They have us, cbt, uh aws also have incremental snapshots and and azure, and so and so on so forth, and we after we discussed about you, know that the different way of the vendor offer their um differential snapshots.

B

We come up with this um design that uh that we proposed to add it into the csi kubernetes csi that that outlines the um the way we present the difference between two snapshots, and this is one of the we want to focus right now.

B

We want to focus on just the block chain only and then later we will expand it into cover the file chain, as well, so with the blockchain, as you can see that um each of the requests uh for the when a service one when the apply and want to request for the difference between two snapshots.

B

It can send a a request similar to uh in this format here, and uh you can see that it have the snapshot id of two snapshots, the current one and the previous one and the volume id of the balloon that is under questions and then the stock offset and the maximum excuse.

A

Me uh phong: can you put this link to the agenda? Doc looks like the link is now right.

A

Yeah, it is it is that that link is our.

A

No yeah that that is actually the production group agenda.

B

A

With the wrong one yeah yeah, I was, I.

B

Opened it that way, it's like.

A

Opening the same arc, okay,.

B

So let me let me copy this link to every cup. Let me copy this link here: yeah, okay, so.

B

B

So all right, let me continue. uh I also let me paste that link in the chat as well. So the suitcase, let me double check that make sure apply to the right document: yeah, okay, so um the req, the the request of the service, will specify uh the snapshot, the two snapshot id and the volume and the start offset of that of that of that, um that's it and it can be from beginning or it can be somewhere in the middle.

B

um We will talk about that a little bit later and the number of maximum entry. I will also talk about that later too. So uh in the response in the response, uh it will be, it will, including the trade at the an array of chain blocks and the maximum maximum entry specified in the request. Here is the maximum number of chain blocks, as specified in the response, just in case that it has so many chain blocks.

B

We want to only return a thick set, a maximum, a certain number of obsessed at the time, so that the client side can absorb the response, and- and this is the um to specify uh the next obstacle that uh so, if it is like uh the beginning of the uh of the of the smith of the chain, then we can.

B

um The response would be uh zero, but if it is, then, if this one is um one of the response in multiple immutable response, then it will specify that the next object will specify the that the uh the object where you're going to start the next difference between the other, the next next okay. So this this net offset here will be used in the start offset for the next request.

B

That's what it meant- and uh this is the volume size and the time now, just in case that uh when we request a knockshot the servers, I might keep this different shortly memory for a specific number of times, so it can continue serving it.

B

Otherwise it will automatically come out and, as you can see, the chain blocks right here. The detail of eight chain blocks will be specified in this struct. It starts with the offset and start with the and then the size of that, so that this way we can accommodate both fixed side blocks and variable side block.

B

So that is the in general. Is the the overview of this api that we're proposing, and I we received various a lot of feedback in this area, and one of them that I want to point out is that, as you can see that in this um in this chain block, we identify the the block by the offset right, and this one might not work this one works with vmware cbt, but for iws it might not work because, as you can see in the aws, uh it has this uh concept of tokens.

B

You can see that, instead of just having uh by the way uh for for aws they're using a fixed size block, so uh when they specify the blocking that's, you can do that with the block file. You can go compute. The offset of the of the block, however, uh as you can see here, uh cbt also is using a concept of token so which is not compatible with the um with the vmware cbt, and you can see.

B

Vmware cbt only have the start offset and the lens right so um one I think someone here uh proposing that we, instead of just having in this same block, uh I think I think sandeep kumar and he, as you can see on the screen. He also suggested that we shoot at a an opaque field into the chain block so that in case the cvt api like aws, they want to add their own information there in this block.

B

They can do that. So, for example, I do not make a permanent chain. Yes, I want to bring it house, so so, let's just say something like this right, so so something like that. That's what the sandisk uh proposed so that we can.

B

So if the vendor wants to specify the block a little bit different, they can just simply add their identifier of the block into this contact here, uh so that in this case, if the vendor doesn't have any, uh if the vendor go with the standard that we use here, then they can ignore this context block, but if they, if they want to, you know, add additional contacts information into it, they can add it into this opaque field that that we have here. So that is one idea that sandy kumar proposed here.

B

Another idea that I saw another person asking is the best. That is not it's not this right here. This is the when we have two steps up, if we specify two snapshots, then the service will return the difference between two snapshots.

B

However, if uh what if the request only specifying one snapshot id, then what should be the return? uh One possible answer for that would be if the customer, with the request, send only one snapshot id, then the return would return. All the chain blocks would in that snapshot, that's equivalent to like a shame view.

B

This view block um this, uh even in your blog, so it so that is one one one one possible answer for that, but I want to uh leave the floor open so that we can start continue opinion here on these two uh points and then we all we can add more for asthma. Our opinion here.

A

B

A

Handle are those uh required or optional the those two fields. Do we make them required or are they optional, so you're saying that you can only you can specify just one, not the second one, the.

B

Other classes, I want to open right. I don't know uh from my point of view. uh My original was that my original talk was that most of these are required. If.

A

B

Only uh so if we only return, you know the snaps different between two snapshots. If you don't specify them, I'm gonna fail the request, but then you know another idea I heard from the community is that instead of failing that, then, if you have only one snapshot id then just return all the the block that chain all that disease in that uh snapshot. So I I don't know I I leave it. Is this.

A

Useful if it's everything's changed, it's still useful to get those two spots. If everything's changed anyway, you do a full back covering. So I'm not sure I just want to.

C

So I had brought this point up because when we do a when you do your first backup, you need you, because you don't have a previous snapshot, you would be backing up everything which is say the whole this size, which is, could be a lot of zero blocks.

C

So what you really need for the first backup is the allocated block path.

C

So my proposal here was, if you set the second snapshot id2 to ever to empty, then- and you specify you know the tracking id or whatever disk id in snapshot. One right uh then you'll get back just the allocated blocks of that snapshot. It's not a it's a difference now to the beginning. Point put it that way: okay,.

D

I like that idea, um can I ask a higher level question: um did this this api? um Where is it going to live like how is it going to like what is going to implement it and what? What is the communication path between backup, apps and whatever is implementing this api going to look like.

D

That, or or is that still like a future thing.

B

So we have some sample uh workflow. That's implementing uh this novel, implement that explore how to use this cbt service here, for example, right.

D

As you can see, yeah I mean I, I understand conceptually like how you would use this and how it how you could like even maybe implement examples showing how it's useful and that all makes sense, but like I'm, just trying to wrap my head around like the concrete implementation of this in a in a kubernetes world, and I didn't know if I thought about.

A

It I think we have a, uh we have the kubernetes api and also csi right, so there are both sides right. So this.

E

Is a this is more extension to csi itself.

A

E

Yeah: okay, okay,.

A

You have other suggestions, of course right. This is very early right, so if you have other suggestions, we're open to that right now, I think this is something.

D

Yeah, I'm just trying to get more of an embodiment in my mind of like how one would write this so so the the vision is new new csi grpcs, the csi plug-in implements them optionally and then some sidecar can hook onto it or maybe even the backup, app hooks onto it directly somehow and consumes this information.

A

Yeah uh fun: do we also have kubernetes apis? Do you find? I have not looked at this for a while? So I don't remember the stats. So I think we only have this honesty. Oh yeah, we haven't. We haven't got that point: okay, yeah. I think we probably we need that. I think because we need to look at like how how to call this right from a kubernetes side, car or something some controller who should be.

F

A

This yeah, we should also look into that.

F

That that might be really tricky.

D

A

D

Csi specifically, the way the grpcs work is the vendor controls. What sits, on the other end of that socket by picking a sidecar and running it in the same pod and wiring it up. So it's secure and everything but yeah once you want to plumb it out to some other application, then you have a bunch of security issues and data transport efficiency issues and no.

A

No we're not saying backup, I don't think we're saying backup when does controller they're actually like deployed together with this. I think we are still thinking about. uh We should have some kubernetes aka right. Okay,.

D

A

D

A

Yeah, I think that's what we're thinking but.

D

A

D

With that is, is you know most of the kubernetes apis are relatively lightweight in terms of how much data gets slept around, but like this one, even though you're only just doing a list of blocks like for a 100 terabyte volume, the change block list itself could be gigabytes in size and so pushing that through the kubernetes. Api might still be yeah.

B

That could be a yeah that could be a concern for yeah. So that's uh yeah, but that's why we we have this maximum maximum entry number here so that we kind of limit. So we don't want to.

A

uh We don't want to you know. This is probably not not unique right like when we have like list volumes list snapshots those type of things, uh although listener charts are not really used, but this list volumes is so that could be.

D

But but like list volumes is never.

A

D

Back into the kubernetes api, no.

A

It is called in kubernetes.

D

No, it's called like. We don't take the contents of that and shove it into cube api server ever right. We use it inside the sidecar to perform a computation, and so, if we had.

G

Something similar.

D

Going on we're, like whatever was cons on the other end of the grpc, socket seeing all these change blocks and it could consume right there. That makes sense to me if you turn around and try to push it into cube api server and then something else consumes it. On the other end of q, a cube api server, then you're, actually, q, api server is in the data path and you have a.

G

Rv pushing anything but.

D

I think we're just getting the.

G

Launch pad this is not even right right, but we're not saying the list of.

D

Data, the list of change blocks itself could be hundreds of megabytes in something this.

A

Is actually not okay, so I see a question for noah. This is actually still just the control pass, we're not really talking about the data yet right. Actually so we already initially, we talked about data won't be in this api. Yet so it's not in the past.

F

But but you know uh I, I think that this is still a concern. I kind of agree with that.

A

Yeah yeah pretty sure this will be a concern, so we need to certainly think about how to handle.

F

Yeah, I think this slight slightly different angle uh because of this paging thing right, uh we can put max entries within one. Let's say we have a api, a kubernetes api. Now, how do we represent this as a whole?.

D

Yeah yeah, I think a kubernetes object is limited by what ncd can handle, which is like two megabytes or something right before two years. Something like that yeah. So there's some relatively small limit that we'd have to fit each object inside.

A

So you probably need to yeah. I need to see like what is.

G

A

How what is the size that we could see? um I think that something definitely will be part of the change line. Doc. We need you.

F

Yeah, especially for the cases where uh one snapshot id is not there, the first snapshot you want to okay.

A

Yeah, then you have the entire book that.

F

One can be big right.

H

Yeah yeah, so don't we have to solve this problem anyway for the data path, because we no.

A

We have not we're not even looking at the data path yet right so right now, it's just uh we right now. Just look at the control paths. We stay the pass, that's another big discussion. I think.

D

Right but like the list of change blocks, itself is yeah.

A

This could be as well yeah.

D

Yeah, it's not the data, but it's still a large amount.

A

Right, at least, you need to write down, because this would be something that is you're actually required to write down this when we submit a cap or something like what is the image.

B

A

B

I have this this um answer here for this page of where we can, if we we have two options right number one is that uh we, if the natural id the second naturality, is no, we can simply scale the operation. That's one choice. The other choice is that, if, uh if that we know, then we return everything that change. Let me use in the in the first snapshot at the beginning, or the way we can get around that is.

B

We can depend on the vendor to support it right if the vendors, because each of these depend on the solid vendor that supports.

A

This but nobody you, but we are talking about, introduce this in kubernetes api, so the information will be in the edcd, so we can say this is only of anderson. We still need to uh know. What's the impact at least.

D

That's one proposal, but I think we should explore other options.

G

I guess yeah, what do we gain by exposing these and kubernetes, because this is, unlike you know, some of the other features that our users are facing. This is not so much easier facing. You know. There's no need for users to interact with these crs.

A

Oh well, this is, for I mean this is for tobacco vendors. I guess right.

G

So- and you know.

A

G

The fact that such a functionality exists doesn't really have to be exposed to users. You know the same way we do, like you know, snapshots or other functionality.

F

The backup vendors are also users.

G

Right but you know like.

F

G

Not like uh I mean like what what do you gain by having some corresponding objects in kubernetes.

I

Let's see, I think about what you gain by having some sort of kubernetes presence for this. I'm sorry, this is noah um is that you get a sort of standard way to talk to the api, and that is important um I would suggest so since it seems like having the like.

I

The full messages in kubernetes is sort of extra data and also might be more than std can handle what about having a kubernetes service. So you, you know it's exposed the way. Every community service is there's a standard way to figure out the ip address and talk to it, but a kubernetes service that you talk to, and then you call this api via, like http or grpc, or whatever yeah.

D

That's exactly what I was going to suggest.

I

D

Sidecar, the sidecar that talks to the grpc just has a rest interface out the front and the url that rest interface gets published in the kubernetes api.

D

But then the actual backup service can just go find that rest url and go consume it directly and then you're not pushing tens or hundreds of megabytes of data through cube api server. Every time you want to do a backup.

B

I like that idea. I like that idea.

I

Yeah, I think the other key thing would be that you'd have to require that this api endpoint uses kubernetes are back for authentication, um so it all sort of integrates like if you're running in a kubernetes pod. You should be able to look it up and call it and everything will work smoothly for you.

B

Go ahead, I meant in the white paper. I just spell out this new uh different snapshot, which is a service running in on the kubernetes cluster.

B

uh However, I do not uh discuss much about the whether it's being stored in the fc or scd or not, but the main idea is: we want to provide this different snapshot as a service in the kubernetes, and then it seems to play very well with what ben did mention here.

B

uh In in that, if that is the case, if we want to, uh we really agree that the cbt- this is a different snapshot, is the service and it then it doesn't matter how um how long that the uh the response will be right because uh it is between. At that point, all we need all the customers. The fire needs to do is to find out where the service address is and use this protocol to talk directly directly to the service to query the differential between two snapshots.

B

Then, let's focus on discussing whether this uh the what is the behavior when one of these contract id is not specified.

B

What do you guys think we should uh decide here whether just return all the chain block or fail? It said so far. We have two options right, so it seemed to me. It seemed to me that um the suggestion of if one of the snapchat id is missing, we decide all the locks that change. I think that seemed to be very useful.

D

Right, it would be useful if, if, if the vendor can support it, it might be the kind of thing where you have to have a capability bit and see if, if the machine can do that or not, and if not you just return an error and say you have to specify a base snapshot, but if it can, then then you get something I mean it feels like a nice optional thing that would be really useful in the cases when you can implement it. Okay, no. We.

C

D

C

Can I can I add, to the I mean this: I I've used this in vmware's thing and this corresponds to the query allocated block api. uh You actually cannot it's about that's a boundary condition for a differential backup.

C

You actually cannot do a a complete delta backup if you start off with a full dump of the entire disk, because what's your first backup going to be it's going to be, you know, if you don't have this feature, your first backup is going to have a lot of zero blocks for all the all the unallocated and unwritten blocks, because there's no way to query that so again, you know, I'm not saying you have to implement it this way, but you do have to address the starting, the very first backup problem of how do you just get the blocks so that you're only doing a change from nothing to the first set of blocks address that problem implementation doesn't matter yeah.

D

But I mean like that's something.

J

D

J

Have to deal with.

D

J

D

C

Is no change implementation right, so this is how we do it right now. We actually use query allocated block, so there has to be either another api or you you know make this api. Do that.

F

C

D

Waiting like, yes, you can just read the whole volume and then you can detect, which block.

A

So, okay, so, let's back up so since we have this effect, why is that? We can't make this one useful for the case that car is talking about right, so you can so the first snapshot id is required. The second one is uh optional: if it's not specified, then you know, then you just return all the change blocks.

K

I mean: is there a way- and maybe this is I I worry about optional? Is there a way to have sort of a specific? It's probably not fair, uh uh just to say what you need to specify some value to say yeah, so that I don't accidentally trigger something like this. That would because I agree with carl- I think it's a super useful thing to have. I just don't want to accidentally trigger it and and pull all the pull pull that just because I had a malformed request. I'd rather explicitly.

A

So so you phone, let's see if it's set to something like minus one, then that means.

F

A

F

We need to specify something well, we can discuss those details, but uh I'm more interested in the naming. Do we have any implication on the name, the sequence matters right.

A

Oh, you mean the.

F

G

A

Yeah it's a little weird yeah. Maybe we need a better.

F

If the first one is supposed to be the base, and the second one should be the next one, then I want to support this. I think we should make the second one required. The first one optional right.

C

Yeah yeah: we need to establish the temporal relationship between the two things, which is first, which is second in time.

F

Exactly and crosstalk for for many storage vendors, you take the first snapshot. The snapshot. Only the size of that snapshot will only be the size of the uh used capacity of that disk.

F

It typically doesn't contain the zero blocks you were mentioning, but you know it depends on the storage system.

I

Sorry so when you say that, if you, if you emit, say the satchel id1, the one that's earlier in time, you'll get all changed blocks um changed since when like changed, since it was a an image, I think and you.

H

I

All all non-zero blocks, okay, all non-zero blocks yeah. So.

A

You said, like initial, I don't know initial snapshot id or something.

L

Maybe like base snapshot id and target snapshot id for the two parameters target being the required one.

A

I

I also think you should: the api should be able to fail if, let's say, target was actually before base right. That's the expected.

A

How do you know the target? Is the body check the timestamp? No.

F

Storage system shouldn't know.

K

There could be a benefit, though, if you're, if you're looking to.

F

K

D

But if all you're returning is the list of change blocks, the order doesn't matter the the list.

A

Is identical yeah reverting over because you don't need change blocks right? We were just yeah.

I

I imagine the storage system yeah um for each snapshot holds like for each snapshot. The storage system has like a pointer to the the parent snapshot and then a list of blocks changed into the parent, so you just have to. In order to get the reverse list. You basically have to um walk the tree and, like reverse your your data structure,.

K

I mean yeah, I I guess I guess. I view that, as if the storage vendor you know said it was hard for them to do that reverse list. They could fail it, but I I don't know if I would operate.

A

I was just wondering the word case. I do you actually need to use the change blocks. For that I thought reverse just you have that specific snapshot. You just revert to it. It's a different function. I thought yeah.

K

Think of it more this way, so so let's say I had a schedule um where uh I do. I do sort of a. I don't know a scheduled sort of transfer uh every four hours um and then and then a customer comes in and says. Oh, you know. I took an application consistently.

G

K

Important point in time snapshot and I'd really like you to salt, that one away and and and the system goes- ah that's that's great other, except that you know that my new my base is now after that one you created before so if they want to manually, create it, I I want to be able to then sort of go from where I am now to that to that previous point in time, so so think of it as maybe scheduled versus manual.

K

There may be a reason to create a uh to get a copy of a manual one. That's maybe app consistent or or important in some way. So so again, I'm not saying it has to be supported, but if the storage vendor supported it, there's value in it you're.

A

Saying it's actually, some sort of vendors can actually just take. Those can actually remove those uh additional blocks. Yeah, I mean.

K

In the old snap mirror snap vault days, we would be able to do that, and it was exactly that that case that the customer.

M

K

To be able to to salt away, usually an app consistent copy uh that was created before the the most recent scheduled transfer. Then.

A

How does so so it's the storage system has to figure out. I mean that's, I'm just trying to figure out how. uh How does the controller handle this? You look at the the two snapshot.

A

uh Look, you have to compare the timestamp or do you already know this one is earlier, then I have to somehow do something differently. I'm just wondering how yeah how to.

D

I'm really I'm really confused by this too, like the difference between snapshots, a and b is identical, the difference between snapshots b and a and every case that I can imagine so.

K

That's that's true. That's true! As long as and again you know putting putting say if it's a snap mirror a data domain kind of replication, where I just have two points and times- and I diff that's true, and that's where I think the storage array wouldn't have any issue in either approach. There are other approaches, though, where I know they. You know the systems sort of you know build up.

K

uh You know sort of build up chains of changes that may be more difficult for them to do uh just just in terms of managing those chains. There's there's a couple of storage arrays that that I've seen that do snapshots really only move forward and and that backwards sort of here's the list of transfers is tricky for them. So I would agree with what stephen.

C

Said my experience is that you know mathematically. They are the same, I would agree, but you know if implementation-wise I've got failures from couple of different vendors, the apis, where uh you know I've made by mistake, I've made reverse calls and they fail.

D

Right but I mean we could we could write the grpc definition just say any two snapshots and then it's up to the it's up to the vendor, to go. Look at those two snapshots and sort them in temporal order and then compute whatever it is because again it doesn't it's not going to affect the return value. It's going to be the same. So as long as we just say, hey the user can support any, can supply any two snapshots, and just you figure out what the difference is and no guarantee what order.

C

They're going to be supplied.

D

C

I hear you, but that also puts a big onus on the vendor. To you know. Snapchat id I mean if it's as opaque as it is to good that puts the onus on the vendor to then look it up find its time stamp. Look right, do a whole bunch of stuff.

D

Yeah but like that should be way easier than the job of actually diffing them, so, like you're gonna have to do that work anyways! You may as well force the vendor to figure out which one is the older one, which one is the newer one in their particular representation and do whatever they need to do to pull that diff back out.

F

But ben to the storage render, this is the same thing right you, you hold all the information of your snapshot, sequences.

D

Yeah, I'm just saying that, like as far as the api definition is concerned, the order really doesn't matter you're going to get the same results either way. I can't imagine an implementation of destroyed system that can't figure out on its own what order they should be in and just reorder them if needed.

D

That that's my opinion. I mean if we want to make a hard and fast rule that you supply the older one first in the newer one. Second then yeah, I I.

F

Kind of I'm kind of thinking from the end user's perspective right who is going to use this api and what this is used for, if reverse order works for sure, as you said, the change block will not change but to the user. It's kind of useless if it's reversed and mine easily can make mistake based on the return result.

D

A

To get the same, they're going to get.

D

The same result.

A

For the this is for the backup underwriter, so I mean I I'm still have enough. If it's reversed, how does the backup vendor use this? Do they have to know the ordering, and then they.

K

Yeah so to some extent, then I mean again as a backup vendor. I guess I I like having the extra check where, where the system tells me you know again just in case I get it wrong. I guess I I mean I I guess I just I like apis. That are, I guess, more prescriptive, but that may be just a personal thing, but but um you know so to your question, though um yeah so so, basically again in in this use case, what I've done is.

K

Is I basically sort of build my my synthetic folds if you will off off of incremental snapshots, uh so so uh I might for example, then say: let's say I still have the um the previous. So let's say I have that previous base in my backup cloud.

K

If and then I have the new base, uh the the new version, I took what I, what I do need to know is: okay, the changes from the backup. I just took back to that previous point in time. Here's the blocks, I apply cool now. I have that that previous that previous point in time, so so similar to what ben's saying it's the same set of blocks, whether I was going forward or backwards.

K

So I just apply those blocks to my my backup image create a new synthetic and- and- and I just mark that as a previous point in time or uh or what have you so where it gets more interesting, then, is my next incremental backup. I do have to remember that my next incremental in my cloud is based on. You know that that other point in time this would be a lot easier if I had a diagram.

F

Student, the the tricky thing over here is right now in the csi specification. If you do at least snapshot, it gives you a list of snapshot ids, but it doesn't give you the snapshot, creation, timestamp right. uh This is from the csi from the storage system. It is gonna, be harder for the backup vendors to use this api. If this information is not exposed, I mean I'm totally fine if we want to expose the created timestamp in the from the storage system in the this snapshot api. But we're not doing this as of today.

D

A

D

A

I thought this snapshot returned timestamp yeah, because at.

D

E

A

Snapshot object right. Each snapshot object has the crate. Oh.

F

I see okay, if it's a snapshot object, then it's fine, I'm sorry! I thought it only returns the id yeah. You should have that. um Let me see uh listen quickly. Each.

A

List snapshot has a snapshot message itself. The inside the message has all the information, so you should have it.

B

So if we just put a comment into the api to specify the timing, like I just post on the screen right now, would that be enough to make it easier for users of this api to use the api.

B

G

B

The feeling that that, uh according to then like, if you flip the order, it would not make any difference in terms of the results and then, if you just specify the in the comment, you know the the use how to intend it is intentional to be used. Then.

G

B

It should be clear enough for the user of this api.

D

Yeah, but I think we could implement it either way as long as we write down clearly what what the expectation is of the caller and the colleague.

A

Yeah, maybe you need to agree if you can write it down, I'm still, not okay. I know.

C

A

C

Yeah, I'm still gonna interrupt on this one. You know mathematically it's the same, but how do we express the fact that blocks have disappeared? If we have a reverse snapshot, I mean yeah, that's the same negative.

A

C

I mean it's a you in 64.: you'd have to have a negative size over there, make it an n64, because if a block is going to go away right because you got the order wrong, you have to express that yeah.

A

I just still have not don't quite understand how how to use it. I mean, if maybe fine, if you can write down some details like regular, I mean normal case. uh You know how we are going to use na if it's reversed and how is the whoever's using this effect, you're supposed to use this.

A

I still not very.

D

Clear how if we, if we want to return, not only which blocks have changed but which blocks have been zeroed, then the order matters and we got to specify it.

K

Carl, it's not clear to me: you need to zero block. I mean the only time that becomes a problem is that the volume grew because because if I transferred a block, um that's fine right and then there's got to be some some some indirect block, some some metadata pointer that points at that block.

D

If someone did like an fs trim and a bunch of blocks just got zeroed you'd like your backup to shrink as a result, okay,.

K

Yeah yeah, okay yeah, so so that that's that's the only one is if I, if there was some sort of resizing operation, if there was a resizing.

C

Here right, that remains true, even even in the case of this, if there, if we, if the volume supports the trim, even the existing api, how do you say that we.

A

Don't yeah, we don't support that yet right so do we have two.

D

Not even not even a resize, but just like a hole, punch operation right if.

G

D

Bunch of data, and then you punched, a hole in it.

G

D

Know the the diff will tell you the block change, but if you read the block you're going to get a bunch of zeros and if you back up those zeros you're wasting space, so you it would be nice.

C

To know you don't need to know that the blocks have been removed.

K

I'm not sure you do. I mean I guess, but but it's only from a space consumption standpoint, because if the blocks got removed, that the metadata doesn't reference them and and given that we can everybody compresses. So all.

B

The case that the block being zero out can we specify in this chain block structure somehow.

C

Yeah, but is it necessary? That's the question stevens brought and brought up right because from a data perspective, would we well?

C

I think we would need to know right, because if one this is, I mean again not assuming any format or assumption of the data. If we were to read the raw volume, we would completely. We would get zeros in that area. If we were to create a synthetic full where we didn't have these zero entries, we wouldn't get zeros in that area, so we're wrong.

K

Well, yeah, but but but nothing references it I mean to me: it's.

C

That's assuming that's assuming any semantics on the data. Nothing references like.

K

A compile system.

C

Or something if it's just purely bytes, uh I say: okay, yeah, yeah, yeah, yeah,.

K

You're right, you're, right, okay, good point.

B

I was thinking from the back of vendor a point of view where we, after we received these chain blocks right. They made it. uh We want to do the next step, which is backup or restore these blocks, and if it is 0, we want to optimize that we want to. You know. Instead of transferring a block, that's always 0 we're going to do something instead.

B

So um I would love to see some guidance from this service saying this chain and it can form something to something some, it kind of some. It doesn't have the old data there, but they have some indication if it is zero block right.

B

So that is something that I I would love to have so that when we backup or restore we know hey, this is a zero block. Just don't do anything, just wipe it out. Instead of transfer the you know block over.

F

Right are you: are you looking to expose this kind of information in the api? I'm not sure whether api is the best place to do that.

B

I don't know I'm questioning here whether that's a good idea to have some additional indication of whether.

D

I think you could add one one additional bit to each change block that says you know. Was this a? Was this a hole that got punched? Therefore, you don't even need to bother reading it. You can just mark down that this is a hole now.

D

And furthermore, when you're restoring you back to the physical device, you don't have to write a bunch of zeros. You could just punch a hole for your writing operation and save or get more efficiency that way.

H

You're saying to avoid reading, because I think we're going to have a zero bit on the block read anyway. So what you're saying is you'd avoid the additional blockery that would return the.

D

You could avoid both the read and the right right like if there was data there in the old snapshot. Now it's all zeros and you were doing a restore you could instead of writing those zeros. You could punch the hole with a more efficient like scuzzy operation or something.

D

So so, knowing that it's that it's all zeros is handy.

H

Are there are the providers that only have the zero information on the when you read the block, because we could just add in both places- and you can actually set it in either way right it kind of matters. What the providers give us back.

K

B

My mic, a little bit- I have a little bit concerned here- is that rather zero block is associated.

D

With snapshot id 1 or snapshot id 2, well, yeah yeah. So so, if we do the zeroing thing, then the order matters that was the I forget who said it, but but that that's one case where you actually do need to know which one is older and which one is newer, because the zero bit would would indicate what you know that it was zero in one of them like the newer one in general.

D

So so, if you want.

M

To include the zero, but then we have to.

D

Get the order straight yep.

B

Yep, so if we specify this zero out bit, uh then we have to say that when this flap being set uh this the block, this block in snapshot id2 is zero out. That's what it exclusively like something like that right.

B

Otherwise, you're gonna go confuse you're gonna, confuse the consumer.

B

I, uh by the way, with just a time, check we, we have eight minutes left on the clock, so uh we're going to continue talking about this, or should we um wrap it up here and then we can continue at a later time.

B

I think this. This yellow outfit is a very good, um very good idea. It doesn't matter uh the vendor would exploit this feature or not. I think it should be nice. It would be nice if we put it in the api so that that if the vendor happened to have a huge voice, then we he would. She was.

F

Able to respond well, I would take actually the opposite approach. Typically, adding a field it's easier than remove one, uh I would suggest maybe check out whether this is actually doable in many storage systems right, because if you want to remove this filter, it's going to cause trouble.

D

But it but it's trivial to just leave the field blank in all cases and get correct. Behavior right. You.

G

D

G

Can't detect that you.

D

Have zeros you just say it change and- and I don't know if it's zeros, so it is false and then the user goes and reads that block and indeed there's all zeros and they still do the right thing.

F

This this will go into the csi specifications, so yeah technically I understand the use case over here. It's just a matter of you know we could. We could wrap it in a capability.

D

Bit like we do with other csi things we could say: do you have the capability to detect zeros? If true, then, this bit is valid. If false just ignore the val, the value of the bit.

F

Yeah, that's that's also. Fine again, you can add in one more field, brings more argument. Point a discussion points yeah.

D

F

D

Reason to get the order straight right so that you can get this field in the future. If we leave the field out and we don't care about the order, then adding it is a lot harder, because the order could be backwards.

F

D

F

It depends on how we define the api behavior right. If we say the behavior is that uh the first snapshot id has to be has to be before the second one. Then then the contractor is already fixed in the beginning, but yeah I see a point.

B

Is that that's okay, they put some contact like this right. Zero out is a boolean right. If you write two, then this block in this snapshot id2, I exclusively spelled out here zero out. That's okay, because in the previous one I explicitly saying that snapshot id2 is the this might be a little bit complex. Is that is a later in tab. So if we reverse the order, uh this one might not be good.

B

Let's see if it work, if we reverse the order.

H

So fong, I think we can probably answer that question by looking at the providers, because we have concrete examples of implementations and if there are any that require knowledge of ordering, then we should require ordering here and if, if there are some that don't, then they can still take the the ordered uh values right.

H

It seems to me like in this interface. We should we should just require ordering and we can make it explicit in the api names, because if there's any provider that requires ordering, then the the vendors, the backup vendors will have to put ordering down into when they call this api.

K

And I think, as we name it again, I would stay away from previous and current. I like I like bass and target, I think that's a clearer terminology earlier.

C

K

We'll see, but that implies time right so if we say base and target, then I'm assuming you you as the you as the caller, if you're calling for a previous point in time. That's fine because you've told me that's what your target is so so I like base and target, because it really is saying you know: here's where you're starting here's, where you're targeting and and ending the the storage vendor can say. Oh we don't support going back in time.

K

That's fine, but but but I you know, basin target makes it very clear what your intent is.

B

A

Okay, with that, I did, did you just enter the name just to change the name itself, so it's.

K

That's what I was thinking, it's like you know, snapchat based snapshot, target.

L

Yeah, it makes sense.

B

At some time it doesn't, I mean at some point it may not. uh If that is the case, then we don't even need to specify.

B

Right I mean the target anyway I can. I can. I can touch up the document later.

F

B

F

Is the base is optional.

K

The bass is the one going back to carl's. First point: yeah, the the bass is the.

M

K

Yeah and I'm fine now if bass is optional, because now I'm explicitly saying target and I'm not putting in bass, then then I know it wasn't a mistake. So.

F

Very good discussion and phone: could you remind me why we need a volume id again.

B

F

B

And also, I also forget that, does anyone remember.

D

It could conceivably be useful for finding the snapshot, but the semantics.

A

Ide is, uh I think, for some three system rewarding id is required like to you know, for for any operation. You already need both.

D

A

Id snapshot is not enough to identify.

D

But in kubern, but in kubernetes and csi, like we've, decided that snapshots are independent of volumes like you can in in principle, you can take a snapshot and then delete the volume and the snapshot still hangs around.

A

No, but this one is for your take. This is the request right, you're taking a you know, request we have the wording id. Why is that we don't need the.

D

Really like placements in a csi delete snapshot, call.

A

But this is the but this cpt. This is like when you're taking snapshot right. This isn't.

D

Right but you you can have a situation where you have two snapshots and no volume in csi.

J

D

It's perfectly legal, I mean it's weird but yeah.

A

Yeah, we actually apply that, so we actually have to do something to uh to insert that voting id there for two for our elite snapchat to work so for vmware yeah. It's required.

D

Like, like, typically what you would because snapshot id is an opaque string if.

G

D

System requires the volume id you just embed it in the snapshot id strings in some way that you can pull it back out.

A

uh Sorry, I think we are running out of time. Actually, okay yeah, you can add a question mark. We can come back to this one yeah but anyway. um Thank you very much, yeah! Okay! Thank you all right. uh I think. That's it right! So uh we're running out of time, but shenzhen do you have any update for the white people you do you want to just talk about that next time,.

F

Next time, okay.

A

All right, let's, let's get you next time, all right thanks everyone, great discussions.

I