Kubernetes Data Protection Working Group, 14 Jun 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes Data Protection WG - Bi-Weekly Meeting 2023-06-14

Description

Meeting of Kubernetes Data Protection WG - 14 June 2023

Meeting Notes/Agenda: -

Find out more about the DP WG here: https://github.com/kubernetes/community/tree/master/wg-data-protection

Moderator: Xing Yang (VMware)

A

Hello, everyone today is June 14 2023. This is the kubernetes data protection or in group meeting I think today uh we want to continue with our CPT update uh Yvonne. You want to give an update here.

B

Yeah I moved the um old pull request to um a new one. So this uh the the older one was still created from a fork um which we no longer have um permissions to change the collaborator settings and rewrite permissions.

B

So I just created a foreground from my GitHub account and then I, just uh granted like a column or sort like um push on permission.

B

Do it and in terms of the cap documents itself, I copy, all the the metadata I guess again from the previous uh cap to this one I added some like a really high level test uh plans in the in the test plan section.

B

um So I think that part would be um that part will continue to evolve. As more um you know, review feedback is um captured, also I included, like I, updated in the Alternatives um alternate solution sections like um all the previous prototypes that we have done around like uh aggregated APA, server, volume, populator um and also the you know the initial, like um those rest endpoint to hop on proposal and provide like um just a bit of information on why they were previously rejected.

B

I didn't go into too much details and whatever is relevant. I just provide links because there's so many um uh discussions um from those previous um proposals. I also updated the production, residence review um sections um just at least like them, those uh sub sections that are required for the alpha release.

B

So there are other more involved like um checklist like the the metrics, like um you know, all those things like they're not required until beta, so um yeah I think like also showing, like um those part, especially the test plan, Parts they're, always like uh not clear to me like what is how much detail we need to provide I. Just look back at some of the previous uh six storage caps.

B

uh I think most of them are pretty high level without going before being you know, boiled down into the details, so um you know just um yeah. If you have any thoughts, then just let me know how much to involve uh how much like your details, you include there.

A

Yeah I can review that uh so also go ahead.

B

I, um so I didn't include any of the proposal or no like um the slide deck um I'll, lift it up to column Prasad to determine um you know when it's the appropriate time to include those and how to convey and communicate those um uh details.

A

Oh, the design, details section basically, okay,.

B

Yeah yeah, okay,.

A

This is the placeholder okay, yeah.

B

This is the latest. The latest template yeah.

A

B

Okay, there's a big chunk uh yeah under design detail is supposed to be a big chunk in.

A

C

Before yeah I will not get started next week, I'm pretty busy this week, yep.

B

Yep and then also like the uh you know like um the cap, um MD was, is included as well. The production Readiness uh MD is included in as well in the pr so.

A

B

The prr review, where it might change it, used to be John um but, like uh I, think now they have more people at least I'm shadowing on in the group. So.

A

Yeah but I think he should still be the approver because he reviewed this one before nothing about what normally what he what they do. Is he probably will ask the shadow to review it first then he will be the one who still does the approve.

B

A

B

Because, like all of this is going to be our out of three like um so, hopefully it won't. There won't be too much back and forth.

A

Yeah, we can paint them later, I. Think after uh yeah this the designs.

B

Yeah yeah yeah, like yeah like I, think, like um you know, if we um so yeah I, think we chatted about this on slack I. Think the next step uh would be for us to share this with um the CSI at the CSI community meeting uh next week. So.

A

Yeah uh I think all right. Let's see if I have a link, I think okay, I put a link here. uh You actually should add a link there right now right. You attended that meeting before right, I.

B

Did yeah yeah so.

A

You, actually you see that there's already another okay.

B

Yes, another thing here.

A

Which means I, don't even know if we, hopefully we don't have time, but you you should just add earlier Topic in the queue right. If we can't get this meeting, then next meeting right.

B

A

Just can you add your name and your you know.

B

A

I I have this one in uh here yeah. So it's in the agenda. Doc, if you can, you know, add your name and then add the link of the CSI spec PR yeah. Is that updated or do you need to update that one I do.

B

C

There's I made us I've made a new revision for that. So you know so we can strip out the discussion some last time, I'll send.

A

Up the CSS background.

C

Or not the CSI spec one, but just the slides, no.

B

No yeah, we haven't updated.

A

This yes I was talking about this one because next week, next Wednesday there is a CSI CSI community meeting right. This is just happened like once a month, um so you guys should uh have this discussed over there. So add is like like similar to this one right at your at your name. Add this uh your cssback PR here.

B

A

We could discuss in this meeting because you do have to have that PR merge first before the cab can be merged.

B

Okay, okay, um I think, would uh I think it would make sense yeah so yeah. We can include that for sure.

A

Yeah but I think you should have your, but it's better to have the cap updated before before that time, but even if it's not I think it's still good to you know start talking about it. Yeah yeah, I.

B

Think um you know right now, like our current thinking is like at least like yeah, so we will include that, like uh the slide deck and the CSI spec PR, uh our current thing thinking is at least go through the this, uh the slide deck that Kyle has first, because if that changed, the CSI spec will also change right.

A

Yeah, that's what.

B

I'm asking yeah sure how much details.

A

uh We'll be changing and see if.

B

That's that yes, yes back, is a lower details. uh It has lower lower level details.

C

And yeah there's a section there on the CSI spec changes. Basically of you know: trpcs.

A

D

It possible to have like provisional cap approval pending CSI spec changes merging because because we have this, this uh yeah.

D

Well, this one too I mean we have this. The.

A

System relational I think provisional should be fine, I think yeah and it was a provisional. We.

D

Could I I feel like every time, there's a CSI spec proposal. People ask like well what are you? How is this going to be used? What is the cap, so we can go see what the higher level you know interface is going to be, but then we say well, we can't merge the cap until the CSI spec change is done, and so it's like we have this yeah.

D

You got to break the cycle at some point and get everyone just to sort of agree in principle and how the features should work and then say Okay step. One is getting the CSI work done and step two is getting the kubernetes work done.

A

D

And we got to break that Deadlock.

A

I think provisional should be fine if if we just merge the card, that's provisional, but if it says implementable, then we need to get this. uh You know the CSI side. uh Pr merge first right so.

D

That's okay, yeah.

A

D

There a mechanism for moving something from provisional to implementable in the middle of a cycle. If the right things happen,.

A

Merge that line says it should not change the status like that right. So that's like okay. That means yeah. So that's the thing.

B

Yeah um yeah, actually we got the anyways yeah, so um I would yeah I would include, like I'll update this. um The CSI community meeting um agenda right after this call. So okay, but I want to confirm that so um one thing um Colin saw. Will you folks be able to attend this next week because I'll, be um you know, time timing, wise, I'll, be out of office next week? uh Would you folks be able to attend.

C

Yeah, this is the same time this slot this time slot next week, right, yeah,.

A

It's the same time, yeah and then the meeting this will be the sixth storage meeting Zoom meeting link okay yeah, not not the data Protection One yeah.

C

Is it okay, though, that we present what we what uh I mean, essentially what we've gone over the design as opposed to the cap update? That's the main thing.

A

B

I think you can present the the the design without you know, I like like we discussed last week right if we get the cap, updated, that's great, but if we don't have time for it, then.

C

Yeah most of the presentation is about the kubernetes implementation right.

D

Yeah I I was going to say for the CSI meeting. You really want to focus on the CSI spec changes and try to view it through a a CO neutral lens. Yes, so uh I mean it helps to be able to point to you know: kubernetes is an example of one Co that will implement this, but you want to be able to present it in a way that it's not kubernetes, specific yeah.

A

So do you do you think I think they are planning to show those slides that have been showing in the data Protection One group, you think that's a good idea or you suggest something else.

B

Yeah, because a focusing won't have any idea.

A

D

Was gonna say we want to focus on what the csir PCS are like? That has to be the starting point.

C

Like I think a lot of the people in that meeting.

D

Will be the same people that come to these meetings? I, don't.

A

Know if there will be how.

D

Many people there will be that will not have seen this presentation uh in that meeting and in that meeting, what we're really concerned about is like just what are the implications for CSI? You know at that level. Yeah.

C

Yeah there are three slides on the changes to the RPC. I mean I, didn't Focus very much on them in the presentations in this meeting, but there are three slides there.

C

um I did not bother to put together anything about how the packaging changes and how, because that's specific to the implementation, but I don't know if those three slides will be enough to.

A

So maybe just to uh focus on it's basically shorten the those slides right. Maybe just uh the.

D

Summary, do you have a? Is there a I'm, sorry for not knowing this in advance? Is there already a PR that includes all of the spec changes that are proposed for this.

B

We have a sorry hang on a sec, we have a PR, but it was still um the old, the older um I guess uh we we have a yeah, we have a PR in the CSI repo, but it is still based on the former like um solution, but.

D

My recommendation, then, is get that PR in sync, with the new proposal and start from that. That should be the starting point of the presentation. Is here's the pr with the exact changes that we want to talk about and then flesh out? You know, and this is why it slides.

A

So, if that's the case, then folks may not have enough time to complete the right. Is that right, car I heard you are saying yeah.

C

I'm not this week, I'm pretty busy.

A

C

D

Yeah so so there may not be a bandwidth as Shane mentioned, there's another topic we have queued up for next.

A

Week, that's true yeah. It.

D

May be the next month before we actually get to this topic anyway. So you have time yeah.

C

Actually, you know, one thing we have not talked about at this meeting. Right is the proposal for the RPC. Essentially, there were two things which were left dangling in the original PR and that is do we do fixed Block versus extent based and all that type of stuff. So the proposal- the The grpc Proposal I, put out right in this series of slides, has kind of support for that.

C

But no one over here has debated that you know for our games just said it's stupid or whatever right, we should do really spend some time looking at it. Maybe 10 minutes.

D

C

Meeting! Yes, because now before.

A

C

A

You want to talk about that now. Then then, I.

C

Mean yeah sure I.

A

Will stop sharing? If you have anything you can pull out the slide.

C

From last meeting, uh whatever any of the slide links that hasn't changed yet.

A

Nice, which this side this one no.

C

No I'm not sure about this.

C

So I think it's uh the RPC proposal, so scroll number six onwards. This.

A

C

The next yeah, the next after that yeah yeah.

A

It might be easier that you have to share yourself. You know when to move all.

B

A

You have problems sharing I, make you co-hosts, because otherwise I don't know when you want to move things.

C

On my desktop right now, okay I'll try to bring it up hold on. Okay, stop.

A

Hey we'll do that maybe uh uh Ben, uh so you were asking about the you know in the middle changing from provisional to to like implementable in the mid like after the cap deadline right. But if we do that I think we have to go through the like release team lines like requesting for exception. If that's the case, then nothing that's fine. We could give that a try.

D

Okay, I, just I, know that it's like a four-month cycle and some people are trying.

A

To yeah, yeah I know deadlines.

D

And I I feel bad when people are told. Oh, you, gotta wait, wait for four months, I.

A

Know sometimes it's the appropriate answer, I think, but maybe largely the appropriate answer, but I think it's probably worth trying so um but.

D

I think we should. We should try to help people figure out like what order to do stuff in, because we tell people to go in circles and that's not so helpful. Very much appreciated.

C

Okay, uh so is this is so in this. This is the CSI part of the spec right.

B

uh And in this is how we are seeing your slack instead of your oh.

C

My God, you see my slack oh gosh,.

D

Yeah you're sharing some other desktop.

C

Yeah this is my.

C

This is a zoom problem. It's a me problem actually, but yeah. Okay, this oh yeah I was like, let's see, yep.

C

All right, I hope you can see the proposal now.

B

C

Yep, okay, great um so um so in this one we we had proposed, uh adding uh snapshot metadata service to the to the spec dot MD, and in that snapshot, metadata service there'd be two rpcs uh get allocated and get Delta uh the first one being to return metadata on the allocated blocks of a snapshot and the second to get the changes between two snapshots. The first one would be used typically for a full backup and the second one would be used for incrementals.

C

uh These are I, think relatively straightforward. It's this one that starts getting. uh You know where we have a lot of stuff. This is trying to call out the. What does? What does the sorry? Let me go back once this one takes, uh you know and get allocated requests the first one or get Delta requests the second one and both return uh streams of data. Once the street will get allocated responses. You know, get Delta response, um and this is the grpc stream stuff. We talked about so um Delta request. Right um is.

C

We know these arguments from the discussions. It's like opaque session tokens, you know identities or volumes and snapshots Etc a starting offset a bike offset right and maximum results. It's the return stuff that we, you know, which I think we need to talk about. uh So this is returning a stream of uh get Delta response, and in this uh that's the last thing down here.

D

Just before we before we skip over that the starting byte offset this is this: is our pagination mechanism.

C

No, no! No, but sorry by the pagination mechanism is basically this. It returns a stream and uh the stream can contain Mac. You know n results in one and that the Max results would be kind of the pagination thing um starting by the offset is so that you can you. If you want to right, you can start uh recovering stuff. Only after a particular, you know point in the object, you're targeting.

D

Well, yeah, that's what I'm getting at is suppose, there's a suppose, there's a half, a million uh blocks that have changed and so you're slapping that over the grpc connection and someone reboots you after a hundred thousand of them, yes, and and now you want to like pick up where you left off like are you meant to use the starting bite offset to figure out where you were so, you can continue where you left off. Yes,.

C

That is the intent, so so that is that sounds like pagination to me: oh okay, if you okay, first semantics yeah, but that's the intent, correct, okay and you can control how many results come back in each each callback right for Max results and I guess the server side. This is client side.

C

Of course, the server side will of course, put a cap on Max results if it wants to so um so what what you get back in the Callback is the is the get get Delta response, which is the last structure down here and uh in this one right I was, you know there was I was looking at some of the original documents for right and um it wasn't very clear whether we settled on uh extent based or Block Base, so but but I think the original document had both so I kind of copied the data types from there, uh so the data type essentially is.

C

Is it fixed length or is it variable length? All right, fixed length will be a by block and variable length will be by extent and this kind of matches I.

C

Think what we see in a lot of the low level vendor stuff- you know like AWS EBS- has got a block based and VMware has got extent based right so anyway, so uh it would come back with the with the metadata type and then we we need to know the volume size bytes and what you have to put it into every one of these things or have a separate RPC, which makes no sense, uh but that's only one one integer here and then this is the repeated block metadata and in the block metadata it would have the uh you know this data structure here, which is a byte offset for this piece of data of metadata, the size of the metadata and some vendor specific field.

C

uh This would address. You know things like AWS required to send a block token. If it wanted to it's none of the business of this API. It would say what the vendor wants to wants to have, but you know well.

D

I was going to say this. This will be a huge problem in the CSI. Spec is having something that's vendor specific like right, then.

C

What would you suggest.

D

Not having it would be, the food.

C

Then then that's fine, that's fine! But but let's say let's say: I wanted to use this API to get you know to get stuff back uh then I have no I mean we don't have a companion API to go with this right, which uh would recover the blocks in the kubernetes space. We don't have a problem.

C

We would Mount the volume of create volume, a snapshot and mount it and return data back whether or not they're sufficient for the infrastructure right, but um so we could see the blocks, but otherwise we have no companion API to get it. So if I look at the Spectrum of apis right, the EBS ones, for example, give a block token and I need that block token to recover the the block through that to the network. Api.

D

I I want to understand this you're saying in at least one implementation like there's an optimization that can be performed if you have per block I guess this, this vendor-specific metadata and then as a different API, where you can just get the the actual block without having to mount the volume and read the read the blockers and that's somehow faster. Is that what I'm hearing that's right, I.

C

Mean not just Mount the volume it's create a snap, create a volume from the snapshot and then Mount that volume okay. So there, the creative volume from a snapshot depending on the infrastructure can be cheap or expensive operation right and beyond that right, depending on the API. The context of that call right so on the EBS API, for example, and I, don't know what's behind that token, they return a token I'm, assuming it's got security and other connotations it expires and you've got to give that token in the call to get the block so.

D

Right right right, so so I I totally get why like on EBS. This is a better way like we have to build things that are portable and so like. There is no way that that's portable right so.

C

uh Well, I, don't know if it's not portable, it's it's opaque.

D

Right but I'm saying that the consumer of the API would have to then know that, like oh, this is a special Amazon thing.

C

On Amazon right.

D

And there's that's just not portable, so.

C

Yeah and that's why I want to discuss this stuff here? Okay, because we never have a town school or discuss this so yeah.

D

Well, I guess my my thinking is like if, if we think that is a problem, that's worth solving then like. We should also solve that problem in a portable manner like having a having a data plane level version of this. Exactly because everything that we've everything we've talked about has been metadata. Yes,.

C

Right we're in.

D

The metadata path and- and we think we can do that in a portable way, which is good but like this, this just isn't portable and like it, and if, if we think that the other that the portable way of doing it is so bad as to be unpalatable, then that cries out for trying to standardize the data path too.

A

What do we do with the sorority class? So have some class that have those opaque things here and there would that work.

D

I, don't think so because, because here like with the storage class parameters, it's the SP that has the magic right and- and there are many many different SPS with different magic. You know with different parameters that mean things, but the clients are all the same right. The client doesn't need to know. What's going on here, the client of the API would have to know oh I. It would have to know two things.

D

One I'm talking to Amazon or EBS and and two I need to have a special code to take that token out and go call a different API than what I normally do. So the client would have a whole separate branch of logic. You know if, if EBS do the good thing, otherwise do the fallback thing.

A

Within one snapshot class.

A

Snapshot class can hide this information. Would that work.

D

No, because this is the direction is, is backwards right, like snapshot, class stuff goes from the co into the SP. This is stuff coming from the SP back to the CEO. Now the CEO has like processing uh vendor-specific stuff and, and we can't write down what it is that that that's the fundamental problem is, if you.

C

A

Write a sentence that clearly.

D

Says what what is in that? But.

A

I was I was wondering if this is something that can be set up by an admin ahead of time register, then that kind of he didn't know kind of.

D

But but I mean the the problem.

A

D

Problem is if, if, if other vendors start doing this too you're just gonna have a giant if then else tree somewhere in the code that says you know, if Amazon do this, if Azure do that, if NetApp do the third thing and it's it's and that that's exactly what we're trying to avoid by having a standards, because so you can write it once and it just works. So so this is.

D

This is taking you in the wrong direction to have something that is vendor specific, coming backwards, where you you're, going to end up having different logic, depending on who you're talking to yeah.

C

Yeah, so the thing would be if there were, if there was a companion uh uh session service right, a companion service, to get to get blocks, then that the idea was that uh this block metadata object right. This message could be given to get that block, and so you just take this data structure opaque and give call that thing and get the block back.

B

uh Carl, like I, think even if we have a companion API somehow like the the back end was still. You know, still need to convey this information back to the client first right before the client can call the companion. Oh.

C

Yeah yeah I mean it's the it's the backup application that decides whether it wants to fetch the block or not right.

B

D

B

Still dependent on like this vendor specific things to get back into the coin, yeah.

C

D

Right, yes, so so I I guess what I would say is that there's nothing preventing someone from writing like an EBS, specific uh backup thing that talks to Amazon and gets the gets, the right information and talks directly to the appropriate apis and does a specific implementation for Amazon right that that so you can. You can write that today, if you want to it's great but like CSI, is about writing the the API that everyone can use right? That's that's why we have it so that you can right.

D

If you implement the CSI spec, it will just work with everybody. Oh this.

C

Is fine, that's fine! That's fine, maybe I! Think then you know we can get. We can let the AWS guys come back with some proposal right. If they, you know, let's say we remove this vendor-specific field. Correct, yes, and so that means right uh essentially from a kubernetes perspective, and we know what we have to we have to do. You know volume from snapshot and then seek and get the data if if they were a future API right, then that future API has to address all the aspects of how you get the context back.

D

Right I mean yeah you're, talking about like a data plane API to actually get the bytes of the diff. That's right, yeah yeah, and we should build that. You know if there's enough interest on both the vendor side or the storage vendor side and the back of vendor side to have a standard way of doing that. That would be great.

C

From a backup vendor side, there's a lot of interest: okay,.

D

I'm sure, but but like big it's, the problem is, of course getting everyone to agree that that is correct.

C

Yeah, by the way, the implementation proposal on the kubernetes side, the session stuff we set up- would work for that that, for a data plane API too, okay, there's no change. That same thing would work.

D

Okay, yeah well, we'll have to think about. Then. Is there a now I, don't know if you wouldn't have tied the two proposals together like this, because this proposal so much time and effort has gone into it and it feels like we are at a point where, like it was at least something everyone could agree to that you could, in principle, write a portable implementation of that. Would you know it might not be as good as directly pulling the blocks out of the API, but it would still work um and so like.

D

We could do that and then have a completely separate follow-on effort to add the data API or we could try to expand this to do both and that's that's kind of up to you. I.

C

D

To to do to to get this one done and then go on to the next problem, yeah.

A

Other than that, well, we actually be able to do standardize. The data apis, though, because that's not CSI normally, it's just for the control link.

A

Yeah, maybe think about that later after we got this one yeah.

B

Yeah just tacking on to that very quickly um again right people just keep asking us what can I do with all this metadata if I can continue to use CSI to do the pull data, so there might be a case that we can make there to say Hey. You know this is this. Cap is half of the story. We still somehow need to have a story for the other half of the data path. Yeah.

C

Because this this kind of opens the door right once we say once we can get acceptance that we can use the grpc call to bypass. You know the CEO to get data. Then you know, then we have a an Avenue to say. Yes, this is an efficient weaponism of getting data blocks back.

D

Well, it might not be grpc right, I mean CSI is going to be grpc, but this standardized way of actually getting the box. Maybe we would look at it and say: grpc is the wrong transport or maybe not I, think that's a that's a decision we would have to look at when we were designing a data. Api is what it. What is the right transport for moving, potentially large amounts of data.

B

B

Yeah, so for for what is worth like um in the current like um CSI spec PR, we uh we basically we specifically uh so under like the equivalence of block metadata right. We specifically added, like a a token property there to capture that um the kind of data movement where the back end gives you back a token and you use it to fetch that they have logs the Assumption there that we make is like we try to limit like the data movement to to I guess strategies. One is that you, you restore the PVC.

B

You mount on some sort of data move apart and you read the offset and bytes from there and then the other one is like this uh that we are trying to solve here as like. Oh the back and give us a token and it's a more efficient way to download those uh blocks.

B

So um we just say: hey, you know, here's a here's, a token feel that if you have it, you put it in there uh at least it's I, don't know if it is any better, but at least it's the user of it, the consumer of it will say: oh yeah, it is a token, then you know um that each token Niche type of thing that I know I can use for data, have movement but kind of what we're talking about here.

B

If we just remove the vendor specific stuff, that's fine, as well until like a later uh time where we.

C

B

More inputs on that the.

C

Semantics are horrible, I mean I, agree with Ben. You know the semantics become horrible with this vendor specific, because not only are we having this vendor specific we're also saying we have support for click versus variable right. So, for example, if it's an extent based thing, there's no way to specify a block, a per block token Etc that were required by an API right. So it's it's just weird semantics. So maybe it's pull.

D

It out I I was I was also going to ask why. Why include both fixed length and variable length, API.

C

That's why I said we need to talk, because, because in the informs original proposals they have this type of data structure fixed and variable. There are two streams of thought here. Right I mean if there's a lot of change and let's you know, um for example, if the entire, what you know three-fourths of the volume would change, and it's contiguous, you know an extent based thing- is amazingly efficient in expressing that change um on the flip side right, uh you know, block based, says: okay, these are literally the blocks. I want uh so.

D

Is it the case that you can convert between them in theory, and it's just a question of which one's more more efficient you.

C

Can absolutely convert between them and as a backup vendor we convert between things right. We talk to different subsystems, we get, we get change blocks, we get extends from sub subsystems and we get blocks from others and yeah. We convert because well.

D

Then then, it sounds like this. This design leaves it up to the SP which one they want to use and the SP is in a position to know what's more efficient on their system right and so as long as you can seamlessly convert between them on the co side,.

C

uh So the question is: should we put a stake in the ground and say no, everyone returns blocks, so everyone returns extends that's that's what we you know. That's what I want to ask over here. Well,.

D

I I think if, if neither one is clearly better and it's going to be circumstantial and it's not a big deal to force okay, because because by including both of them, you're saying every backup. Software must support both, but each SP can pick and choose whether to support one or the other or both yeah right and and that's that's not terrible from where I'm sitting right, because the SP might say. Oh I have an extent based system, I'm always going to do extents.

D

Another SP might say: I have a block based system, I'm, always going to do blogs and a third one might say: I'm going to have a heuristic to decide what's more efficient on the fly, yeah um yeah.

C

And that's what I think this is like a compromise API and by itself it's fine! Only if you start tying it with a data, retrieval API, that's when you could run it travel, but again that will function. You can fund that issue for now, because you know we don't have such an API. Yes,.

B

Yes, I think like historically right like um when found first um brought this up, the fixing versus variable line proposal um I. Think if I remember correctly, the conclusion was that we decided to park it aside for now um yeah, it's like an optimization, but there are some like you know, as we already surfaced here, there's some underlying assumptions we back then we just decided that it's something that could be added if needed, type of thing. Yeah.

C

Okay, so um this vendor specific field from The Proposal right um just just to note that you know so all the parameters here are going to be expressed from the client makes a call to a server. The client has to provide all the parameters in the in the namespace in the nomenclature of the server okay. So, for example, in the kubernetes case, the backup application which kubernetes application is talking in terms of kubernetes volume, snapshots and kubernetes uh um volume, IDs Etc and session token.

C

But when the when the site car talks to the SP right, um the sidecar has to translate to the SPs language, which was handles Etc, so I mean I, don't know if you recall that from The Proposal, but that was that was part of what that is what that meant.

D

Okay, okay, that all makes that makes sense. I mean that's what we always do. um Why I just wanted to understand the structure of that message? If you can go back one more time, so the you get a the response.

D

Has a meta has a single metadata type for the whole response. The volume size bytes is the size of the whole volume right right and then you have a, and then you have a an array of these actual metadata blocks, each one of which is just two un64s an offset and a size correct, um oh and I. Guess the thing that stinks is: if it's fixed length, the size will always be the same. It'll be wasting yeah.

C

After your bandwidth, I know I agonized over that and then I said you know, screw it it's a few blocks, but a few bytes wasted. This is all metadata, never mind. You can't have your cake and eat it too. Okay. So the thing is this: if you go back to this right, the way grpc works, it doesn't allow us to return um fixed Properties Plus, a stream. It's either. You return a stream or you return a set of properties, okay, so I'm forced to return in that stream right everything.

C

So I said you know it's not worth agonizing over this. It's just a few bytes right and.

D

C

D

Was gonna say if you did want to agonize over it and and optimize the heck out of it you what you would end up with I think is two completely separate: rpcs like one for one for fixed length, the one for variable length and then you'd have a capabilities API, where the client could query, which one is supported. So it knows which one to call, but.

C

Then the problem is so fast that it's my boss.

D

This is how everything works in in regular CSI right. We have capabilities, it says this is what I do, or this is what I don't do?

D

um It's not that bad um and if I'm just trying to think through, like if you, if you always have to specify the the block size when you're doing fixed length, you lose the whole benefit of doing fixed slang, and you may as well just always do variable length right that that's that's the downside to this thing, flavor of doing it this way is, is uh the only benefit that fixed length could possibly have of a variable length. Is the ability to emit the size with every single block?

D

And if you don't emit the size, then you haven't gained anything, and you may as well always do variable length so that that suggests to me that we should just make this an extent based API and say you can smuggle in fixed length by having every variable length. One have the same size right well,.

C

That poses that poses a problem for um uh API consumers, a client, because we don't want it changing Midstream right. If we just let it be I mean we don't have. Is there a facet, understanding or specification saying that you can't change it from one block metadata to the next right.

D

Okay, so so I guess I want to understand this. So the worry is if we just omitted the the metadata type, and this had an array of block metadatas with offsets and sizes right and like and and most of them were 4K, but then one of them was 8K like that. Somehow causes an issue for a client.

C

um You know I mean it could cause an issue, but it certainly is smooth. It's easier. I would say it's more often it's easier on the client. If, if we knew, if one knows up front what it's what it is, but no it wouldn't it's not that much of an issue if you're gonna, if you're, going to consider everything an extent based and clearly fixed length, it's an accent based right. It's it's easy, so in that case you know the client I would assume would maintain something like a bitmap or whatever right and.

D

Okay, so I'm beginning to see now so so what we're saying is it really is an array of variable length blocks, but if you specify that the type is fixed length in the response, that's just a hint that basically says hey your block length is always going to be the same, yeah and I'm going to send it to you over and over yeah um all right. All.

A

Right: okay, okay, I.

D

I withdraw my complaint. This makes perfect sense.

B

I think part of it was also like the reason why we have bite size or size size um bite.

B

There was um previously in the one of our really old prototypes, where we try to do like an end-to-end from like CPT metadata all the way down to like reading and seeking like um the blogs like we use this as the input parameters into like a tool like DD and said you know hey now now that you have the metadata go, and you know find this off-site right, and it is this long on this and fetch it and do something with it, so that I think that prototype kind of have some influence.

B

Essentially, we use this to to to to stick um the blocks on this when we restore the volume, so I think that prototype kind of influenced like um at least previously I, don't know about um this case, but previously it influenced our thinking based on how the data was being pulled out or sick out of the block. The volume okay.

A

D

We can continue on to the next page as far as I'm concerned.

C

Okay, I mean the next page is pretty much the same thing. It's just that this is get. This is the other. The first apis get allocated request. It just doesn't have two snapshots, so the response is the same.

B

So sorry go ahead, so are we expect it's like essentially right based on our current design? There are two um grpc calls. One is between the backup software and you know the CSI driver and in between the CSI driver and the I, guess the CSI Plug-In or whatever. So are we expecting to use the same? Just PC spec for both colors.

C

Absolutely I mean you recall the speaker right. uh This API right between the client and the sidecar is the RPC, but between the sidecar and the SP is the same grpc. It's just that it's now over the Unix domain socket.

B

C

And different parameters: yes, a different translated parameters right, the namespace has changed.

B

Okay, I think, like um it's just uh I, think previously again, we don't have to use soft or addresses right now, but just previously, like uh one of the feedbacks that I received from James um the fornic is that, like um the currency as I expect like um its version in such a way that you know it, um it doesn't expect like um an external consumer and not saying that we can't, but he just said, like yeah, you know something that you know when the discussion comes like the versioning would be how we're gonna version this RP's grpc is going to be.

B

It will be something that will have quite a bit of a discussion um on how we version it, because currently, the way we position it in a unique socket call like Paradigm is going to be different when you expose it to the external client right. So again, it's just just a you know for it. That's why I think for all of us here.

C

I, don't know how your version grpcs.

D

C

I was gonna.

D

Say I I I'm confused, because we we have versioning solved pretty much for for CSI and the way we handle grpc. You know we. We have a mechanism for introducing new new uh fields and new new methods, and we have an alpha mechanism so that if we introduce something that we later decide, we don't like and get rid of it, but once it's approved you can never get rid of it.

D

um So I'm not sure what the versioning concern is like this proposal. If accepted, would there's no intent for it to immediately change right? It would you could implement it and it could stay the same forever in principle,.

C

Yeah, my understanding of versioning and the CSI is that you can add additional rpcs to uh a service and add capability with saying these new rpcs exist. But you know by that capability bit and uh that's the extent of change. You cannot these get cast in concrete once they accept it.

B

Right I think like um yeah, that that versioning.

C

Other than with one with one caveat these data structures here the data structures can change I. Think if we do something correctly right up front and say put some, you know a version type fee on the top, then you can add additional Fields below you know to the end, but the order, order and existence of these properties right cannot change. But let's say if I wanted. To put another thing: I could put a number seven over here after match results and extend the request right.

D

B

D

The the requirement is uh a recipe that ignored parameter. 7 would still have to function correctly, even if parameter 7 was specified.

B

Right I think, like um the the the discussion around the grpc version, it was brought up in the context of like what does alpha mean right so right now the alpha is like the definition of alpha is, like you know, between like um it's like it, communicate like a certain semantic between the CSI driver and the SP right. But now, if you exploit a certain external client, that's that Alpha, that's that's the semantic of the alpha and beta or whatever, still carry the same like meaning behind it.

D

Alpha means all Alpha means is that we reserve the right to yank it out, so don't depend on it like we won't literally break something that depends on it, but we might remove it from the spec and people will stop using it.

D

That's all it means it's just a mechanical way to make sure that, like if we, if we do yank something and then later we introduce a new feature that it won't cause horrible things to happen, and there is, there is no concept of beta. It goes it's Alpha and ga that's how it works on the CSI Spike um I wanted to get back to the to the slide you were showing about the get allocated. I just want to understand. This is basically the equivalent of get Delta compared to an all zero snapshot.

D

C

D

Okay, so I'm curious why it needs a whole separate RPC. You can't just oh.

C

We we could I mean I over here. Essentially, it comes down to a base snapshot being specified, but no Target.

D

You mean a Target, but no.

C

Phase or Target, but no base right, I mean essentially or whatever just whatever you want. There's only one there's only.

D

One basically saying what is what is not zeros and.

C

Then it tells you yeah.

D

C

So I mean if you look at if you look at uh uh I, I confessions to some extent right, I was looking at vendor apis and when the apis have, you know, have these. You know this type of variants because sometimes in essence, the get allocated request is actually not necessarily talking about a snapshot, but it could be a snapshot, but actually it is a snapshot. That's true I mean yeah. The f apis I looked at VMware, apis I think even even EBS apis is looking at those right.

C

They all have this equivalent that you know it's a it's. What's in the snapshot, what are the non-zeros on this snapshot?.

D

It just it just seems to me that you could reuse the other RPC and say if, if base snapshot is the empty string, then interpret it as a comparison against the all zeros you're.

C

Good, that's what that's why you know I put this out here to be you know, so: dot set right, yeah yeah, but.

D

um I I guess we would have to ask the question: will every implementation support both ways of doing it, support both the get Delta and get allocated, or will there be some implementations that can do one, but not the other, and do we want to support those actually.

C

I I, don't think it will help anybody if you do one and not the other, because seriously as a backup vendor how in the world am I supposed to do an optimized anything if I thought for the first for the First full backup, I'm gonna have to read the entire volume. Okay I mean that's crazy. I need to know what I need to know. Only what blocks were written to the volume.

D

Yeah no I I agree it kind of makes sense. So so then, then we should just yeah say they're, both mandatory or ID, you implement them both or neither yeah and uh and and I. It makes that's even more of an argument for smashing them together into one RPC. That just does both things. You.

C

D

C

It's just clear in terms of documentation and reading, but you know if you want to have one I: it's not a problem. As you said, you know, base is zero. Well, bass is an empty string here.

D

Yeah I I, just I'm thinking in terms of future Evolutions, we'll probably affect both and it's. If you only have one it's easier to evolve it. Okay,.

C

For discussion, because.

D

You see right the data.

C

Structures are the same in terms of the response.

D

Can you briefly explain what stream means in grpc, because I don't happen to know that yeah.

C

Actually, uh do we have, let's see the link to uh prasad's uh reaction we had. This is the code we showed last time.

C

And um Prasad this followed me to the uh you want to see the client or service okay yeah climb. Let's start with the client, which is the client this one yeah yeah in CMD, there should be okay.

C

Okay, so in this client we have any grpc setup sessions with session object, get session status, get change blocks. Okay, so here here is the call to get Delta. Okay, this this guy gets back a stream.

C

Then it goes into a loop doing a stream receive each each data structure returned in that Loop right is a response which is, which is that data structure I showed you, and so this just reads from the Stream: does that make sense? So the stream comes back like this.

D

At the Go line level, this makes sense, I'm just trying to think through, like the security implications and what what actually happens on the wire. Oh.

C

Over the wire I believe it uses the same. um If you think about the transport whatever Scholars made over here, it uses pocket to get the back.

D

Right but I I'm thinking, if you like, error, handling like what, if you get you know through it through a bunch of messages, and then an error occurs like how. How does the client interpret that error? You know, there's I, don't think we use stream anywhere else in CSI, so I'm just I.

C

D

To think through what.

C

The applications are right, so the grpc spec there was a lot about streaming. Let's see we'll have that out somewhere, I, don't I think it closed it. A long time ago, yeah there was yeah, they had examples of streaming. So it's it's like I I, don't know if there's anything special that I can recall. It's like any any protocol where you're reading you know blocks through stuff. The client has it. The server has its own state right and the client will have to retry or something based on its state.

C

So now you notice when, when it's doing, uh let's go to the client side for this, because when it's doing the server side for this, but the server is sending data um server has a loop itself.

A

Sorry, one minute of warning.

D

Yeah yeah I I didn't want to go down the rabbit hole in this because we don't have the time to but hey I, just I would say: CSI doesn't fully take advantage of all the grpc features as sort of a subset, and this would expand that subset and maybe that's fine but we'd have to think through the implications of it.

D

um If there was a way to like get the same effect without leveraging to stream semantic. Maybe we would consider doing that instead.

C

um Actually, I think it would be a different proposal. I mean the whole. The the whole reason we were able to do. What we did was because of the stream, so it would be a totally different type of proposal, I think, to keep and require.

D

More, maybe we couldn't just have like a chain of calls where each call returned an array, a variable size, but.

C

We'd have to keep State and have a pagination token right. We should return back things like that. Okay,.

A

C

A

Okay, the hour.

D

Okay, yeah all right. Thank you, Shane! Sorry, no.

A

Thank you, so, uh okay, so we tried to bring this up in the next uh CSI community meeting. But if not that's something right, I think we're not ready. Are we ready.

D

Absolutely absolutely put it on the agenda.

A

All right, yeah put it there, yeah yeah I.

C

Will remove that vendor specific thing.

D

Oh yeah, thank you that that would be a big sticking point. Yeah.

A

All right, thank you. Everyone, hey bye,.

B