Kubernetes Data Protection Working Group, 21 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes Data Protection WG Bi-Weekly Meeting 20210421

Description

Kubernetes Data Protection WG Bi-Weekly Meeting - 21 April 2021

Meeting Notes/Agenda: -

Find out more about the WG here: https://github.com/kubernetes/community/tree/master/wg-data-protection

Moderator: Xiangqian Yu (Google)

A

Good uh started today is a april 21st wednesday, tuesday. uh This is the kubernetes data protection working group meeting. Today we have a couple topics to go through uh the first one is that I recently found uh interesting discussion on reddit.

A

I think some of this some of the members within this group were on that reddit thread as well.

A

Basically, tldr is backup solutions or simple backup solutions in kubernetes context to provide data protection. It's one of the top wounded items. uh That's one message. The other message is that existing tools like in kubernetes.

A

Seems to be complex.

A

From many users perspective, and they are looking for simple solutions which I personally don't fully agree, because doing backup is probably easy, but doing restoration is harder. So, uh if you guys are interested, please you can take a look at the discussion on reddit.

A

uh There are a lot of uh a lot of threads discussion, discussing different methods of backing up your kubernetes volume uh over there. It I I personally find interesting and uh looking at those ones, makes us clear that the uh with the what this working group is talking as it seems to be on the right track to providing to provide fundamental building blocks uh so that they can do that, a backup, william backup being one one popular change box tracking.

A

All this uh really really interesting ones that can potentially solve the problems discussed in this threat. Take a look please. The second agenda we will go through will be the data production white paper.

B

Can is the feeling that snapshots are what people want and that the existing snapshot design meets everyone's requirements.

A

uh You ask a great question, so one snapshot is actually a lot of people are asking for what they are asking a little more. It's on top of volume snapshot some kind of orchestration such that they can do simple scheduling.

B

Oh okay, so so people aren't even satisfied with the status quo for snapshots necessarily.

A

Yeah, basically, what they were, the initial idea, the person who raised this question- I don't know who he who they are basically what his requirement is.

A

A simple like crown interface that can literally schedule snapshots over the volume in a in a regular manner and that's one. The other one is uh a simple mechanism to schedule: backups volume backups uh on a regular manner as well. Basically, what they want to achieve is relatively low rto, using uh snapshot to restore the volumes and the relatively longer but more durable volume, backup solution for uh using the one backup they didn't talk too much about orchestration over the workload.

A

One of the interesting idea was utilizing: git ops for kubernetes resource backup and this lightweighted volume feature for volume, backup and snapshots, but I I don't I I think this problem is oversimplified in that threat, but it definitely worth reading- and maybe you have some comments you can. You can leave over there as well.

B

Sure, okay, thank you.

A

C

uh Really cool actually yeah, it does seem to be dominated by um one or two very noisy people who have very strong opinions. So.

A

uh Right right, that's the reason why I think we need our ways over there. I mean I personally feel some of the problem has been greatly simplified in their cases.

A

uh Theoretically, you can make it work for backup using git, ops, plus one snapshot on worm backup, but during the restoration I I couldn't think of a simple way. You can do a restoration without complexities lies in tools like where I kesten to do the coordination so yeah they paid a lot of attention to the backup piece. Let me put this away.

C

Definitely are you asking for help putting together a response? Do we want to kind of put a response? No.

A

No, I'm not yeah, I'm not asking for help, but just the two things I get out of this thread right. One thread: one thing is that backup is a thing that people want right to for data protection, which means what we are doing over here is valuable right.

A

The other thing is that people seems to be make this problem oversimplified, as I said, and that makes me feel stronger about a white paper right as soon as we publish a white paper and explain why it's so hard to do data protection and the restoration in a kubernetes context. I think people will get a more clear idea.

A

Does that make.

B

Sense, I agree with both of those points that uh everyone has a different opinion for how backup and resource should work. And fortunately you know you can make as much noise on reddit as you want. But if you don't write any code like nothing is going to change. So um you know people who are going to write the code have to have to define the problem correctly and that's. I think where the white people will help.

A

That's that's! That's exactly what I meant why, if you're interested right- and if you have your own opinion, you know feel free right, I'm not going to just go ahead and do that. But uh it's your it's your choice. Anyway, uh the white paper updates. uh We will go through a couple of items tom, I think you've got the quiz hooks phone has finished, the change, block, tracking and uh missing building blocks diagram is updated. I guess uh the third item is very small.

A

uh Sheen is working on phase two of volume snapshot, ga, basically swapping the storage version from we want better one to be one I'll, kill her I'll, allow her to explain and then we can open for issues and now, let's get started on the uh white paper thing. uh Okay, tommy won't get started on that. uh I can give you you want me to show the screen or you can do that.

C

uh Yeah I mean there's not too much worth showing. um I think we, I think the motivation section is unchanged, but it's probably okay to stick in.

D

I see that you added another doc, but I don't. I don't think we have permission to read that talk.

C

Oh, let me have permission um yeah. I do I've been formatting the appendix section um trying to make it a little more consistent and I think we'll probably more content. We want one out there to make it uh more balanced. I think um so, I'm I'm chipping away at that. Let me modify the permission settings there.

A

Okay, why don't I do this I'll make you a presenter.

C

A

Oh actually, I can't uh only.

D

Yeah yeah, maybe.

A

E

C

C

Okay, so I think I have impressions. um Let me share my screen too.

C

Yeah, so basically, what I'm doing here is just formatting that the appendix section, I think, having existing format, that we can kind of flush out a little bit more. Basically, I just copied over all the data from here.

C

From the bottom of this right, we have this kind of big unformatted section from here. um So that's that and then for this one. um Is there any feedback? We just want to stick this into the document. um You know it's a little bit. It doesn't make sense to just add in the quiet section in the main document. Are people happy with that? We want to do something else. Yeah.

D

We can, we can have people review that give comments right. We can, we can add it there and that's fine either way I mean we can hopefully just go ahead and review the separate dog, maybe that's easier. The whole dock is a little long, so maybe just to have. If uh so now everyone has permission to comment to the doc right.

F

For this right.

D

The separate doc- maybe I think it's probably easier, the both uh the you have this one and then you have the appendix if both are ready for review. Maybe just to you know, give everyone permission to comment and then this way.

C

Let's make this up and.

D

Just yes yeah you.

A

Can change you can change and make a now you can you only give uh read ability, not comment right to share and that link over there change.

C

I think this one is, anyone can comment on. This sounds good.

A

Yeah, thank you. Yeah.

C

C

Yeah, so that's it um not much here I did want to ask to um do. We do anything for kubecon coming up. You know, I know like where we at castin are working on a bunch of things. um I was curious if people were wanting to coordinate on anything, um you know I can give a brief overview. What we're looking at.

D

uh So it's a virtual right. So um how do you want to coordinate yeah? We can there are some, so a social channel has a session it's a session about our data protection when group is an update. uh So what do you have in mind for everything this is workshop. I'm not sure how how to coordinate on slack.

C

um Well, so we're we're doing, uh I'm I'm giving you a short talk on ransomware, which might be interesting people work here so that we can. I can give an update on that and then we're also captain's doing a cloud native data management day which.

B

C

More, I think, xing you kind of know about this already right. I.

D

Would yeah, I would yeah uh this I'll, give a talk representing uh sixth story. So actually six three has two talks there. uh One is uh just about seaside. The other one is about cozy.

G

D

That's not specific to the protection.

A

We can just put all the things in the uh dock.

D

Oh yeah good idea, yeah, let's just have a have a section on the top. uh We just uh put other like all the talk sessions. We think people are interested. We can just put them there.

A

Yeah we can, we can put it.

D

Then we can sharing, we we have our slack channel. We have our own slack channel. If you want to discuss over there, then we can do that as well. uh Any other ideas.

A

I I think the uh in in the past, the uh the cloud native team is going to create individuals uh snapchat threads, for each talk.

A

uh If you want, you can update the doc, along with your link, as well as with the snapshot snapshot. Sorry, the uh really, I'm sorry, sorry, not the uh slack slack channel. It's.

D

Not for each talk right, I think it's it's nervous, like one uh there's one channel for speakers and the people who attend and communicate it's for all. It's not like for.

A

Sure but each talk has a threat.

D

I don't think it's for each talk. It's just one. It's just a one common channel, otherwise there'll be so many talks. How can they it's.

A

A it's a threat, not not a channel; it's a thread each thread. Yes, anyway,.

A

uh Okay, tom is that all you have at this moment so.

D

Yeah, so in the past there are well well it's face to face where they have this, like um meetup section uh at the end right so uh virtual, they may have something so we'll see. I haven't seen any information that um yeah.

H

D

It's a little bit different. It's it's a virtue. Can we move.

A

This thing did this: that discussion to the uh open issue once uh I think yeah, why is it waiting for the change of block checking.

D

A

You have, you are already a co-host, so feel free to share your screen. Please.

A

uh Are you talking? We cannot hear you if you are.

H

Oh, can you hear me.

A

H

Let me try to share my screen.

H

Can you see this screen now.

H

Yep, okay, so uh this is the white paper segment related.

B

H

Show snapshot that we have um discussed and we have dried up. I got feedback so far from zing and from ad, and uh these are this is very short segment uh this. uh First, they talk about the motivation and the requirement that we come up with and the uh sample workflow for the backup that we used a different snapshot service to help with the backup to make it more efficient.

H

So you can see the motivation is very much that, instead of backing up the entire volumes every single time, if we use a different snapshot that tells the difference the the changes does happen between the current situation, the current status and the previous backup, then we only need to back up the chains. So that is the motivation behind this one, and we also talked about the difference between the differential snapshot and the incremental snapshot here. That incremental snapshot is just a special case of differential snapshot.

H

We prefer the differential snapshot, because that will help the backup vendor, like mice, like my old company, to create more flexible backup policy. It can be uh backup um instead of just backup the compared to the previous one.

H

We might be able to uh compare with the one last month or so, for example, uh this will give us the um practically, I think, most of the time it will be like influential backup, but this differential snapshot capability would give the backup vendor more flexibility, and the requirement that we spell out here is that uh we spelled it all out here, and I can talk briefly about some of them, but you can, you can beat it, but basically we want the snapshot to be between any to snapshot and we want to support uh both the block and the file system, uh so, instead of just uh so we're also talking about uh the the the um the detail level that we should able to provide so that the backup vendor can base on this detail level to do uh appropriate backup so that they don't have to back up the entire volume and so and so forth, and then for the um backup workflow.

H

This is just an example of how the difference of snapshots can be used um it. So each vendor may have a different way to do this, but this is just one example, uh and this is the workflow is basically first the um the the controller. The backup controller would uh create a snapshot, the volume snapshot of pvc and then, if the previous snapshot is available, if the controller have this information, somehow, if that available, then it will call the different source network service.

H

With this choose natural information, then the differential snapshot survey will send us send send the controller back the list of chain. This list of the content of literacies of chain is depends on uh either whether this is a blocked device or is it a file system device and the the detail will be provided by this different concept? Concepts would enable the next step, which is the controller. The backup controller at that point will only backup the chain whether it is a block that chain or is a five chain or a new directory, and so on.

H

So it depends on the detail, so that is the the the typical workflow and after that uh you know you can do continue with other backup of the user, namespace or application or whatever. That is so. That is how the backup workflows use a different snapshot to improve the effectiveness of the backup service.

A

Yes, I I thought the cpt will apply to both block and the file.

H

No, uh the cbt, the name mean in blocks tracking and we intend to make another thing called chain: 5 list or change. Yeah yeah.

A

You see my my sorry I was. uh I was not aware of this change. I probably didn't know.

H

The the service, the service that we we propose here should able to do both, but maybe when we do implement, we might start with the cbt first then.

A

Right but our list is it's not as useful right, because you know you, you can simply just touch a file which is five gigabytes and.

B

Change the timestamp and tell you how you design the change file list, sure you could get change, file ranges and and try to make it smarter.

B

Although I I was wondering, is there some confusion because the my memory of this feature is it can be used to back up file system volumes by relying on the ability to convert back and forth between file system and raw block volumes in kubernetes.

D

And that is if the the back can under underline, uh if it is.

B

Blocked back into it.

D

Yes, yes, that part is right, but I think this one was talking about pure, like file volumes is just files. You know there are.

B

Yeah yeah, that's something we should do too, but.

D

Yeah yeah, but it reminded.

H

B

The problem that we had with uh with uh in kubernetes tracking whether a snapshot was of a volume or was of a raw block or a file system volume, did we ever make any progress on updating the snapshot api to remember what it was.

A

We didn't get any chance to work on that ben yeah.

B

No, we haven't, but that was that was one of those issues that we discovered is. Is that.

D

Yeah yeah that that's something that we will be looking into in when that initiative. Yeah.

H

Okay uh for this, uh this different so snapshot later uh right now we just talked about uh the white paper part, which is just give a general um id workflow yeah, sorry to distract.

B

D

Yeah no problem uh yeah so think that one it's just because it's not uh it's not straightforward. So that's why progress is slow, so we will get back to that. um So I think yeah so the white paper. I know that we talked about that. When we talked about the cbt design, we said that we will look at the block, just change block. First, we don't look at the change file to start with, because it's they're very different um yeah, but.

A

D

The white paper yeah: we should talk about both so.

A

I don't know- maybe maybe maybe banking some insights from netapp site, but change files yeah it is, it is certainly doable it. Just I don't quite understand the scope over there. uh Does it cover file metadata as well.

B

I'm pretty sure that there there was an implementation of an api like this in by netapp years ago. I think it still works, so I'm just saying it was designed a long time ago. I believe when we did it, it was at just the file granularity. So if you touched a 10 gig file, it would report that the file was changed, even if you only altered one byte um so yeah that those kinds of stuff are weaknesses, but it's it's better than nothing. So I don't.

G

B

Don't know if you'd want to propose an api that sort of has those kinds of trade-offs or limitations.

D

I think eventually we should right, but I thought we were talking about it. We will start with vlog and then we'll go there because the human block, we still have a lot of unresolved issues and the file is uh more complicated and it seems to me the backup vendors will start with the block first.

D

Well, it's my that's. You know it's to my knowledge. I could be wrong. Well,.

B

That that's for the for volumes that either are raw block or can be converted to raw block. Once.

D

B

Up a file system volume that can't be converted to raw block, you have no choice but to deal with the file system.

D

Right, I just haven't heard many people are doing the doing this change file listing. But I'm saying I I could be wrong.

C

Yeah on the data protection vendor side is, is to do you, know, kind of the opposite of what ben just described, where you actually track the modification time and then avoid doing full file updates if needed.

C

um You know we, for example, do a sampling thing that will will still do a hash of some percentage of the files, but you know we we think it's okay to maybe reread the the metadata of each file. uh If you're looking at.

B

Backup, oh yeah yeah. If you don't mind like reading the contents of the file, you can implement a space efficient or transfer efficient backup, but you end up reading a lot of data that you might not have to. If you had like a change file tracking api that had really good information.

A

uh What what are the the rustic way doing things? It seems to be okay right because, because what they do is the the first traverse the file system to find out any file changes and then do a block comparison.

B

A

Only shift the block block differences out to the red backup repository. I was wondering whether this is something we can consider.

A

But anyway it's.

B

Yeah, stick relies on having some additional state that it remembers about the last time it backed up the file system so that it controlled.

A

B

That hasn't changed.

A

Sure we have the previous snapshot right shouldn't that preserve the similar mechanism or similar states.

C

Yeah, I think it it does make sense to do that. We we do a trade-off to because you can always modify m time. You know which is a little bit dangerous. I can, um but you know we do a trade-off where we we both uh rely on m time, but then do a sample to make sure that you know nothing. Tricky is going on under the covers too.

D

So, what's the suggestion for for this uh white paper, this is showdock by phone, uh should he add more information on? How did you change.

A

I would say, maybe just leave the change file to the future, because I I don't think we have a finalized solution over here. How do we define it and also.

D

We don't yeah also, we don't have to come up with a solution in this right in the why people just want to identify problems. What problems we want to solve so maybe.

A

Sure that's my point right. Is this the right approach to solve this problem for fire systems?

A

That was my point right, but anyway I you know it's minor. Oh.

D

You're saying like change file list is that the right solution.

A

For file systems, yes,.

B

It could only be additive like if you already have a rustic like scheme where you have you remember the state of the last snapshot, and you know your only vulnerabilities are to people goofing around with end times then having a changed file list. Could let you detect when someone had goofed around with an end time more cheaply. um So it's it's better than not having it, but it's not necessary.

D

So we just say this is like one um optional thing: people can do, um but do we need to add other things to this, such as stock other than you know, the change block tracking chain regardless? Do we also want to describe other other ways that the.

A

Yeah, you know the requirements over. There makes me wondering whether this this will be a first class supported thing right, like the last time for file system volume, the service should handle shared volumes right. That means that this is going to be a feature of the service itself right so.

D

We, maybe we don't call those requirements.

D

Maybe we should I'm not sure what what do you? What to say because we're not really. This is not a design doc. We're not writing a design, doc so sure.

B

D

Write requirements so um so.

H

Oh cool, maybe what is the problem? We want.

D

You, what are the problems? What is software? It means explain that, um and we identify change block. Trucking is something that we need to solve our problem, but change file is something maybe we need, but we're not sure. So maybe it's like optional and then are there other things we think we might need still optional. Maybe we can also.

I

D

I

Change file, it's just that we don't know what's possible, so I mean there is according to ben the resting way where rest keeps track of in all the change files. There is the rsync way where you, actually you don't maintain any state, but you just compare the diffs and you do check sums and then you send the.

B

I

Blocks there and there is the netapp way, which is the I can share. Actually, if you don't mind, I can share a little bit about how netapp does it yeah.

D

Okay, so should we actually yeah so that maybe I'm thinking that maybe that name should we just here, we explain a few ways of doing things like you know. Just you know.

I

I think we can generally talk about today. Talk about.

I

Changing tracking changes for files, but how it's done. I guess that um so this says host disabled parties spend screen sharing. Can someone enable that.

D

Oh, are you trying to share something yeah.

I

I'm trying to show you.

D

Let me okay, I need to.

E

D

Your co-host, okay, nice.

I

So I mean this is public, so you can't just google this snapdiv. This is basically the api where you can give two snapshots to netapp snapshots.

D

But this is not. This is not change box you're. Seeing this this.

I

Is not this is this is change files, oh okay, right and um basically on tap. The netapp operating system has a two building blocks. One is snapdiff. This is basically what we were talking about.

D

Okay, so this is a.

I

Difference in engine that generates a list of change files between two.

D

I

Copies, okay- and this is a sniper cloud. This is just a way basically to back up all the change blocks to a remote endpoint. For example, an object store okay and basically, but you know this is smart. So if one block of the file change, you only send that change. um You don't send the full file and also you know we do dedupe compressions.

H

I

Efficiencies are preserved, as you backup.

I

And you can recover individual files and so forth. I believe these apis are not public, so I think these are only.

B

I

To backup vendors and.

I

But I imagine, like I mean some are listed here, so I imagine like these vendors.

B

I

Take advantage of this, they snap the paper.

D

I

But I don't think these are public, so anyone, I don't think anyone can write a solution that would take advantage of these files. So.

D

D

Yeah, so maybe we can like at a higher level, just to explain the different solutions.

C

Yeah, I guess what do you want to put in the white paper and what do you want to follow up with as design proposals, or you know other follow-ups later.

B

I think we would want to emphasize that change. File tracking is an optimization that is, is nice to have, but not presented as something that's absolutely required.

I

Yeah, I guess you know look. Basically we can do change track file tracking at multiple levels. You know rsync is one way to do it. You know it's like a very efficient poor man's way of doing it. You know um this is like a more optimized version where the storage system will tell you exactly which blocks got changed and you only send those without doing any check summing without doing any comparisons between source and destination.

I

So I don't think we have to be too prescriptive about what the right solution is, but we can say you know these are some of the options as far as how you can do. Change fall, tracking.

H

Yeah, I think that's the the valid idea that, like I like I mentioned in the segment of the white people, that it's just an example of how it can be. It doesn't say that it's the only way, so we can enhance that document with this kind of workflow as well right, and I see that a lot of these um basic, uh the workflow that you are showing on the screen right now is also kind of similar to the other workflow that I present earlier.

H

That means basically the snapshot and then we, the the the difference between the two and only backup the difference and so on and so forth. So uh that is a good enhancement for that paper. I think.

E

D

Thanks: okay, so fun, uh can you take this feedback and update you.

H

So uh I I think I would uh could you guys do me a favor and and and send that right either write the feedback directly on the shared document.

J

I will do that.

H

Or send me an email either way, it's good for me,.

D

Can you add the um what link can add a link.

I

E

I

E

I

It to the meeting links.

C

Sure, if I had a question on nomenclature here, so you you have a definition of differential versus incremental, um I um you know we use a slightly different definition. I'm curious what the what the kind of community thinks is the right definition here. So for us, when we say differential, we actually are talking about that full file comparison or doing the read of files and comparing the differences.

C

uh An incremental is using something like a change block tracking api is, is your definition here of differential? You know based on your specific line of products, or is it something you've kind of pulled out of the community.

H

So I think the the high base on my research on uh a couple of uh vendors- one of them- is uh azure. The other one is vmware cbt. So this kind of uh the concept that I pick up yeah. No, it's I'm not 100 sure that is like a common uh terminology in all the backup vendor. But these two are what I space on.

C

I see so in azure and and vmware they make the same difference between differential and incremental. Where differential is you can compare any two snapshots and incremental? Is you can compare uh two consecutive snapshots? That's what you're saying.

H

If you're looking to, I can send a link for um azure for sure I can. I send for azure that's two um online document that available that I spell out the differential snapshot and uh and incremental snapshot. I think at the s, speed and also uh helping on that uh link. I think I got the link from him.

C

A

I guess I share with the toms hum over here. The incremental snapshot thing seems to be a storage specific feature where they have a way of tracking only keeping the differences between a snapshot on a specific chain right.

A

Whether we want or not it's related to change, block tracking, it seems not.

A

Related as long as the storage vendor can return you a change block from any arbitrary snapshots from the same volume, it should be sufficient right.

C

So sean, are you saying that, basically we we frame this as just what the apis are, what apis are available for cbt and then say you can build. You know, leave the differential versus incremental a little more general, and so you can kind of build those those workflows with a cbt type api. Is that what you mean.

A

Right right, I I think the uh individual storage providers features with regarding to incremental snapshots.

A

Seems to be not quite relevant over here, I'm not sure whether we should be included. It should include that.

A

H

What do you think about these things? I I guess I remember I incorporated oh.

D

I just I I initially found only mentioned the different show, but we have been talking about incremental for quite some time, so I just want to make sure that we actually have those definitions somewhere. So I that's why? I don't know if you are actually reading the same um chanting. You're, not sharing your screen, not sure you're you're talking about why we are explaining those, I think the I think we should have a definition somewhere. Otherwise it's uh it will be confusing like because we sometimes talk about incremental. Sometimes we talk about different show.

D

We we should have a term described somewhere. Even if it's not you know, even if this is not the right place, we should have it somewhere before we talk about differential snapshots. Suddenly.

A

Sure uh the the pieces that I feel slightly not accurate is the uh the last paragraph right we can say this is the features that what is the incremental snapshot? What is the differential snapshots.

A

The statements say saying that differential snapshot should would be more desirable. It seems to be.

D

Oh okay, yeah right.

A

D

A

D

That, okay, we'll be more okay, then we just let's remove that. I think yeah. I just want one. We have higher terminology somewhere, but I don't want to say which one is better.

A

Exactly right, so we are proposing. We are proposing a position here, so uh it's okay to explain concept, uh and we think this is the right approach of doing things. That's that's all. We need to do right.

D

Yeah yeah: that's fine yeah, I'm probably fine with that.

H

So I just delete that line.

D

I think he has already some uh who is. Oh, is the phone okay you're modifying that okay yeah see that'll get deleted; okay, that's fine! Yeah!.

H

uh And you answer a command that we should should not say is a requirement right, maybe yeah um this kind of feedback. Maybe we just put it on the document itself. Okay, sure sure yeah. We have.

D

No remote video, we might.

H

Run out of time, otherwise, yeah.

D

A

A

Thanks phone nice work uh shin you want to go through the uh diagram.

D

uh The diagram- actually I didn't change it, so I think last time we just started to show that and then I asked everyone to provide feedback, but I don't think I see anything. We can show that again.

A

Well, if it's not different from last time, then I think people got the idea.

D

Yeah, it's the same. So basically uh I think tom, you have some question last time. uh Can you actually maybe add your comments on that uh in that doc, or I think I think, we're not sure like how you want to change it? I mean we don't have to include this at all. I just thought uh this actually showed all the things we want to talk about in the end of my paper.

D

So that's why uh I thought maybe we can show the style from there, but you think if you think the diagram is actually uh missing something then we can either add it or we don't have to show that.

C

I think it's really good. um You know. My comment was just a little bit high level, which it was that um you know from. If you just read the diagram, it's a bit unclear what components are actually things that you know kubernetes alone versus things that are. You know the underlying infrastructure versus uh backup vendors.

D

C

D

Like this is the just the diagram right, we can add some text because un underneath to show what what what that means. uh Okay, we can yeah. We can think about that, because this diagram is already very busy.

C

D

Don't know why we can add. You say this is a uh you know. What component is that uh I think we mean to only show the kubernetes. I think the green, green, yellow and orange just a kubernetes component and then the in the middle. Those are processes so um yeah.

D

I think I I I see what you're saying: uh okay, maybe shenzhen, maybe you know I need to think about how to explain that more clearly, maybe just through some text or maybe we need to add some note next to the process somewhere to show it's actually in the reserve. Back back of vendor has some software kind of coordinate everything like the orchestration piece right. It's actually the commander.

A

Yeah, I I'll probably do a do a lucidchart rejoin of this. This diagram. Maybe it makes life easier because, okay, this this thing is basically withdraw this using slides components, yeah.

D

Yeah, this is not that nice.

A

That the lines is hard to adjust the phone etc. So maybe I maybe I I will spend some time to draw a lucid chart of this. Okay.

D

Sounds good thanks all right.

C

You asked me to add feedback to this. uh How would you like to get the feedback.

D

Oh, you can add. Why don't we just add it on this uh this this doc. This uh slide you it's, it's actually shared in the. If you look at the agenda doc, it's there uh yeah just you can add comments there. I think it's.

A

Yeah you're going to point the other comments to the slides itself as well. Yeah.

D

Everyone can comment, I think, right now, yeah.

C

Perfect thanks all.

A

Right, uh we are 12 minutes away shin. uh You want to talk about the ga thing.

D

Oh okay, uh well uh so I'll just share this.

D

Okay, so well, we know that snapshot is already a190, we added the v1, um and so we still support both volume snapshot, v1, beta1 and v1, and then currently the storage version is v1 beta1.

D

So we serve both what we want one better one, but the storage version is we are better one, um but now what we are making a change, we're changing the storage version to v1 still still serving both um and then so then, after that, we're going to cut a release, cutter external snap, shutter release and then the last phase will be removing the support for even beta one, and I don't know how long we have to wait. A new check.

B

D

B

This change aligned with kubernetes, 121 or 122.

D

121 so the second one phase, two phase, two.

B

Okay, so this is for the release that just went out we're going to change the stored version to ga.

D

uh Change the storage version to viva yeah before we cut in the release and then.

B

My understanding is that, as long as you have one release between each of those changes, so you you add ga support in 120, you change the storage version of 121 and you can remove the beta and 122.. I think it's one release each.

D

Yeah that'll be that'll, be.

A

Is it I thought it's half year ben.

D

I thought I heard something about like that. You know application you need like one year or like three or four releases.

B

Beta, yet what have we deprecated beta yeah.

D

We're talking about, we are talking about the removing bit. No, it's not deprecated right right now, we say.

B

Okay yeah, so I think that's.

D

A separate thing is.

B

We have to announce that it's deprecated and that you should be using ga and then there's another clock on deprecation.

D

So yeah, I don't know if we can, we can say deprecate, do a better one. When we uh change story version, maybe we can so I don't need to find that out. I don't know how. How does this thing work? Maybe maybe we should do that yeah? Maybe we should do that. Otherwise,.

A

Last time I spoken with, I speak with uh jordan.

D

A

His suggestion is: keep a list for three release cycles for beta api.

D

A

If you want to.

D

Follow why don't we have weighted right so we're beta one? That was what is the window 17 yeah.

A

D

So now it's already 1.21 four releases already.

A

No, no, no starting from 120, okay.

D

Three releases right, so that's basically, where we.

A

We officially support. We won right.

D

A

And then this phase two is a transition period.

D

Okay, so it's probably one yeah.

A

D

Or something yeah, it's probably 1.24, I'm guessing because it's a three release, probably three release after we declare duplication because we didn't say when beta value is deprecated, so probably we should say deprecate it now. I wonder.

A

Oh, no, uh it is in the api, you cannot say it is deprecated. Oh.

D

We cannot say: okay, yeah.

A

D

Basically, just remove it yeah no.

A

Actually, actually,.

D

I do see the application notice for other apis yeah.

B

You have to deprecate it before you remove it. Yeah.

D

If you look at the like the cid extension, those things you do see, the application notice yeah.

A

I I missed understanding the question.

D

Mark okay: let's question.

A

D

Let's find I mean.

A

I misunderstood to you so basically you said your deprecation is sending out announcement. If that's the case. Yes, okay, I thought you are removing that in 1.21. That's not going to happen.

D

Yeah, no, not not. We not okay, the application.

D

We need to call the application of 31 data one in.

B

But after we run out the clock on the deprecation which might be like 124, then then we remove it.

D

Yeah yeah, I think I think likely when that change works. That's like a year right, they're, slowing down the yeah there's.

A

A good amount of work as well soon right for deprecation.

D

What the application is no, this is this one should be fine, I don't know if we need actually add some messages somewhere, because some other apis have actually messages. If you use the like the beta idea, you have to get something if you check the logs you'll see something so I no I'm not sure so that means there's.

G

A change yeah, there's a um there's, an enhancement that was put in for cube ctl such that, like you, can send back a header message that says this is deprecated and then, when you get that version it will, it will do.

B

G

B

G

Say um I think in the crd you can define this um and define the deprecation message for that version. But I don't remember exactly that's where I would look and then uh and then for um the other question I had was: do we have to do anything to support converting the storage versions uh once we switch it over, such that, like the the cubesigs storage converter or some or one of those other tools will be able to convert storage versions um or will.

D

That just work does anybody know, I think, that's I think, that's built in uh at least like when I'm testing this one, it's beautiful, it's fine. It works yeah, okay,.

B

Yeah, you need a version conversion web hook if you make like any complicated changes. But if you didn't make any weird changes.

D

B

D

B

D

Changes actually it's just a version change between right.

A

It's automatic except one fact: uh if your we want better once they are it's invalid, it might have errors.

D

Yeah but the problem, is you don't even let that succeed you you know you create a invalid one in snapshot.

A

No, no existing ones.

D

It's not possible because we added that in the controller a long time ago, it can never create a volume snapshot that is invalid and banned.

A

Sure you have a univariate one before 120.

D

Well, we're not supporting what one one yeah before not training, we at that logic a long time ago. Actually, I think, the in the controller we have that quite some time ago. It's it's it's before and trendy.

B

But I think the concern is, if you had a snapshot that you took a long time ago and you upgraded kubernetes across like six versions and that old.

D

Broken snapshot was still.

B

D

Think we need to figure out what is that.

G

I didn't think upgrades like that were supported.

A

It was to call it out saying that okay for those invited ones which has been in the system for.

B

A

Time right, uh you will have this issue right. Basically, we have this document in the cap when we introduce the when we're introducing the uh web hooks those invalid ones. Basically, the the the bigger problem for that ones is that when it is converted and the b into and we want, it will fail and it will remain. We want better one in the system. Then, when user try to delete it because of the web hook it does, uh it is possible. There are finalizers on that resource. Then it goes.

A

The request gets sent to the webhook and webhook will deny the request, say: oh you're not allowed to remove the finalizer, because the uh resources in in is an invalid one and that will make the resource not deletable.

D

So yeah, I think we need to check the because we, I think we added a check long time ago, it's probably in 2.0 like in the controller. We don't even allow those type of snapshots to be bound, so I mean yeah. So we need to check that.

A

Yeah you can update that shin. You can always update this resource directly against. The api survey doesn't necessarily need to go through your snapshot. Controller at.

D

All but then I'm just saying that they have to be using a very, very old.

E

A

D

What I'm saying like when did they well, so we need to find out when that logic was added and then any snapshot created before then, which is invalid. Then there's a problem right, so if they have like two, so if it has two source basically right, perhaps two both the.

D

If we had both the like the um volume handle and the snapshot handle.

B

Pvc, something both on pvc.

D

And has the content as source right? So if, if that's the case, uh then there's a problem deleting that, because.

B

And we had some action required release, notes telling people like when you upgraded to various versions like you're gonna, have to fix these problems and if somebody ignored all of those and they.

G

B

The way to the to 120, 122 or whatever one twenty.

D

One four talking about that, then.

B

D

Yeah, it's gonna, I think, should be okay, because.

B

No I'm saying like if you have, if you've had a very a very old snapshot that was incorrect and you just haven't touched it in like a year and it's sitting there when you upgrade to 121, you will no longer be able, in fact, when you upgrade to 120, I think you can't delete it anymore. You can't update it anymore. You had to fix that back in the prior.

A

Release when you want friendly, you can still update it then, uh because we uh yeah the the web hook will allow you to do that, uh but in 121 you won't be able to do that. Okay,.

B

So that's the key is if, if people haven't fixed it by 121, then then they're going to be in a world of hurt.

D

So right so actually I don't know how I still don't know how that would happen. 1.21. We just changed that, so we changed the search version, that's something we.

A

That's right because, because the api server needs to convert all we want better one into, we won.

B

A

D

B

The validating web hook to say no more, no more updates on invalid ones,.

A

uh We actually did a couple of things ben right. One thing is that controller will actively label those invalid ones, yeah.

B

A

B

When will we change it so that you can't update them anymore? When will we tighten up the validation.

A

After the convention has happened, it will be tightened because the web hook will get we want. We want uh validation request. Instead, we want better one. What additionally.

B

Okay, so you're saying the mere act of switching the defaults of e1 means that you're going to get the new validation behavior.

G

Exactly because.

B

A

It converts it to your.

G

Stored version before it sends it to your conversion web hooks, okay,.

A

Exactly because the we won one, the white edition on the v1, it's much tighter than the uh edition on the way, one better. One.

B

Okay yeah, so so, hopefully, people have been reading our release notes and if, if they haven't they're going to be in a world of hurt.

A

J

That's all we can do.

A

Yeah, that's we want to make it clear and that's all.

A

All right, we are on time, I guess uh any open issues.

A

No all right, thank you. Everyone I'll stop, recording.