KubeVirt SIG - Storage, 22 May 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG - Storage 2023-05-22

Description

Meeting Notes:
https://docs.google.com/document/d/1mqJMjzT1biCpImEvi76DCMZxv-DwxGYLiPRLcR6CWpE/edit#

A

Foreign, you type with interest. I love this topic, it's good to see somebody else interested as well, so we'll definitely try to discuss that a bit. um In the meantime, we can go ahead and get started um since it was added. First, we have uh an issue here that we can take a peek at in CDI. So let's just take a look at that. Really quick well folks, are fleshing out the agenda, so um you guys can all see my screen share.

A

I, trust, yes, okay, all right, so images that require too much space actually get imported, potentially overwhelming the disc okay.

B

There's a little context on that one: um we use a cumu image info on uh especially crafts to invalid disk images. That return like you know, my virtual size is a gazillion petabytes or so, or something like that, so it should never import um and we run that test inside of a.

C

B

So you know if it's whatever using a lot of memory or CPU or whatever it should get rejected after a little bit, but uh for some reason it's no longer being rejected and actually imported, and it's actually using up the entireties. So.

A

And it's using up the entire disk, because the test bed is HPP. Presumably uh it shouldn't happen on like on something with a meter uh block device. Yes, okay,.

B

A

B

It should it should reject the image, because the image is specifically invalid. uh I think the origin of the images I think came from openstack, where they do something similar where they check. If the image is valid before they actually do anything with it, um it's basically to prevent the denial service type attack where you, you know, load up an invalid image and it uses all the CPU or memory on the host.

C

Okay, is that link to a DOT raw.xc file? Is that the actual image that's the problem.

B

B

I actually didn't even notice it from the header of type NC I. Don't think those uh no I guess it is.

C

C

Does allow you to sort of make these kind of bombs that will explode when you want to compress them, I think they're called zip bombs or something.

B

I, don't remember us having those images as an Etsy file at all I, don't think we supported XZ at the time that we introduced all of those.

A

Okay, so yeah I I know this. uh These images were here or I. Think taxi was here quite early, um yeah I, don't remember exactly what moment we added the uh these files they'll be they get. These are actually because. Obviously this is a cluster URL. So the where are the images actually maintained? Do we have them? We have a.

B

A

Repo somewhere right, yeah they're in our git repo, yes, okay,.

B

C

B

Know in reality, they're very small files. It's just that the metadata on it like is modified to be enormous.

A

um Okay, yeah, so it's interesting that it started.

A

um It stopped rejecting them at some point and yeah, not sure why that is.

B

So the target size looks like it's a four.

C

Gig, something like that.

B

B

That would explain why it's not being rejected, because the total size on on these is probably in the 30-day game. Well, mm-hmm.

A

And as long as it fits within whatever PVC we allocated or is supposed to well.

B

Different things in both you know, yeah right.

A

Okay, so I just want to make sure uh just for the purposes of this call. uh What should we do with the issue right now? Maybe there's nothing uh to do I'm, not sure. If we have someone, that's already known to be working, it.

B

I think Alex is looking into it. Okay, I haven't really had too much time to look at it. So yep.

A

All right sounds good, so I'm I'm, seeing at the moment this is new. We kind of discussed it here, so people have contacts, but there's no, uh nothing really further to do here, no notes it should be kind of just new and getting looked at. So um that's fine for me on the review.

A

um Let's take a look at the next item, so yeah Mark. Do you wanna uh to pop in by welcome, by the way, um welcome to the Sig storage call.

D

Good morning, good morning, um I I've met Michael a few times before, but uh and I I've been a regular on the rig on the other Cube vert Community calls, but this one's really early, so I apologize not having attended any of the prior uh issues. I mean prior prayer meetings but anyway um I'm the open source product manager at Caston and we've been working. Although I've only been at the company about seven months, uh we've been working on kept 3314, um put a link in the agenda.

D

I'll put it in chat as well, and that is to add uh we're part of the Sig storage and Sig app data protection. Working group and kept 3314 is about adding change, block tracking to CSI effectively uh through basically a metadata service. uh It doesn't it's it's purely control plane. It is not a data plane, we're not going to give you the change blocks, but we'll give you the metadata and then it's up to the storage provider. To then you know, um take that metadata and then go retrieve the actual data.

D

The goal here is uh and- and forgive me for grandstanding but but hopefully I can just finish up the summary. uh The goal here is basically to address the use case of people, doing persistent volume snapshots through CSI and those being basically Atomic huge Blobs of the entire volume.

D

This proposal basically says: if you can give me any two volume snapshots. The metadata service can provide the Deltas between that and therefore we can greatly shrink the the window needed for offline backups for application consistent snapshots and our goal basically is to bring this which everybody has in the bare metal and the VM industry forward to kubernetes. Well cloud yeah. So that's that's it in a in. In a nutshell,.

A

Awesome yeah, thank you for bringing this topic up like it's always been uh I mean this has been something on the back burner in my mind. Maybe not even the back burner but um I wondered. Do you have any uh uh insight into kind of where this cap is at in the process? Absolutely.

D

Yeah, absolutely um in fact, I should probably find the design doc because uh we may not have the most recent version LinkedIn here.

D

um So so, as you can see, this started uh a bit ago uh and we're basically in after three three Heats or design attempts uh we're in the third iteration and we got verbal consensus from the working group and that included people from uh AWS, VMware, Google uh and Dell um on on the on the current design, uh proceeding to prototype, which we're already in process on so uh long story. Short. It's it's gone.

D

It's run the gauntlet three times and now we're at the least bad, which makes it the best uh design at this point in time. As far as uh the consensus of of people who've been involved so far, that doesn't mean things, won't change and it won't get shot down, but but we're in a pretty good State, considering where we started from.

A

Yeah, that's awesome, uh yeah, so I think I think um in order to bring it home kind of to cube vert. um It sounds like you're sort of curious what it would take to benefit from this cap in the cube vert ecosystem.

D

Precisely because, because we're also a redhead partner, uh my engineering VP showed me some some email thread that that came through to him and I. Don't know if Michael is is aware of this, but but mention something about Qbert CBT. So therefore I'm a little late as I mentioned earlier, but I wanted to bring it up here just to to see if we can align or at least make you aware of any of our work uh and and align it.

D

If, if, if that's possible, I don't know if it is, don't see why it couldn't be. It should be a good topic for discussion regarding.

A

Yeah, absolutely so what I would say just from like the my first pass at this from my understanding is since Cuba discs are a single disk to a single PVC and backing up virtual machines involves taking CSI snapshots.

A

I, don't Envision there to be actually anything to do for cube vert, um because if we take a regular backup of these PVCs and then really, the efficiency is gained by only transferring the changed blocks to the off-site backup, storage location that there's those are just Black Box objects. Those volume snapshots that.

D

A

Hope yeah, that would be the Hope yeah yeah. So this is, you know, part of the I guess part of the design of everything, we're hoping that we kind of get this one for free. um So anyone else have any like thoughts or comments on it. um For me, I'm actually wondering um if we know about which vendors seem to be the most prepared to implement this once it becomes more of a thing because it has to have it. Basically, it's an optional thing that a leader a driver would Implement yeah.

D

Yeah we're talking about now. You know when, when is the time to finally do the the Sig storage review and so on um and and I think we're we're pretty close to once we get the Prototype going. That'll be probably the final comments of people in the working group, so that might be as soon as you know this month or next, uh and that might mean we'll probably miss trying to submit it for an alpha on the next kubernetes release. But thereafter it.

A

D

Assuming everybody uh we get, you know Sig storage and Sega app reviewers to approve it, Etc architecture, etc, etc, etc.

A

Yeah, so how does this this is a CSI feature, so how does it impact? um You know, I think this is an interesting thing for me. You know: CSI is originally kind of a uh container orchestrator, neutral storage interface, but obviously it's driven by the needs of kubernetes. So is this really? This is a CSI extension. It would have to go into CSI and then it can be consumed, buy kubernetes just to be a little I guess pedantic about the process, even.

D

More to the point, um CSI storage providers would need to adopt basically this architecture, which is in essence, some side, cars, a controller service, Etc right and and um and then ultimately add in to their their CSI functionality of their storage driver effectively a way to to make the API for the data retrieval of the change blocks, which, in theory, isn't all that hard. But nevertheless is yet another thing to implement.

A

Sure yeah I kind of just assumed that people would, if you had the list of blocks in hand, you would just uh restore the snapshot to a PVC and then seek around to get what you needed, but there's maybe a better way to do it. I suppose.

D

Yeah, but that that's one way yeah exactly so so that's the whole point of the of specifying everything in in a cup um prototype.

A

Yep awesome uh all right, so the last question that I had on this and then I hope that maybe others, if they have questions, could ask um the pro the Prototype that you're talking about I assume this is some sort of CSI driver. That's implementing it. Do we have a link to that, because it is something that actually had put on my team's radar um to look at essentially implementing this cap in some prototype CSI driver, so we could play around with it.

A

But if, if that's already in existence, that would save us a lot of effort. Yeah.

D

It does does exist, uh it's iterating rapidly, right now to catch up with the lattice. The latest feedback uh that we had in in the constant design reviews that we've been prototyping. So, yes, I'll, try to I, don't know where that is off the top of my head, so I will go, look for it. Okay,.

A

Cool yeah, if you could uh link it in the on the meeting doc, that would be great.

D

Or a presentation I'll find those two things: cool.

A

Anybody else have any other questions on CBT.

D

Thanks for your time,.

A

Thanks for bringing it up today, you're welcome all right. So, let's pop back and see did we have any other topics so that so far appears to be the last one. um Does anybody have any other uh Small Things open floor type items that they want to raise.

A

All right seems like we might have a relatively short agenda event. So if there's no other items, I don't want to keep you further. So um just wait here for one more second and uh and we can call it.

A

All right thanks, everybody for joining, have a great week and we'll see you at the next one.

B

Thanks. Thank you. Thank.

C

B