VMware Velero Community Meetings, 16 Jun 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Velero Community Meeting/Open Discussion - June 16, 2020

Description

June 16, 2020

Status Updates
steve:
community support & reviews
nrb:
Mostly focused on internal build tooling last week
Intending to catch up on community review this week
ashish:
Focussing on internal build setup
Finishing up on PR 2611
community support this week
carlisia
wip: add kubebuilder/controller-runtime - https://github.com/vmware-tanzu/velero/pull/2561
wip: add BSL controller w/ controller-runtime - https://github.com/vmware-tanzu/velero/pull/2617

Discussion Topics
[nrb] Deletion plugins?
Would we want to take this opportunity to also do pre-backup plugins to? I don’t have a use case right now, more asking because it’s an opportunity to do so
Next steps: Design doc
Use labels, not configmap
[nrb] Restore hooks design should be close to merging. Maintainers, please take another pass!
[duffie] Linters for Velero, possibly also image handling
[Vishnu] Restic snapshot specific differential size and https://github.com/vmware-tanzu/velero/pull/2543
Problem: How much object storage did a given backup consume?
Is there a way to know differential size from the previous snapshot?
This would help to prune object storage
Restic does deduplication; if I have 10 backups and I delete backup 6, how much do I get back?
Could we add a field to progress to tell report this?
Restic’s format design documents: https://restic.readthedocs.io/en/latest/100_references.html#design
[tony] Update to Jekyll 4.1.0, https://github.com/vmware-tanzu/velero/pull/2625

A

Hello, everyone: it is June 16th. This is the Valero community. Meeting Jonas is off on vacation this week, so I will be emceeing. I'll drop. The meeting notes in chat there and I will share my screen.

A

And just a quick programming note: we have an archive of the older meeting notes actually got about a year's worth of notes in and then we maxed out on the length of the hack MD. So we moved all the old notes into an archive note or an archive event: hock, MD and now we are starting fresh. So we've got a new note here and with that I think we will get into status updates. So first up is Steve yeah.

B

Everyone last week, I was on community support which kept me pretty busy and plenty of PR reviews. The main thing I wanted to cover is I wanted to let the community know that I'm gonna be transitioning off of the Valero project and onto a different role at VMware fairly shortly and Nolan will be taking over as the tech lead from the VMware side. So I just wanted to say you know big, thank you to the community.

B

It's been for me, a really incredible experience being part of this project and and watching the community grow around it. It's been been really rewarding for me. So definitely a big. Thank you to you all, really kind of proud of what we've been able to build together and I'll certainly miss the project, but I think you know they're they're great things ahead for it. So I'm excited to see where it goes.

B

Happy to answer any questions: if anyone has any.

A

Yeah, thanks for all you've done Steve and for those of you don't know, Steve has been I. Think Steve was basically the first person on valero falero started at hep. Do as EDS snapshot er before it was even Ark, so Steve. Thank you for everything you've done on the project.

A

Everyone who uses it owes you a lot and good luck in your future. Endeavors and yeah I work look forward to working with everybody on this project going forward and yes, Steve's not going far and he'll be around for a little bit longer, but I will be taking over kind of github maintenance, stuff, making. Sure PRS don't fall through the cracks. And if you have any questions, please let me know.

A

Yeah and for my update last week, I was mostly focused on some internal build tooling around velaro still I know. I said this last week, but I intend to catch up on community reviews. I know those those pr's are stacking up, so it is a priority to start whittling. Those down I know, there's some big design, big design documents that are getting pretty close to getting merged. So hopefully we can get those in this week and yeah. Those are my two big things and no one has any questions.

A

I think we can move to a quiche.

C

For me, I have I was working on like the internal build stuff for the plugins last week this week, I'm taking a break from that to finish up on pyaare that I have open and I'm on community support this week. So I will also be doing some PR reviews and reviewing your design Docs. So that's all I have on my plate.

D

D

E

D

Sounding a little bit muffled I, don't know if it's for everybody is.

E

It better now! So! Yes, yes, sorry put.

F

That expensive, microphone to use I have.

E

A not a knob and if I leave it on I, get a feedback thing and I have to reboot my computer. Sorry anyway, I am behind on POV videos, I need to jump on there. We have quite a few open PRS lots of contributions. There and I am simultaneous. So last week, I spent time working on the adding attack, so I spent I'm, adding the PSL controller, the backup, storage location controller, using the controller runtime package, Thank You builder and I pushed the PR, that's in draft and then I added the initial test setup.

E

It took me a while to make it work, had to dig for examples of how people did it out there that would fit with what we are doing. I haven't pushed that, but that is working. So it's looking promising I just have to now actually test the controller and then my my first PR, which is to actually add the computer control our own time things and also convert the BSL resource to use those those packages.

E

It's pretty much almost done, but now we have in a problematic way. It's a race condition and Angie took an interest in maybe helping me, but I talked to him this morning and then I think he got busy.

E

I pushed commit, that's actually breaking the test, but the code would run so he could check it out. What's happening is when this think when the the server starts, the backup sync controller kicks in, like it's one of the first things that starts calling the DSL resource to see what backups are available and what backups needs to be synced. But at that point the cache is not completely populated.

E

If we didn't do the validation and just left it running, you will quickly stop Aaron outs, because the cache will be populated, but we do need to get rid of that error and the way to do it is having a populated before the sync controller runs, and that's the order of things that I'm having trouble making that work, and hopefully it will help.

E

A

A

Okay, if not I, think we could move on to your discussion topics.

A

My first one actually I put on but I think this is largely for Dave on the call she should have actually opened this issue for CSI and I. Think Dave has a similar use case in which he's creating.

A

Custom resources with item action plugins and needs to clean them up, but Dave do you want to talk about your use case and yeah, so um one.

G

Use case actually will boil down to being the CSI use case in the long term, but I think. The other thing is that we are going to have other objects other than PVCs and PVS and volumes that were snapshotting and backing up at some point, so the so with with our project pacific mark, which is the was it these here with kubernetes. There are some restrictions in the supervisor.

G

What we call the supervisor cluster one of those restrictions is we don't we can't just directly statically provisioned volumes, the way that we're used to so we can't have a couple choices here inside velaro, one is to make a provisioning plugin so that we could call out to this special provisioning code, because I don't think that what a big if-then-else statement in the middle of velaro and the other option that we've been looking at is to just move our plugin up a couple levels so that it's no longer a volume snapshot, but it starts actually working at the PVC level.

G

The same way, the CSI driver does and there's other objects that are coming down the pipe that are going. That we'll want to be taking as objects that actually could be shotted, for example, operator driven databases, so the CSI plug-in is it's currently implemented. Is a backup item, action and restore item action? I? Think that's the correct nomenclature, but what it doesn't have is they have a delete snapshot on that path, so I think we need to figure out a way to get snapshot. Deletion for objects other than volumes, working and I.

G

Guess that CSI plugin I'm not sure exactly how that's just I plug-in hooks the deletion together. So I'll.

A

Go ahead, if you should I see you in community I I.

C

Was going to say hooked up into the backup deletion controller, which is run on backup, expiry or backup deletion? So that's how we handle deletion for the CSI. Does that actually call to the plugin on.

A

The bolero core code based so so she should originally open this issue to implement deletion well go ahead. If he shall I won't speak, real, no III didn't want to.

C

Like update code velaro for deletion of CR DS created during backup because we put mean making in things that don't really belong into valor of core, so that's when I had this idea of maybe we should have a backup deletion plug-in, which is letting people run. Custom, deletion, logic or custom cleanup logic to delete, see our DS deletion of CR. Ds is not just the only scenario. There may be other cleanup that others scenarios may require.

C

So I wanted to have a plug-in to do that, and this issue was surfacing that problem for us to make a proposal and fix that the data point yes I would.

G

Call my issues I think.

C

G

Term we'd want to also look at maybe having an object, snapshot or plugin, but that doesn't need I think we can. You know that finally actually needs some thought and design yeah.

A

And, from my perspective, the the reason I thought at the time the reason I thought that this wasn't necessary for CSI is because I think CSI long term I think CSI long term will become core enough to kubernetes that it can. It can justify I think it can arguably be integrated into quarter bolero, but, as we see other needs, I can I can see that there's needs to.

A

Model things with these item action plugins and then you need to clean them up, but we have no way of getting into that life cycle. So I also made this comment. Last Thursday, yeah they're like in this this instance there's people who want to do a very similar approach, but they have no way of cleaning it up without modifying Coralville arrow.

A

The other other reasoning was we have a lot of cortical error logic, that's implemented as plugins. So there's no reason we couldn't say just do the CSI clean up as a deletion, plugin and.

A

Yeah, the third piece was kind of similar to that that it would allow us to make the the deletion controller smaller so that it it would really just manage the backup custom resource, and then it might invoke individual deletion plugins and that may make the move to cube builder and controller runtime easier. I, don't know, that's a that's conjecture and I've. Also another idea was possibly making one of these deletion plugins that had a config map that listed custom resource definitions.

A

It said, okay, just list a bunch of CR DS, so volume snapshots, volume, snapshot, contents, I. Think the CNS thing is C NS register volume, I think that's what it's called but define a bunch of CR DS and then have this deletion plug-in just go: delete CR DS that mention that are labeled with the backup name.

A

So we just make one: the user could define a list of CR DS that would need to go get deleted and then we don't have to define individual plug-ins for CR knees, but these are ideas to toss out there and I'm I, guess I'm, asking generally for the community. If this makes sense. I think it makes sense for Valero architectural II and probably for VMware, so I'm curious, I, think Vishnu. You had commented I think for open a PS.

A

The volume snapshot er makes sense, so this probably is a little orthogonal like it doesn't really matter there, but I'm curious about other people's thoughts.

C

I think the config map approach is simple enough.

C

To implement and then and but the only downside there is, we are only handling deletion, CR DS, any other sort of custom logic that took custom cleanup logic that may appear in the future use case.

C

We might not handle that. Well,.

A

To clarify I would say that this would be one one plugin, that it would be one deletion type plug-in type that takes a config map, so anyone so we would implement the plug-in architecture generally and then we would bundle a a plug-in that knew how to delete CR DS instead of implementing one plug-in per CRD, just as like a quality-of-life convenience measure. If that was my thinking, yeah.

C

I think that's that's a good starting point and we can start with that and, as needs approach rise, we can improvise on that yeah.

A

And then folks, you have non kubernetes resources, so if they need to go clean up something in their storage system or in a cloud provider or somewhere else outside of kubernetes, they have that hook.

A

H

He deserve ich Newt the conflict map concept with the CRTC need for cleanup really make sense for artistic abilities based applications who had a ciass, usually so the custom logic can go into the controller of the CRS as a ticket event. I mean the code can be written to other cleanups, but yeah as non kubernetes has a say like this can act as an example to extend them. This is this for sure helpful. This convict, Mac CI.

E

H

Be little thought of like not only for delete contact map kind of thing, but even for let's say for even is syncing up to the object store and then again, I mean back into the object, store and thinking back. The restore process I mean.

E

Not even restore the.

H

Backups in controller can think of sinking into the clusters.

F

Hey guys, um this is away from from the redhead camp, I I'm, sorry, I, don't have a lot of understanding in the problem we are solving, but um ah has anyone looked at from slapping like owner references on the fly to DD sources created by the arm by the plug-in, so that when the backup gets deleted, we get that automatically garbage collected by cube system itself.

G

So in at least some of our cases so like the the plug-in that we're building is really talking to something that's outside kubernetes, to store the snapshots so the same way the volume snapshot or is talking directly to EBS and the snapshot isn't actually represented inside of kubernetes. So I think having this explicit delete action is is useful and we can also do things like not just necessarily clean up, see ours, but also do any.

E

G

Of clean up logic that might have to happen there.

C

Another scenario where the ownership references wouldn't work is a fluster migration. When your cluster is nuked all your. If you delete all CR CR DS that that are there, you will not be able to restore, and all your references in your backup object store will all be dangling.

C

F

F

um I, okay, yeah, I, I'm, sorry I! Don't follow that last part of how you beget dangling owner references but the custom inability of running custom logic with the owner. References make sense. So thanks yeah.

G

So if you have a backup right, the backup has the current Blumer backup has a list of snapshot IDs in it and if you say to Valero, delete this backup. Valero goes in and pulls the backup right. I already pulled the backup, but it has all the snapshot IDs that needed to be deleted. Those are not necessarily surfaced into kubernetes, so even with CI CSI snapshots like if we recreate it. If you've deleted the cluster, and then you start using a repository someplace else. The CSI snapshots don't automatically surface as CR DS.

F

Yeah, that makes sense Thanks.

G

Actually, that does kind of beg the question: if we do, if we're deleting a backup, does that mean we have to statically provision each CRD, each CSI snapshot and then delete it? That's.

A

What that's, what we do right, ashish.

C

When we delete the backup so.

A

When we, when we sync for CSI, we pull in each volume snapshot content object, so when CSI is orders enabled we actually build it. Pull in the volume snapshot content so that we have a pointer to the actual snapshot so that when we delete we can clean it up. We go change. We go change the deletion policy so that Valero doesn't fill up your storage so yeah. We do some of that. On the CSI side, we do some of that accounting, yeah.

C

We want the snapshot saw on the backup to persist accidental deletions, but we want to clean up when somebody really wants to clean up so.

A

Yeah and in to clarify this is for when Valero is doing garbage collection and when somebody explicitly says Valero backup delete. So if you do cube ztl.

A

Backup, delete or I forget the order of commands and you CTL, but if you delete the backup with you CTL, this would not necessarily get invoked, because if you did the owner references, they would get deleted, but the this plug-in would not get invoked because it's out out of order, if we get it as webhooks, we might be able to get it. But that's currently, not in the Valero design. We would have to revisit how we do stuff.

A

But I'm not hearing any major objections to doing this, so I think we'll have to talk like Stephanie will have to talk about timelines for when this is necessary.

A

But I think this is mostly in addition and won't be a breaking change to Valero.

I

One thing I would add to this is that you know consider using meta data instead of a consume trap. If you label things, then you can also just filter on labels and you you're not tied to the fact that now you have something that you have to maintain sync on there's somebody at exact a big map. Perfect, it's modified! A label match, wouldn't actually cost you as much the other one that is interesting is in CR DS.

I

You can specify that CR D is our part of a group, so you can do a cubic it'll get group name and you would get all of the resources to find within that group. Cluster API implemented this and it's actually pretty mean it'll get closer API you get, but the result of effectively like a cube can I'll get all but specific to cluster API concede would see machines.

I

You would see clusters you'd, see, cube, am control, planes, those sorts of things just on that one output, which I think get you pretty close to where you're trying to get I.

A

That's a good point: I think the difference here would be that we would want anyone to register so in I think Dave's case we're trying to register stuff. That's outside of Valero. It's it's part of the CNS like the VMware CNS stuff, and it would be in a separate group from Valero IO I said: oh, so we wouldn't be able to retro actively like group them I think um so we kind of want anybody to jump in and participate yeah.

I

It'd be kind of interesting to see the community group on like backup.

A

As a group, or something like that right, if we could, if we can write a virtual group, that would be cool, that's but that's kind of why I suggested a config map. But if we can make a label that said like delete our like, we could label the CRT or something that that may be better than a config map. So that's a good point.

C

It was bad the label selector to say, get all resources with this label and delete them. I, don't know if, if we are being progressive with such a delete, we can yeah.

A

We could move that to a design discussion, but I think this was more of a see if there's any early objections, but this this would be totally optional.

A

Nobody has to participate in it and and we could move some of them- velaro internals around to use it. But that's that's a good point. Duffy I think that would be useful at the very least if we could do a specific label that might be better than config map that we would have to sync around and people edit well.

G

We're gonna have two things right, I mean so the plug-in API itself is going to say, hey, you know, here's this object, then I need you to clean up and then plugins could be based on different things. Right.

A

Right, we could make it plugin that did the label select or people could make their own plug-in. That does whatever right. You could bundle it. You couldn't make your own bundle of plugins. That includes a deletion plug-in that does who knows what yeah.

G

And I would say that we probably want to keep it very targeted, so that Valero is explicitly saying, delete. This snapshot not delete all the snapshots that happen to have a label X, at least from the API point of view, because that's scary, yeah.

I

Yeah I think I'm more looking at like how we aggregate the snapshots outside of having to maintain a list sure if there's a way to leverage metadata to create that list or to understand that list. That was probably a better implementations and trying to actually use a good thing to have a solid object. Yeah.

D

I

But yeah I read like delete all something that you know we'll.

A

Always remember, you know, like yep yep yep Valero tends to the the the design has tended to be not to delete stuff but yeah. Even our current snapshot. Delete stuff is pretty targeted so and my thinking with config map was all see area all instances of the CR D with the Valero backup name equals the specific backup name, that's being deleted, so it wasn't, it wasn't all CR DS. It was these things but I think for now we can table this one and move on to other other topics and I moved this one out.

A

Just so, I wasn't jumping in in the middle I'm. Gonna make this one real, quick. The restore hooks design is very close. The last thing we had last week was a discussion on and it containers versus how it's currently working and really I just want to ask that the maintainer take another pass on this and that's all I had and then I was gonna pass it off to Duffy. You wanted to talk about winters for Valero yeah.

I

I was basically an update on the some of the things we've been working on. Let me share my screen here.

I

Technical difficulties so once again, okay.

I

All right, so we've been working on a few things. This is a hack MD that is tracking this stuff and it's linked in The, Associated PRS.

I

There are three pr's up and right now that we're working on the first one is a the first and simplest is the updating, Jekyll 2 for 100, and that one's I think in a pretty good state, so probably want to get that one reviewed and merged in, but this basically just updating the packages are moving in two for one, oh looks like the builds are about three times faster, which is pretty awesome. The second PR is image handling, and the first part of this is basically just decoupling.

I

The building of the image from picking the image in the make file and the way that that works is that you would have in your make in your McDonald's that, having instead of make shell using make image as a dependency, it would use make build and present it as an ultimate NC, and what that does is it actually has some logic built into it to determine one or three things?

I

If we already have a cash local image, that is in line with the content of the docker file in this local checkout, then we'll just use that cash to image. If there is a change in that docker file that hasn't been committed, then we'll build a new image based on that docker file, the Builder, the image builder docker file and the third one is.

I

Escapes me at the moment, but it's all on the in the commit, so it might be good to get some feedback on that see if that makes sense folks, but it basically really shortens the time to being able to iterate using the containerized build environment.

I

The third one is the linter and the linter. I think it's probably the more interesting change for everybody. First thing I wanted to point out was that the linter is completely configurable via a configuration file which would be hosted in the root of the Valero directory. It's called go legacy IMO, and it has the ability to configure each of the linters well pretty pretty extensively right and so by default, we're not electing to turn all of the linters on, but we, but we have the ability to turn on things like go.

I

Second, and a number of other linters that are supported by this upstream project goal in CI. Flint is an upstream project that is like a lot of folks are using to do linting for their billing projects, and it actually has a pretty good output for, like a ton of different, a ton of different plugins. A demonstration of that one. Real quick.

I

There are three: there are two targets that we initially started with. One is make lint, which is a containerized version of the linter, and then there's also make local lint, which were which inquire, which implies that you have Galang CI, lint, command-line, locally and you'll just be able to use that same configuration and everything locally by default. We run these linters and the output that to the to the output. So you can see what litters are run against the repository.

I

Let's go, set: go cons to go format, go important on pram. These are some of these are duplicates, and so, if we think about like make a verify work, whether it make verified, does right now we're kind of redoing.

I

Some of that with with this output, and then we've also implemented a linters argument, so you could have it so that it will run only a particular winter, which is great if you're like iterating on, like you know, specific changes or if you wanted to like even try to do the auto fix, like some of these lenders, have an auto fix capability like go imports and that sort of stuff. This would give you the ability to speak target just that one linter and then pass the the fix argument into it.

I

That's where we're at on those three things and so yeah I think it's probably be good to get some time to, like you know, to review what we've got so far and see if we're on the right track or if there's any like I'll sending feedback on those three yars.

E

Hey Tuffy: is there any way to integrate that winter? The the loop the winter in the specific rules we're going to use with IDs with.

I

Ids, what do you mean.

E

I

Yeah configure.

E

E

I

Yeah well link see. I definitely has like the project itself. Has a ton of implementations for different IDs I think they have a vias code, one if there's like I've been plugin for it, but basically it's just a go binary that, like you know, provides you a way to config, like all of the lenders that are supported by that guy and so yeah. You could totally integrate it locally. I'll.

E

Search for that.

A

That is awesome, Duffy team, the the build image thing is something that's been burning, my personal mental to-do list for a long time, because it always bugged me that our our process would build the image over and over so yeah and I. Think your next steps we're like getting this all running locally and then moving it into CI or getting it all running locally and for the linters we can start fixing some stuff and ignoring some stuff if we deem it ignore balandin, getting it into CI, exactly yeah. The next part.

I

Of this is gonna, be basically taking a little interrupts put and then opening issues based on it and like take it a Hacket like getting some of the a little low hanging fruit smash down and then, if we and then figuring out like what the what the right configuration is for to like ignore generated files and that and that sort of stuff within the project, but I, don't once we actually get that you know and a pretty good, steady state, then we can talk about who we get into CI and how and how that mechanism of work but I think that yeah at the moment, it's I don't think it's ready here, nearly ready for CI, that's ready for sure play with it.

I

Okay and then the other one on the image handling. I do have another piece to this which I'd like to see happen, and that was actually a request by Steve Chris, who has requested that like is there a way for us to actually cache the build image like in a row in the registry like in the Valera registry and that way. If we, if we see that, there's no changes to the docker file, then we could just use the one.

I

That's there, and so I've been starting down the design path on that what I've been thinking about doing is actually having it so that on them on a master build. We build a tagged version of the malarial builder image based on a checksum of the docker file, and if we see that the checksum of the docker file matches one that we have an existing tag for that, we just pull it.

I

And if we don't, then we build it and if there's a you know outstanding change to local doctor fire, then we build it and then I think would catch most of those cases. And that way, if you were on an earlier branch like editor panic or a release or whatever, and we would likely already have a cache for that builder image that built that particular release and if not, then we would know how to build it from the local docker file. So it's kind of backwards compatible by default.

A

Yeah I think that would save a ton of time on our PR build yeah. That's that part for me is huge.

A

I'll follow first review. This week's it awesome thanks cool. Thank you. Any other comments.

C

I just made a comment on the winter PR I think we can enable it for CI so that no new letter errors were added as part of PRS and we can ignore the existing ones and, if folks feel inclined, they may opportunistically fix some small small errors. That's.

I

A great point: yeah they do have a flag for like Han or whatever, which will only show- and it's actually kind of smart it'll only show things of the winter caught between the current commits and the one previous which is wild. So it'll only show like things that you introduced in this commit if it's based on the DIF yeah reflect.

J

I

Maybe I'll make that a separate PR well like yeah, I, guess I guess pushing that in as part of this initial commit would make more sense. Yeah, okay, cool.

A

A

Alright! Next up we have Vishnu so we're talking about the progress on backup/restore operations, which I think is pretty close and also rustic snapshot, differential, sighs.

H

And just before starting my discussion, I would like thank to Steve for starting the value role being part of a temple in the community and the recent interaction I had with him. Really I learned a lot from him about whether or not we done things thanks team for all the help.

K

H

It's about currently resting stores makeup shows the total bytes and the bytes to back up which tells like what is the volume size if I restored this particular backup right. But the point is I want to ask.

H

Size from the previous snapshot that helps like, if I give this snapshot, how much of the space actually gets reclaimed coming back up, location, object, storage, so that.

H

Will help like for anyone to know what backup to delete or like so basically, each backup is the one taking a lot of space on the object location. That kind of information will be helpful. Any thoughts.

E

C

For starters, all Bolero backups are like independent and there are. They are full backups right, so they are not incremental. So not sure what you mean by which backup totally.

A

I think this is in rustic right, so rustic rustic snapshots are definitely they definitely have a chain. If you use okay, if you use rustic, those would be incremental, even if the data is full.

A

Associated with it may have been.

H

But the point is they have a duplication of the so kind of.

H

H

H

H

H

H

Which tells the differential the size from the beginning snapshot I need to explore, but I just want to see if well.

G

Are you planning to store that in the Valero backup, because that may not be static that may change, depending on what other snapshots exist?.

H

Kerning, so basically it's amount of the changes which happens. It can change, I mean Changez, helping yeah.

G

So, for example like if you delete some snapshots that this one depends on, but it keeps those blocks the amount of unique data for this snapshot will grow.

H

H

It's already dynami from that way, I think. Okay, they.

A

Said I think this yeah I think this was gonna be stored in this new volume, backup CR, if I'm understanding you correctly des that you're gonna, add a field to this. Is that correct.

H

Yeah again and thought of where it should come. First, my machines about like how their istic the snap shorts differences sizes will come to know. Okay, okay,.

H

G

Would put that one off because that's actually a really, it seems simple, but it's a really ugly topic and we've got other things happening under the covers as well like, for example, in the beasts or plug-in. We currently have a local snapshot and a remote snapshot. So we take a local snapshot.

G

Then we copy it off to as three right now we delete the local snapshot immediately, but we've thought about having policies where we can retain those for a while, and you know, depending on what point in time, you're at or maybe a local snapshot plus remote snapshot that you could reclaim it space in different places.

G

So it's tricky I.

A

Also, just copied I have not read this in depth, but rustic has a document on there posit or II, and data format that you could take a look at. They have commands and there's like rustic prune and there's commands that can do maintenance on the repository and I think that alters the math. That's done as well. It helps rid. You know clean up some of these. These things and I think that would alter what what some of these differential numbers you would get so I, don't think.

A

I have a clear answer for you right now and honestly: I have not I have not looked at this closely, so I don't think I I can give you a good answer. Here looks like they also have a.

C

Other question that I would have question / concern I would have is what happens when you are trying to either delete our d-do from rustic, while somebody requests at a store like what happens then, because we've, if we have multiple entities acting on the same rustic repository at the same time, we might need that might lead to undesirable behavior.

B

Just to jump in on that, we do have some loc management, so rustic repositories have the concept of locs exclusive and non exclusive locks depending on the operation. So when we're doing a arrested prune, which is actually deleting blocks of data from the repository there's, an exclusive lock on the repository, which means no other rustic operations can go on so Blair shouldn't be respecting that. But it's definitely something to keep in mind.

A

Hey I think, even if you, even if you went and got the key and you operated directly on it, that lock would be put in place right. Is that right, great yeah.

B

There's a I mean the the rustic COI, the rustic binary rights, a lock file into the repository, so there's a there's, a file in the s3 bucket. That essentially says this. Repo is locked exclusively and and no other operations will take effect, velaro just kind of gracefully handles it, meaning it's aware that the lock exists and it doesn't try to run operations so that it doesn't get errors yeah.

A

In Vita so to understand the problem you're trying to solve, are you mostly just trying to communicate?

A

Are you just trying to surface what rustic did in a given backup.

H

Yeah to understand, like I, mean what is a but snapshot or backup is consuming space at the object location at the dual point of time, but III understand that it's very dynamic based on the data coming in and deleting of other snapshots depends on that. So so there it is like for user. I have a lot of snapshots, and what is the story is capacity used as my object location for it.

C

Would you not be able to track the growth of footprint in your object, store, location itself? Wouldn't that give you that answer directly, instead of trying to aggregate all the various components that contribute no.

H

No, actually, you see the overall aggregated value, so it's a I have ten backups and now what is my object? Okay should I know, but what each but your backup is consuming for itself. That is not known.

H

I think I caught the required info, some landing Swanage, the next one is- might appear. I incorporated Steve's comments.

H

I removed the volume- that's not dog, snap short, the need from the code, the design-wise, so it now only have the volume peg in backups here and what Stanford is not there I remove that? Yes, yeah have a look at it. I review comments.

H

Security, but also saying that this and I the impact it can do I mean big turning the road namespace. What all service account.

K

A

All right anything else on this topic.

H

Nothing nothing Thanks.

A

Tony with the update to Jekyll four point: one point: zero. Yes,.

D

The Duff you talked about it already, so it's just to say that I think it's done, but I don't know the website that well and I was hoping to get somebody to review it and make sure there's no changes, because technically it should be like exactly the same and I didn't see any difference between like the online version and the testing version, let's say but I don't know the whole website yet so I was hoping to get somebody to review that a next step would be to focus on trying to migrate Jekyll to hero, which seems to be slightly more difficult.

A

Sure yeah we we will definitely take a look at this and yeah.

C

A

C

Under Jekyll update, I'm, not sure how to do like I've looked at the website and select click through a couple of links and everything looks the same.

C

So I thought you imagine if things are not working connecting read but dissing fix.

A

Yeah, if it's working, my my gut check would be to like try it locally with the the make serve Docs, I, think and then, if our previews are looking good, like I'm I think it looks good and it runs locally, I'm cool with just going with it. Okay, that's my my two cents I'm.

C

Not either yeah the preview looks good and thank I'm going to try and run it locally and let's get this marched cool. Thank you.

A

Okay, anything else and yeah the change to Hugo would be bigger, definitely and the good the good news there would be I know our make serve. Doc's target takes a while because it has to go every time. We we don't build that docker image such that we cache all of the dependencies. So it takes a little bit to get started cuz it downloads every time, but with you go, we could just catch the binary and be done that, at least from that perspective.

A

So, but changing all the stuff would be to perform at and I know we carry some hefty Oh custom website legacy there that that's not really pretty so so that that will be more involved.

A

But thank you for getting that updated and we'll definitely take a look at this soon, because this this one should be fairly easy to close out and then in contributor, shoutouts I took a look through the repos. Unfortunately, we've been slow to get pr's reviewed. We've got a lot of big things going on that the team has been working on so apologize that we have been slow to get. Reviews done. Thank you to Vita and mark Campbell and I'm, going to pronounce your name wrong. I'm, sorry Fung for putting in design documents on.

A

Especially so, for we've got just let's see here: we've got design documents for restore hooks progress on the volume snapchatters and its new one, on the specific order on within a given type, which we need to take a look at. Thank you for getting those in, and hopefully this week we can get back to getting our PRS merged a little bit a little bit quicker pace. So unless there's anything else, give people just a second to unmute if they have anything.

L

Hi this is the shuttle from India and I. Just had some questions about Valero, so I mean can am I audible, yep? Okay, so I have a bare-metal kubernetes cluster, so this might be very basic. I'm sorry, I missed the most of the part in the meeting because of my office timing. So the thing what I was trying to do today is I: have a bare-metal cluster and I just made a bun, 2/4 and I attached a persistent volume to it and I kind of mounted the root directory in the PD.

L

So today what I try to do is I took a backup of the same pod of the entire name space using Valero and I stored it in my mini o bucket. So when I I deleted the name space and when I try to restore the thing I got my pod back, I have got the PV back, but the data inside the PV was not present. I am using rope safe 1.3, as my dynamic storage provision er for my kubernetes cluster.

L

So is there anything that I have missed in the Valero installation steps because of which the data inside the PVC is not backed up or is it not a feature of Valero, yet I mean I am just trying to evaluate as much as I can. So that's that's my question, because I really have this use case in my company that we need to store the data inside the pods.

A

So it sounds like my. My guess is that the there is no to my knowledge, there is no rooks F plugin for managing snapshots on rockin Seth. So you may want to try installing with rustic, to grab that local data velaro relies on plugins to grab data out of the persistent volumes. Otherwise it use it. It relies on rustic to take file system level snapshots actually.

L

I used use stick while installation.

D

Volumes because the volume is it.

G

The pods need to be tagged, also correct, I.

L

Just hope I mean I'm, sorry if I might have mistaken novio, but I just did bolero backup create include backup, name include namespaces and my names.

G

For rustic, you need to tag the volumes on the pods is being backed up with rested, there's a in the dock that is I'd.

A

Send again, there's a there's, a document on using rustic and there's actually a we're going to be adding this in Valera 1.5 that we can default to including everything with rustic, but for now on on the pod, you have to I forget the label off the top of my head, but you have to say backup.

A

Thank you, Steve in the chat Steve linked to the domestic documentation.

L

A

L

For that yeah yeah.

A

There's an annotation that you have to add to.

L

The pod, so if I do this, my data inside the Peavey's will get backed up because I really have a huge production which we need to which were for which we need to set up some backup policies. So I am relying on Valero a lot so I'll surely try that and I'll obviously put it in the slack Channel as I face any difficulties.

L

Thank you. Thank you. So much yeah thank.

M

You just one quick thing: just in with work stuff, you also might be able to use to CSI snapshots, because that's a CSI, plug-in I haven't done that myself with work staff, but thinking that ought to be an option there as well.

L

Snapshot feature by chef, but I contacted the developers on slack, but they were saying that the project is still in v1 alpha. One phase and the complete implementation is different from v1 beta, one on which they are working right now, so I wait and the evaluation won't be of much use as of now, so they are planning to stabilize it in the next release. So I'm just waiting for that work for the backup thing. I, really trust on well arrow and I want to achieve the data inside the PVC backup thing so yeah.

C

Self is still using the v1 alpha. One snapshot, api's, okay,.

A

They're not on the beta okay,.

L

Just for this thing, when I I mean what I'm doing is I'm just running a mean IO docker container on my on my NFS, so do I need to I mean I just saw this setting of one arrow snapshot location, so is there any I mean I'm, not using GCC or anything. It's just your pure bare metal, so do I need to configure any default snapshot, location or anything like that. I mean I, just got confused between.

L

There was one talk and it said that I had to I mean create a snapshot, location name like EU, something I can't recollect it right now. So is there any talk around that I mean if it not too much. Can you I mean I'd acting in the you know, evaluate well arrow with menial dock that you already have that you can create a default snapshot. Location thing. I.

C

Would say the odds are welcome if you have docks improvements that you want to suggest a please feel free to open a PR on on the repo. If you need, you need information on how to do that, we can help you with that.

C

Exactly I mean it would be greater for you.

L

Thank you so much.

A

Okay, well, thank you for the questions and thanks everyone for attending and we'll see you all next week, I'm great week, everyone bye.

J