VMware Open Source, 21 Jul 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Velero Community Meeting / Open Discussion - July 21, 2020

Description

July 21, 2020

Status Updates

ashish
Getting Release 1.4.2 released and testing the release.
PR reviews and issue investigation
PR to add metrics from restic backup 2719

nrb
PR reviews
Breakdown of restore hooks work
Merged insensitive language PRs
Added design doc for DeleteAction plugins
I have a couple small PRs ready for review: https://github.com/vmware-tanzu/velero/pull/2735 & https://github.com/vmware-tanzu/velero/pull/2705

carlisia
Community support last week
We now have a BSL controller
Wrapping up conversion of serverStatusRequest to use controller-runtime
Fixing a bug on the verify script
We started filling out our FAQ page: https://github.com/vmware-tanzu/velero/wiki/FAQ

stephanie
restore hooks product requirements reviews (https://github.com/vmware-tanzu/velero/pull/2699)

Discussion Topics
stephanie
discussion restore hooks use case #7 data quiescing (https://github.com/vmware-tanzu/velero/pull/2699/files#diff-1e4163f56a0d34eb56295c1901279753)
dave
DeleteAction discussion
Started as a high level delete, but wanted a DeleteItemAction
Act on items in the backup tarball.
Options:
Pull down the whole tarball and look through it
Add index of things that BackupItemActions created
Reminder to let us know if you’re using any Velero code at https://github.com/vmware-tanzu/velero/issues/2654

Community Shoutouts
Phuong for testing out the main branch and reminding us we need to get update instructions documented
@kathpeony: Add backupValidationFailureTotal to metrics
@fvsqr: StorageGrid compatibility
@laverya: k8s 1.18 import
@bryanro92: [FIX] add --bucket param to workload identity install
@stephanwehr: add support for incremental snapshots of Azure disks

A

Hi everyone and uh welcome to this week's episode of the valero community meeting and open discussion today is july, 21st, 2020 and per usual we're going to start with some status updates and then dive into the discussion topics. First off we have ashish.

B

Hey everyone uh for me. Last week I was getting the 1.4.2 release, tested and all good to go, and I think we cut the release or we did cut the release last week. So if you got a chance, please try it out or please try it out, because it has a critical bug fix for uh with respect to backup and the store of crds uh other than that. I was doing pr reviews and github issue investigations.

B

um Don't have much to report on those. Yet one of the things that I found missing while investigation was metrics related to rustic, so I put up a pr to help us add more metrics into uh rustic. So that is that we are linked in linked in the hack md. So please to take a look. If you are interested other than that, I don't have anything else on my list.

A

uh The metrics here do they tie into the progress report that we've been working on as well.

B

uh No, these are metrics uh like prometheus metrics exposed while backups are running or while the server is running backups and restores. We have a metrics inside of core valero, but we don't have anything from rustic, so we are pretty much flying blind there. If something happens, we just have to dig in and come up with a hypothesis and try to prove it. So these metrics kind of give us some visibility. There.

A

Okay and these would be uh stored, so they can be historic metrics as well, not just current metrics.

B

Yes, you would need to have prometheus or something like mostly prometheus running to which is like a time series database for matrix.

A

Awesome any questions comments on this.

A

All right, thank you. Ashish!

A

Next up we have nolan.

C

I also working on pr reviews. um We got the restore hooks. I I not uh not specifically added there, but I that is not what I meant to link there, um but I meant to link a uh a zen hub um view of the work, but um basically all issu uh all issues labeled with the restore hook label now.

C

Are the broken down work that we have from the restore hooks design document um that we can that we can move on work? Now, um that's the first pr that I'm kind of trying to excuse me trying to roll with um there's a few like pre-requisite, work things that could be rolled into um that can be rolled into the larger work um stuff like adding, adding the annotation definitions and modifying the um the uh restore spec and things like that.

C

um But I did that breakdown yesterday, um so folks that are interested in grabbing that um those pr's are up there now are those issues are up there now and they're labeled with the restore hook uh label so there it should be easy to find. If you want to grab it, uh you can grab one and uh assign yourself, um and uh maybe ping ping me on slack to let me know you're you're working on it, uh but I'm checking the github pretty regularly.

C

um We also merged the insensitive language pr's on uh late friday, and what that means in her for people on call is um the master branch is no longer being used. uh We are now using main, so uh the master branch is still around for right. Now uh but it will rapidly be going out of date.

C

We're not merging new prs into that, um and uh documents will be on a documentation will be on the main branch, um there's one file for um the style guide and that I have a pr open to move it and get rid of master completely, but we got the helm chart and the plugins moved as well, so that was all uh wrapped up friday.

C

um Also. Last week I started a design document for the delete action plugins, so this will delete any resources. It basically allows allows a plug-in author to take an action when valero deletes a valero backup. So please take a look at that design document and make comments it's in draft right now, but definitely looking for feedback there, and then I have two small prs um that are pretty they. I think they should be easy to review.

C

One of them is that moving that style guide and one of them is uh getting the 1.4.2 change log merged into main um so and I'm on computer community support this week.

C

So I think that's most of what I got for.

A

This meeting awesome. Thank you um any questions comments for nola. I just wanted to comment that I'll probably take one of those restore hooks issues, small ones and try it out today.

C

Cool thanks awesome, thank you, um yeah and, and um I comment I also commented on the design doc pr with the zenhub web, app link that shows like um dependencies and stuff that I mapped out there um so that has uh yeah. I will fix that this uh link in just a sec, but um this has um the zen hub, should have um blockers and things like that.

C

Yeah yeah, that's that's uh the um looking at that. That's the um the link with the browser plug-in so I'll fix that while we're getting the next uh update, but thank you andrew very much appreciate that.

A

All right, any other questions comments for nolan.

A

Okay. uh Next up, we have kalisia.

D

Hello, um I will look at those two pr's right after this nolan uh I was on community support. Last week I worked on oh so I just wanted to let everyone know we have. We do have a backup, storage, location controller. Now that constantly checks. If the look store storage is valid or not,.

D

So so, right now, I'm working on converting the server status request controller to control runtime. It's done. I just have one pesky test to figure out. I should have a pr for that today. I I so I have. Let me see.

D

One two, two more uh resort uh controllers that we agreed to um to do for 1.5. um So another thing too, in regards to this controller conversion. Is that once these controllers are all converted, we don't, we won't need to use the client go stuff it. So that means we won't have to generate them, which means all pretty much all of the the relevant ci uh make targets that we have is they're going to run much faster.

D

I don't know how it is for for everybody else. For me, my machine it takes forever.

D

Every time I have to run, make updates make verify, make ci it just takes forever because it just has to generate all of that. ah So I'm looking forward to that um what else? So there is uh so I helped someone yesterday and still helping him today.

D

There is a tiny bug on the generate script that I introduced, but I fixed I mean I found out what it is. It's not a much of a big deal, but I'm also verifying the behavior for the for the verify make targets because as it is now, I don't know. If I introduced this or not.

D

I I need to rob to check out a branch before my introduction to the to the cube uh cube builder stuff, because I I touched all of these script files, but the way it is now when we, when we run make updates, we have to get add the files, because otherwise the git verify would will check out the crds file. I don't know if it was if he was always like this or not that's what I'm trying to figure out.

D

So if you don't get, if you don't get ad, you get into a loop that you run, make ci ci, verifies checks out the crds dot go file and then tells you to make updates. You run make update. You generate that file and then you just get into this loop anyway, and this takes forever so gosh, it's it's so super simple, but just running this.

D

These big targets, because it generates all of those files, it's kind of making me upset um and and and we started filling out an faq page, and everybody is also welcome to contribute to that. That's how I have.

A

Awesome. Thank you, alyssia. Any questions, comments.

E

Thanks for starting that faq, probably soon it's great.

B

I already said you have a link to the pr to fix the verify thing, I'm curious to know what the fix is or why the problem exists in the first place,.

D

Yeah, I have a tiny bug which was. I was the the the check out. The git checkout was pointed to to the basis where the manifests are instead of the crds.go file, which is.

F

D

Yeah, that's it. I don't have a pr, because I'm.

G

Trying to verify.

D

If the scripts, the way they are now is the same as they were before, I touched them.

B

D

Trying to figure out the behavior I mean one, so I already have one solution that I have for. This is just to add text to this to the when you run, make ci and it tells you oh run make updates.

D

I I'm going to add, make a run, make update and get add the generated files. Just so people.

B

D

B

That's that's what is expected right, so people update and then commit the changes, and that way, next time verify runs. It will detect no changes and it should be fine. So.

D

This is that's yeah, that's what I want to confirm that this is the right, the the the normal behavior, but if we don't, I think if we don't say something like get add your files. If people don't yeah, the files they're like what's happening is yeah.

B

Yeah just saying make god make updated, not make them commit that those changes. It's yeah. I get that okay.

D

Yeah so, but I don't have the there's definitely going to be a pr to fix the path it may be just this additional uh instruction. I think that that might be it. I just want to be super sure that I didn't introduce any anything else. I mean I didn't introduce anything that changed the behavior.

G

I had a question on the faq page um so for for this page, what is the relationship between where we're doing faqs here and the faqs on the the doc site.

D

And how like what goes where that's a great question, um so we talked about this before and I think I was trying to make a case and I think in the pressure everybody agreed that it's a lot easier to dump text on this wiki. Then, on the website, the website you have to open the pr have the pr review the web have appeared merged yeah. um So the the intention for this is is questions that are very useful or very frequent.

D

On the kubernetes slack that we get from users, it would be helpful for us to just dump them here real quickly, um and so what do we do with the faq page? That's on the website. That is the question. I think we should put those those uh questions and answers here as well. Keep the link, keep that faq on the table of content and just add a link. What do you think.

G

I think I think that would work like keep the one that's in the github wiki as um like the main faq page and then just have a link to it from the doc sites that we use that we're saying yeah. I think that makes sense. The most sense, if whatever is going to be the most up-to-date up continuously updated, would be the best place.

D

To link I I can, I can scrap the the questions and answers from that page. Another link can do that.

C

Yeah, I would prefer to just have one.

D

Yeah yeah, we should only have.

A

One sounds great, any other questions, comments for kalisia.

A

All right, uh stephanie.

E

All right, so um just one thing to mention right before we dive into this is uh you'll notice, with, um for example, the rustic annotation improvements, uh as well as the restore hooks that the design was done a little bit before the product requirements and in some cases um you know due to the really fantastic engagement we have from our community members design documents, maybe in draft phase or kind of started, but the goal of the product requirements.

E

um Markdown files that we're creating is to make sure that when we get to the point, where we've we're scoping the work uh that we have the product requirements document for reference on what the scope is for that specific epic and if we need to create an epic for some future scope, uh we'll be tracking it. That way um on the zen hub.

E

Just wanted to make sure to let everybody know um how what the markdown final purpose is, and the purpose of those product requirements, documents and also in the future, we're moving towards trying to get the product requirements ahead of the design documents. So just just to let everyone know that, but on the topic of restore hooks specifically. um So we've got the document. uh The draft was created uh nolan thanks very much for the review that you provided.

E

We also uh had a review from some internal folks here at vmware, and we want to make sure that we let everyone know that this doc is available for review. um It's linked to the hackmd notes. So please feel free to take a look comment and ask questions.

E

You'll also see a lot of the comments in the markdown file itself that I'll uh talk about in a little more detail during the discussion section, uh because there are one or two use cases that we just need to kind of um touch on this morning.

E

That's it for my status.

A

Perfect and you, you wanted to do a deep dive into this.

E

Yeah, if we're ready.

A

uh Yeah any questions comments before we do that.

B

I was going to say I have been meaning to look at this, but I haven't had a chance to, but I will today, but this deep dive is also going to be useful.

C

And I I also want to say, like these, these use cases are going to be super helpful for like clarifying like what we're trying to do with things like what we're trying to accomplish with stuff. I also don't want to discourage anyone from in the community from submitting a design document if they've got something they really want to do like um it's not meant to be a barrier for community folks.

C

This is this is meant to just clarify things we're trying to accomplish. um So we're not going to say if you, if you submit a design document, and we don't have document if it's for something on the roadmap and we don't have use cases, we're not going to come back and say: hey, wait for use cases, um that's a great.

E

C

Put that out there.

E

Excellent point: um we always encourage people to submit design documents. uh We also encourage uh anyone to suggest additional use cases so that, even if they're not tied to an epic, that's in scope for a release, we want to make sure that we give you a platform uh to voice. You know those designs and to bring up any use cases in in whatever format you know you prefer design first or use case first and then we'll work to get them scoped into releases.

E

Also um on the topic of the design dock, uh I also want to make sure you know that the use cases are like no one said the. What and as far as the, how, in terms of how the record the community recommends, use cases or design docs can be solved is something we definitely always want to keep open. As.

A

Well, all right so stephanie you want to do a deep dive into this. uh Do you want to share your screen or.

E

I'm okay, if you uh want to share uh the link, actually the link that uh was provided right next to that section, I think, goes to the comment yeah and then we just need to go down to use case seven. I think it's further down here yeah. I wanted to take a look at that comment.

E

Version, though, share my screen. If that's all right.

A

Sure, okay, there you go.

E

E

All right so, um let's see for use case seven. Let me pull this up.

E

All right, so um we had a use case documented that was uh related to unquestioning acquaintancing of uh data backup, and I wanted to let the team know. You know we had a couple of comments here.

E

It um seems like there was some feedback as to the clarity of the use case itself, and I had a discussion with uh one of the internal vmware teams earlier this morning, um where, essentially, the ask is to provide classing at the container level for the restore hook, but I wanted to make sure that I brought this up as a point of discussion so that if there's any questions or any additional comments, you know we can all talk about it here um so nolan. I saw that you had a couple comments here.

E

Maybe we can start um with anything that you've got related to this one.

C

Can I can I ask by define, by defining, like questing at the container level, what what.

C

I guess what is that looking for? Is that all the volumes connected to the container um is that.

C

C

Yeah, I guess I guess I'm not sure what quieting at the container level is necessarily looking for. I.

E

Well, I don't want to get too far into the implementation details since we haven't um gone through that exercise yet, but essentially, what we're looking at is um yeah ensuring that anything that could potentially impede application performance if it's not quiet uh is done at the container level, as opposed to like the volume level to ensure optimal performance, but that's, of course, up for debate since it's, since this is still an open use case for discussion, and this also may have been resolved to some extent in the design document itself.

F

Are we talking about the restore hook here or the backup hook.

E

We're talking about the restore hook, so this does carry a dependency dave and I'm glad you mentioned that with backup operations that if we did uh have the restore functions um you know. Basically, if we had quieting and unquiesing functions to improve application performance for backup and restore operations, there would be probably a dependency at the backup level too. So.

F

This is the uh the sql server team that you're talking to.

E

um It is not the sql server team unless that's the task for kate pate's team.

F

Okay, so um one of the things I wanted to see us moving towards is um you know we have like persistent services where we have app operator driven or we have operator driven applications and really the application needs to quiesce itself and have some interaction, because going in and trying to figure out which containers to quiet gets more and more um difficult right. How do you know what the right containers to class? Are you just class? All of them request them kind of randomly and on the restore path?

F

Here, I'm not sure what quiescent means, because what we're currently talking about are volume based, backups and restores so in order to start the pod in order to start the container we have to have restored the data, we can't do it the other way around at least the current design. So we're going to restore the data, start the pod and then yes,.

E

No, you quiet the data prior to the restore event.

F

You can't do that.

E

F

That just doesn't work so implementation.

E

F

Way we're currently doing things that does not work. I don't think.

E

Okay, no problem so dave at the app operator level, then, um in that context, that would be a function independent of valero and this could be d scoped from the valero requirements.

F

Well, something that below should be able to call, and that was a direction we wanted to go into with, like the asteroid, where we start looking at more object, oriented. um That's.

E

What I was trying to get to okay, I was just trying to make sure, because this topic came up uh during the last several community discussions pretty frequently and the reason I keep raising it and servicing. It is because we want to make sure that we address this use case and that we come to agreement on the community meeting as to whether we scope it as part of any backup improvements, tied to restore operations or restore operations or if valero in the future could call an external service. And uh so so, I think, that's great.

H

I have a little bit uh comment on that area a little bit because I think dave what the scenario dave just mentioned is actually resort to a new.

H

um When you you scale down the um to restore your first scale down, you store the data on the pvc first and then you restore the part right so that that is one scenario and then there's another scenario that might require create data because, for example, if we don't scale down, but we want to restore the data to the original part that is currently running the application and they the user, don't want to scale it down.

H

uh I don't know whether it's actually possible, but that is something that we uh thinking about was that the customer don't want to don't want to scale down their heart, their deployment, but restoring uh the data live or something like that in that matter. I I try to do that myself and it doesn't work that way. But that is something that I I heard floating on the community about trying to restore a live pot without scaling it down.

H

F

H

Idea, I don't know whether it worked.

F

Well, I think that the trick there so depends on what level we go in. So at the block level. You know if you have a mounted file system, you have a pod, that's running but quiet. So the application has been asked to quiet us, but the operating system is still running underneath of it and you have a mounted block volume and we do like block level restore or overwrite, which we actually don't even have the ability to do that, like csi, for example, doesn't.

A

F

You to override the data volume from a snapshot but assume that you can do that if it's attached to the operating system and you go in behind the operating system and start overriding the blocks on the disk. Usually what happens is the operating system crashes, which means you type down, not only the pod but the whole worker node.

F

If you do it at the file system level, you can do it without crashing the os, but you basically you'll need to restart the application, because the data is changing behind its back and restarting. An application in a pod basically means.

H

I I guess the idea is to floating around, but this small is more on the uh file system based uh pvc and if the file system is being shared, if you have a, I think they have a share mode or something like that. Then it can be mounted by multiple parts and it can be accessible. But I don't know how many uh started vendor out there supporting that mode, because that mode is very dangerous. I'm sharing the same pvc for multiple parts.

F

It's not necessarily dangerous. It's just that all the applications sharing the file system has to participate in sharing. So if you have like a distributed database or a not really distributed, but a database that uses shared storage, those pods have to be aware that things are changing. There's, usually some protocols.

H

Network protocols.

F

H

F

That's easy, but.

H

If you're not playing, we.

F

Can't really do it, that's what if we.

B

So this to me sounds like we are talking about a scenario where we are trying to give the application that is already running a different set of data from what it already has is.

I

B

So that that's a scenario that fellero does not work in, because if whether this detects and a workload already exists, it's going to say it already exists and I'm not going to restore it. It's just going to move.

F

On I think what we could be looking at here is if the application, for example, allows itself to take. You know if you have a backup, if you have a serialized version of say, postgres right, you have post, you know, had a pg dump, you can say the postcards okay, delete all your tables and restore from this uh pg dump, and that's something that you can do without, for example, restarting pods, and so that's that's kind of the direction.

F

I think I see things going in is getting the application more involved in its backup and restore it doesn't have to be. But for some of these features like this, the application really needs to be involved, because it's the one that understands the data format, it's the one that has to know that it's um you know that the data format has changed. Otherwise all you can really do is. Is slam everything back to a a crash, consistent state and restart the app.

C

Right, yeah and and kind of my my lengthy comment here is kind of valero, especially in its current state, um is very, very, very ignorant of individual applications um and without applications participating in in this via the restore hooks, that's kind of what the restore hook. The backup hooks is a way for.

C

uh Someone to allow the application to participate in in the backup- um it's not it's, not super intelligent, but it's it is a way um for them to use stuff like pg dump and things like that and then restore hooks are a way to give an analog to that. So we can get scripts and other utility programs to participate in the restore process.

C

C

This is not going to be something that at least in my mind, the implementation of this. um The goal was end of august of this year. um I don't know that we'll meet that, but the the goal is not going to be like super sophisticated, but the idea is valero itself is not going to be aware of every single application, so the idea is to have a basic building blocks that the applications can be brought into the valero restore process.

C

F

The general idea of the restore hook is not a bad one. I think it's a good good idea. It's just that particular use case of like quiescent and application during the restore process. Doesn't I don't think that will work.

F

But I think the other things you know.

A

F

Other the other goals listed here are good things.

E

I I agree, and I just wanted to make sure that we had that clarification and we communicated it. So I think we're all on the same page sounds good. Thank you. I also agree.

E

All right handing it back to you, jonas. Thank you.

A

There we go mute so yeah awesome. um Thank you. Everyone for for the great discussion here uh really really appreciate it. So uh we have dave who want to talk about the delete action, yeah and.

F

I actually had a uh I put.

A

I had a review.

F

That I did like on friday, but of course I didn't push the publish review button.

F

um So our question as we're looking at this was uh this is delete action, so this will get called on the whole backup, and uh I think we were looking for like delete item action so that we'd get called for each of the items that was part of the backup so that we wouldn't have to go. Because if I read this correctly, this would mean we have to go and look inside. The backup structure find all the things that are ours and then delete them. Is that.

C

That right, that was what I was envisioning. I can share a screen for the uh I can pull up the um just, so we can play along at home um yeah. So um I I envisioned it as a um kind of a bucket that this matched against the backup itself um versus versus versus a deleting um arbitrary kubernetes objects.

C

um That was my original understanding, but so you're you're thinking uh this would be a deletion against any object in kubernetes.

F

Well, not kubernetes, but in the backup so it'd be analogous to like restore item action right so restore item action comes to it for something that's in the back up and says: hey restore this.

C

F

And so I was looking for something that says: hey this thing's in the back up deleted.

B

So, there's a difference in the way we so when we are like restoring or backing up. We deal with each item individually, but when we, when we are deleting a backup, it is just basically deleting valero's knowledge of a backup and not actually deleting the resources. So if deleting a backup meant, we go uh delete all existing resources that are in the backup, uh then it would be.

B

It would make sense to do what you are suggesting, but this delete item action or uh so, but the delete action is kind of extending what we uh extending the changes somebody would have to make into corvillero.

B

To do the cleanup.

F

We were just looking for something analogous to the volume snapshot or delete.

C

Okay, I might have to think about that one a little bit more because I'm I'm thinking so I'm trying to think here on backup delete I'm trying to think of the things that it that it that it specifically deletes, because I think a lot of what so the things I know it specifically deletes.

C

Are it pulls down any volume snapshots in the volume snapshots, dot json.gz file and then invokes the relevant um volume snapshot plug-in to call that, against the the snapshot platform, uh it will clean up any associated, restores kubernetes.

I

C

And any rustic and I well, I think I think I'm not 100 positive on this, because it's been a little bit since I've looked at that code, but I think it just deletes the backup directory in object store, which just gets everything underneath there. So you just say delete that directory and it's.

A

C

C

F

So right I mean so on restore. We currently have a way to figure out like restore item. Action is supposed to get called for each thing. The backup item action has.

C

Snapchated right right because in the restore we loop through everything, I just don't know that we do on delete right.

F

C

That would be that.

F

H

You're gonna have to.

I

Write that code in.

F

Each um backup action plug-in yeah.

C

Yeah or the stock delete action you mean or backup or.

F

Reaction, I'm sorry yeah.

C

F

Write the same: the code to loop through the whole backup looking for our stuff.

C

F

C

My I think, my my what I had in my head was that when you receive a backup, you kind of knew exactly what you had um and you could do it, but you're right in that each each plug-in that had stuff would have to receive everything that was in object, storage or the contents of the backup and then iterate through that so like we may end up having to like create some helpers anyway um to say: here's what's in the back up um so um okay, this is useful feedback.

C

B

Yeah, I think we, the expectation for with which the design was written, was, as as uh you know, what items you added to the backup and you create a delete item action which will call you, and you know what you have to clean up and you would just do it.

C

F

That was my expectation.

C

F

The thing is, I don't know what I added to the backup, because.

C

Unless we change.

F

I

Right now we're getting.

F

Called as backup by an action, so we get called onesie twozies for each one right yeah. So we don't have a coherent view of of our stuff. I mean we can certainly tag it and go looking for it, but I just felt like it was parallel to have a delete item, action, yeah sure.

C

Okay, um yeah and and the reason the the I think, the reason I wasn't going to do a delete. I delete item action was when we were talking. um You were talking about not doing um kubernetes resources but doing um like external resources, so things not directly on on kubernetes, but you're you're gonna have um at least something that you're tying to well. The backup item, actions that you're triggering on will be pvcs right.

F

Right but we're going to return you some data. You know that, for example, keeps track of our snapshot id okay. So that's like this. It's like the psi driver. I mean take the existing way. The csi driver works for it to leaks out, but you know the csi driver on backup returns. Basically, a snapshot id that gets stored in the backup.

C

Gotcha, so what what would you match on as a delete item so like in your replies to what would you match that against yeah.

F

uh Well, I would think we would so I would so I'm not exactly sure, what's stored in the backup, but my thinking would be that if we had a if we had returned data, that's stored in the backup that comes from a backup item action that there should be an option to call a corresponding delete item action with that that uh that data, so the volume snapshot is is you know, is- is pretty much doing what what I'm expecting at the moment right. You call me to do a snapshot.

F

I say here's a handle, you say: okay, thanks fine and you know how we handle what's behind. That is up to the plug-in and the storage system and all the rest of that right um and then at delete time. You come through, and you say: okay.

C

F

Know we're cleaning up, you know we don't need the snapshot anymore. Here's the id and whatever you need to do is whatever you need to do. If we're cleaning up kubernetes resources, that's fine! We're going to s3, that's fine, we're calling veritas or google cloud who cares right.

I

I

Resource selector, I think that's under.

I

C

So from your backup item action you are returning.

C

Another structured it looks like okay, so you're returning an unstructured and valero is just shoving that in the back up, okay, thank you and then I think.

C

Okay, so what what we might have to do, um I don't want to spend everybody's time reading through the data format, but what we, what we may have to do is this. This particular one may have to be a little bit more on the technical side in terms of use case, but you might want to say that this is tied like this. This particular one may have to crack open the the backup yeah yeah, it downloads, the backup and iterates through each thing. In there yeah.

F

But it already does that right, I mean well, no okay, because you have volume. Snapshots are called out as special.

C

Right, those are those are stored. Next to the tarball.

B

F

We we could also add backup, item actions into their own. You know generic file right. Here's all the things we backed up with backup item action since there's no need for backward compatibility right.

B

So here's another thing to kind of extend it further. Would it be how about we allocate a specific directory in the backup tarball for the plugins to write to so, for example, like uh valero, writes the volume snapshots dot, gz file, we give the we let the plugins write their own files uh into the backup and then at the time of backup. When we point we invoke the delete item action, the plugin knows what it read, what it wrote to that particular to its space in the backup and goes and cleans it up.

F

Yeah, so that's that's an option there, um though it's probably good to control, I mean having the framework, you know the the overall valero say you did a backup.

A

F

Action, therefore, if you haven't item action, I'll call, it.

B

F

Given that we didn't so so say, for example, we were at the lead action as it currently stands, so how would we be able to scan through and find our resources we'd have to? Can we download the backup from there is it? Is that already everything's in that.

C

Yes, um if we see we look at backup file format, it would be.

C

um Not in the json file but in inside of the um inside of the tarball. So if you like, you do yeah. If you do valerio backup, download, you'll, see this structure, but inside of it it's resources and then it's um so.

F

That so for, like a delete action plug-in as it currently stands, we get in a backup structure, and that is not the deserialized backup, but it would point to, for example, the backup tarball.

C

And backup I was thinking you would receive um is actually this, like the the um the literal kubernetes backup object, not the whole, not the whole tarvel, but we could. We could flip that and say: do you receive the whole thing I mean.

F

Right, I mean that's just I'm it's just I mean it sounds like that's what I need so okay. If we go down this path, uh I would have to write the stuff. That would, for example, call object. The object, plug-in the object, store plug-in to download the backup file unturned someplace. I need some temporary.

C

F

Yeah then step through it and look for the pieces, so that seems like a lot right right. um So I would say that if we could go down the path of you know parallel to volume snapshot where we start to keep the list of things that were backed up with backup, item action and that's a simple, concise list. Then you don't have to pull the whole tarball down and crack it on the delete side, yeah and yeah.

B

Or we could or valero could download the tarball onto the disc and tell the plugin hey. This is where you can find your backup and the plugin would go inspect the particular folder or directory it wrote to, and if it is there relax on it. If it is not there and say.

H

B

Thing to do and exit.

F

Yeah I mean, I think, that I I think the delete action I mean at the same level, so we currently have like a backup action, restore action, delete action. Would that be.

E

A

F

Is at the very top so these are like we do well. Actually we don't. I mean we can't replace backup action right now. Can we.

C

Backup item action.

F

No backup backup action so you've got delete. Action is as acting on the whole back right.

I

F

So it's like a it, it seems reasonable. I have a delete hook that gets called in general.

C

Right right, that's and that's kind of what I was writing it as.

F

Yeah um but we're gonna write it awful we're gonna each each. You know when we're trying to clean up after backup item action, we're going to have to add a lot of code for each of those uh delete item action, equivalents that make sense.

C

So yeah, so I think what we need to do is revisit this and probably write some use cases here um say what we need is that we want delete, item action here and maybe we we table, delete action for now, um because it really sounds like your use case is delete. Item action. You want something yeah I mean, I mean that's what we need. We need to clean that up. I mean delete item. Action could be implemented in terms of delete action.

C

It's just annoying and everybody's re-implementing it, and um but I get what you're saying and it would be. It would be nice um if valero had some sort of of.

C

What am I trying to say some sort of index of what the what uh plugins created? um So that's like these backup item actions made these things um yeah.

F

Yeah I I guess that currently no one else has really done. Backup, item action, plug-ins right everybody's been concentrating on volume, snapshotter, plug-ins,.

C

um Red hat's done some um the csi. I think I think the the pattern that we're doing has been the vsphere plug-in and csi plug-in now um to to use to use backup item actions as a replacement for the volume snapshotter yeah. um The csi plug-in was kind of the first to use that pattern um and steve kind of prototyped that as a hey.

C

What if um and we just kind of got lucky that we could use the csi drivers to do the deletion there by deleting the csi resources, but in this in this vsphere case, we can't do that um so yeah.

C

um I think what we can do is maybe start start by defining some use cases and then, um specifically, this this item action and maybe see if we revisit from see if we need to make any changes to these restore item, action interfaces or any bookkeeping, we might need to do that. Could shortcut this. um I don't know if we could take the current proposal and just say: hey: here's, the tarball plus an.

F

I mean the thing about: well, I mean so we we can do that and then we can move code up later right, but we're basically we're going to wind up implementing delete item action. Somehow right.

C

Right yeah, I mean if we get the global one, we can in um implement the specific one in terms of that and then clean it up.

C

It's not great.

F

Yeah, so if um yeah, if we do that, then we so so, I think there's two options on the table: one is pull down the whole tarball unt someplace and then rummage through it for stuff to delete and that works without any format changes.

F

And then it would be an optimization to add, like a look aside table of here's things that are specific for this plugin to delete yeah. um But it's not really it's more of an optimization right. I mean the as long as we have enough disk space. You know the tar balls are not that that big.

F

So it's not a huge burden to download and run through it.

B

I think it depends if you have rustic data in there.

F

That's not in the tarball. Is it no.

C

No, that's that's beside it um like we look at.

C

If I should have some backups up here, if it'll load.

B

It's under different folder called rustic.

C

Right, yeah, it's in a completely separate folder.

C

Like the tarball should be yeah, so there's rustic and then there's backups and then backups look like this. So we've got the csi stuff is in its own gz. um The snapshots are in their own little json.gz.

C

The backup itself.

C

F

So how did the csi stuff wind up in its own file? I mean I, you know, I obviously built it, but you know: where does it? Where does that get built from.

C

um That gets built from inside of core valero so that we can upload them and do them do it on sync. So we can like synchronize it across the clusters right.

B

Except yeah, nothing.

C

Else have access.

B

To the storage.

C

Yeah, that's a core valero thing: um we we did that partially because we wanted to be able to do um uh valero described, commands that could see it um and see. um Backups like within your backups. Do you have csi snapshots.

F

Okay, so you probably want to generalize that later on as well, yeah yeah.

C

um These volume backups that's like rustic information.

C

um So, like yes, it's possible, we can put parallel information in here and uh I think steve had added a plugin directory that could go in backups in in the backup directory. No, it is incepted so.

F

It's separate it's at the root, there's plugins, and that was supposed to be where we could store our data.

C

Yeah we're close to the end so yeah I just.

B

Want to make sure we.

C

Care yeah yeah. We should stop here but to to understand the format. That's that's kind of what it looks like, um but, okay, we can. We can probably schedule a separate session for that too yeah. So I'll stop my sharing and let jonas take over that's good.

A

Yeah yeah, please um schedule another meeting and um we can have that a uh be a public discussion as well. I think that would be great. um I just wanna um two things so first one is of course a reminder to. Let us know if you, if you're, using any valero code, um that is being mentioned in the valero public api issue that is pinned to the top of all the issues. If you head over to the blur repo click on issues, that would be the pinned issue.

A

So let us know if you have any um if you are relying on any uh of the valero code in there, because we are changing things. uh The last item are the community. Shout outs: uh you want to do those nolan.

C

Sure let me find my way back to that tab. um First, I want to thank fong for testing out stuff on the main branch um and finding out we needed to get updates for uh the 1.5 release ready, so that resulted in an issue just reminding us. uh We need to get the update document ready. um So thank you very much for playing around with that uh and reminding us. We need to do that.

C

um Also, thank you to kathy uh peony uh for adding this new backup validation failure total metric um next. Thank you to fvsqr, I'm not sure how I'm supposed to pronounce this, but um for uh basically digging into some failures on um pulling down logs from storage grid, basically finding some issues with uh double g-zipping um on storage grid and fixing that also thank you to laver a laria. uh I think andrew is his actual name um uh for updating the kubernetes dependencies to 118.

C

uh I know 119 is around the corner, so we'll need to do this again very quickly, but uh this was a one that touched a lot of stuff. So thank you very much for doing this.

C

Also, thank you to brian rowe 92 for updating the documentation on the gcp plugin with the bucket parameter and, uh lastly, thank you to steven stephenwarr, where for adding this support for incremental snapshots of azure disks. um This is pretty cool and uh allows for incremental snapshotting support on azure, uh and this does work if you take an incremental snapshot and then do a full restore so for users of azure, this is pretty cool and can save you some money.

A

Awesome, thank you nolan and thank you, everyone for contributing and being a part of the blair community. You are all amazing. I hope you have a fantastic rest of the week here. um If you need anything, please hit us up on the kubernetes slack in the valero channel and with that have a fantastic rest of the week. Everyone and see you all next.

A