VMware Velero Community Meetings, 10 Nov 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Velero Community Meeting/Open Discussion - Nov 10, 2020

Description

Nov 10, 2020

Status Updates
carlisia
My birthday is tomorrow, yeay!
I’ll. be on GoTime live today at 12PM PT: https://changelog.com/live/26m
On community support
Tons of PRs to review
Working on a Velero presentation
Storj/Tardigrade interview about Velero: https://www.youtube.com/watch?v=ldWF7cmNEFc
ashish
Bug fix PRs:
🐛 Use namespace and name to match PVB to Pod restore
🐛 Do not run ItemAction plugins for unresolvable types for all types
Pick up on the e2e tests. Release item for v1.6
PR reviews:
Design PRs
Sync restore of additional items from restoreItemAction plugins.
Proposal looks good overall.
Will take another pass before signing off.
Appreciate your patience.
Also working on the presentation with Carlisia.
nrb
Reviewed proposal with Rafael’s team on the API migration work
DockerHub is limiting pulls from free accounts. This may limit your ability to pull Velero images, especially from CI if it doesn’t cache. We’ll likely be moving Velero into VMware’s newly created public repo at some point in the future to remove the restrictions.
KubeCon NA is next week! Bridget and I will be doing a Velero Q&A at 4pm ET, Wednesday November 18th.
Zoom link
bridget
Mostly catching up on things after PTO
Reviewing design docs
Would like to get involved with multiple credentials work.
dsmithuchida
First pass at e2e tests - https://github.com/vmware-tanzu/velero/pull/3060
Released Distributed Data Generator (Kibishii) test tool - https://github.com/vmware-tanzu-labs/distributed-data-generator

Discussion Topics
Rafael @brito-rafa and Frankie @codegold - Updates to Choosing API Group Version to Restore Design Doc: PR #3050
Review priority cases (with examples with an academic API Group): https://github.com/brito-rafa/k8s-webhooks/tree/master/examples-for-projectvelero
FYI: Changing priority system to match that of K8s more closely
Phuong Hoang: Timeout for BackupItemPlugin design https://github.com/vmware-tanzu/velero/pull/3018 Adding timeout to ExecuteRequest may cause incompatibility with prevous version of plugins.
Action: Phuong to add a meeting for this
Scott/Nolan Addtional items wait design doc
https://github.com/vmware-tanzu/velero/issues/1970 is also related; we cannot currently restore a namespace by itself, or create RestoreItemActions for a namespace object.
Pranav Restore progress status design doc
Handling multiple credentials/secrets

Community Shoutouts
@invidian: Fix various typos found by codespell
@a-mccarthy: Add custom 404 page to website
Preview of the 404 page
@mynktl: fixing ‘velero.io/change-pvc-node-selector’ plugin to fetch configmap using plugin name

A

Hello, everyone and uh welcome to this week's episode of the valero community meeting today is november 10th 2020 and per usual we're gonna start up with some status updates uh and then we're gonna dive into discussion topics.

A

First off we have carlesia.

B

I'm getting older, you all that's happening.

A

Happy birthday in advance.

B

You and I'm going to be on goal time today as a guest. I will be talking about distributed systems with kelsey, hightower and there's another guest. I forgot his name. He works for page of duty. I think um I am on community support this week. I there are tons of vrs I need to get through and I'm also working on a vlr presentation. That's it for me.

A

um What is the valero presentation for.

B

It is to it's actually for uh internal vmware group of people who that that do um support and for valero, and that they just need to be very sharp about how filler works and the architecture. So we have a series of presentations for them.

A

Sounds good, uh maybe something that we can share later on as well.

B

A

A

All right, uh can you throw in a link to the go time uh live event there too,.

B

Yeah I'll do that now.

A

You cut out there. Jonas, oh sorry,.

C

um Yeah still, no sound is that just me. I.

B

I could hear him yeah so can I.

A

Okay, no no is having audio issues all right. um Any questions, comments for kalisia.

A

All right next up, ashish.

B

Oh sorry, I have one more thing. I did a recording with um storage and I just found out today the the link to it. I'll drop the link to that too. It was a recording talking about valero, mostly and stuff awesome.

A

Awesome yeah: please do.

D

A

D

So for me, um I had a couple of uh I found a couple of bugs in our code base, for which I have prs open. The first one was on this one. Thank you bridget for the investigation, the when we were restoring. uh We were just using the pod names to find their respective volume backups to do rustic restores.

D

uh So what would happen is if you have a part by the same name, into different name spaces uh while restoring one part, it would go, try to look for the pvb of the other part and say, and obviously it couldn't be present in the rustic repo, and it would say it would fill the restore saying part volume snapshot or the snapshot id not found, or something like that. So I have a pr up to fix that yeah and three zero.

D

Five one is that pr uh and the other one is when we were doing restore item actions. Excuse me when we were doing restore item actions. uh We would the resolution of like types that are unknown to the api server. So when whether restores it would go query the uh it would use the discovery to find out. All the types that the api server knows and if there's a type that api doesn't aps server, doesn't know about, it would apply that restore action to all resource types, and that would definitely fail.

D

So this was also reported by rafael and his team, so this pr should fix that. This fix also applies to the delete item actions.

D

We had a pr earlier that addressed a similar problem for the backup item actions only this pr that I have put up kind of unifies, both those fixes. So at least for now everything should be fixed.

E

Thank you so much ashish yep, it's gonna really make our life easier.

D

Thanks uh and other than that, I'm gonna start picking up on the e3 tests, which is like a release item for us for one six, um I going to the pr outstanding pr reviews that we have. I took a look at the uh I think, scott. I saw him on the call his pr about uh synchronously restoring additional items before trying to restore the actual items.

D

uh The proposal overall looks good. I had a couple of comments and thank you for addressing those, so it looks overall, it looks good overall, I will take another pass and sign off and yeah. Thank you for being patient with us um and yep any deaf other other maintainers and other members of the community. Please do take a look at it and provide any feedback that you may have other than that. I am also working with alecia on that presentation that she mentioned.

F

D

A

Awesome any uh questions, comments for sheesh.

A

All right, moving on to nolan.

C

Yeah so um last week I reviewed the proposal on uh that. Rafael's team is working on the api migration work, so we went over some of the the hitches that they've run into including um the item action, plug-in resolution that ashish mentioned um and some other things with um writing different different plugins, um something I wanted to highlight this happened last week, um but a lot of other stuff was happening too.

C

um If you weren't aware docker hub has put started, putting more rate limits on free accounts um and there's more rate limits on authorized accounts too there's rate limits on anonymous polls and on free accounts, and things like that. uh I just want to make sure people are aware. This might limit your ability to pull valero images, we're not the only ones affected, but um this is especially important.

C

If your ci doesn't cache, um we will probably be moving valero into vmware's newly created uh public repo vmware's already started moving some of their projects into their repo, but that's not immediately on our plate. I just wanted to give people a heads up that all of that changed.

C

I know openshift folks have been talking about this, so yeah, anything that relies on docker hub this. This is affecting that. um So just a heads up, that's not necessarily anything we have. We don't have direct control over um also wanted to say um cubecon. North america is next week. um Somehow it came up really fast. um I dropped in a time zone, converter link. uh The session is at 4 pm eastern time, u.s eastern time next wednesday november 18th.

C

I've also provided a zoom link to. I believe it is a webinar, I'm not exactly sure how it's how it's going to be formatted, but uh yeah we're going to have a q a with bridget, and I I will also be part of the.

D

Q, a okay cool.

A

uh Yeah, so for um for kubecon here uh so for valero and a bunch of other open source projects that we have uh or are involved in. We are going to have office hours like this, so um I will link to the the main page where we have all of those listed as well I'll get that so I can show that during the discussion topics for docker hub here, I know some of you might have seen. uh Docker has released some guidelines if you are an open source project and you want to remove.

G

A

Strict limits there are ways to do that. Not all um we we've looked at some of the uh the the guidelines and the rules for that, and we we don't agree with everything. That's in there. So we'll we'll see what happens but, as nolan said, we might be looking at moving over to another repo as well.

D

And if you are somebody who is impacted by this, please do reach out. We will try to see if there is anything we can do to help you out.

A

C

And just in transparency, um this is something like vmware was affected by too, like we had to set up proxy cache as well. So this is something that will affect everybody that has anything any any dependent layers that are on docker hub, any anybody that has anything touching docker hub, you're gonna be hit by so but yes, as as she says, you're impacted reach out to us.

C

I can't guarantee, we will fix it for you, but we can see if we can help you look into some workarounds there's stuff like pull through caches, and things like that that you might be able to implement for your ci system.

A

Any other questions comments for nolan.

D

All right, if this becomes a bigger problem than it is then we might want. We might also look into publishing a tarball of the image that folks may download and push it into their container registries.

F

Have we investigated using quay.

C

um We have not, um and just in transparency it's more likely that we'll use a the vmware repo than it is that we'll use querky or I'm not sure how you pronounce it. I thought it was key, but um yeah. I think it's far more likely that we will be using the vmware hosted repo if we switch and again in transparency. That was something our our team was pushing for internally to have exist for a year now. So.

D

Yep that gives us more more control and not have such issues come up in the future uh yeah long term. We want to move to a vmware, uh hosted uh repo, but short term. We are happy to help you with mitigations.

A

Any other questions comments.

A

Then bridgette.

H

Hi everyone- um I don't have a huge mic to report, because um I've been on pto for a while, um but since coming back, I've been mostly catching up on things that have been happening and also taking a look at some of the design docs um that we have, which are pending review. um I've also taken a look at the the restore item, action change that scott has proposed. um I looked at that before, but I've come back to it following pto and I'm happy with it.

H

So I'm probably going to take another passover this afternoon and hopefully get that um be able to to sign off on that and give it an approval. um I've also been looking at the proposal for multiple credentials um so give so that is a 1.6 um release item as well, so I was just going to put it out there that I would like to try and pick up that, if not pick it up but get involved in some way.

H

So um if there are no objections to that, that's probably what I'm going to try and focus on next.

C

Please do um awesome, I know carlesio was was doing some work on that. So, if you could sync up with her, uh the proposal is in a, I think, a kind of bare bones state from early earlier this year, but yeah. If you could sync up with her and and brainstorm some ideas, that'd be awesome.

H

Excellent yeah that.

B

H

B

I've been glad to take a dr uh passenger passenger seat on that, and you know guy guides you catch you up and maybe, if necessary, split the work, but that's indeed and I'd be glad to take some of the cli work for 1.6.

B

Okay, great. Thank you.

H

Sure, that's it for me.

A

All right any uh questions, comments for calycia.

B

A

I mean bridgette, sorry thank you.

I

It's it's the dress shirt. It removes 10 iq points.

A

Yeah yeah, I'm sorry all right. uh Moving moving on to dave.

I

Who's not wearing a dress shirt.

I

um Let's see so. I've been working on the engine test, and so we have a basic one and the pull request. That's noted there. This will install the distributed data generator, which we'll get to in a moment. uh Have it generate some data back it up. Remove the name. Space restore the name. Space then verify the data. So you know that's that's pretty much a basic end to end. I'm currently running it against aws we've released the distributed data generator against the vmware tanzu labs.

I

So this is a little open source project we've got, and this is basically a synthetic test.

I

We'll also be testing with some real applications, but kind of the point of having a synthetic test is that we know exactly what it does and it should be doing things like you know, reporting why things are messed up rather than simply crashing.

I

So, if anybody's interested in working on that there's a lot of areas in the data generator where we can add things um right now it it basically creates um a bunch of files. It will run on as many nodes as you need to and it coordinates through an etcd database.

I

It would be interesting to do some raw block work there and also interesting to start looking at actually verifying the kubernetes resources. So what we're doing right now is we verify the data? That's in the persistent volumes we make sure it's all there and that it's correct and I think uh it would be nice to actually verify that you know what we restored as far as kubernetes resources matches what we backed up or we want to restore. So we have to filter out annotations and things like that. So lots of work to do there.

I

Yeah, so that's that's pretty much it for me. We can work through things if we like, or you know any questions.

A

Awesome any yeah any questions, comments for dave.

E

What is kibishi's stance.

I

It's uh japanese for strict.

E

Oh no, it's right there. Okay,.

I

Yeah, I got tired of using greek words and since I have at least a smattering of japanese, I decided that nobody uses japanese one.

C

And this is this is based on the test, tooling, that you used for the the vmware plugin correct.

I

Yes, so it's cleaned up a little bit we're going to continue cleaning it up. I got the yaml files working better and did some stuff with customize there to have different clouds, because currently it pretty much works against vsphere, and so now it's running in aws. Well I mean it was always. It was always worked, was mainly the kubernetes setup.

C

And does I know um you karlesia, and she should talk to some folks internally on the tkg testing. Does this all run against the tkg testing? Now.

I

C

I

I think our next steps here um we've.

G

I

The um we've got the jenkins job that will, for example, step into cluster and then kick something off, and so we can start integrating with that. um I think once we're agreed on you know what we're doing here. We can start kicking that into the um it's about integrating that with tkg ci pipeline.

I

Cool um also um so I I've been working on setting up aws vsphere. uh We want to get azure working and I think we also want to get kind working, so people can easily run this on their desktop. So that's something else for you know if anybody knows who's worked with that a lot. That's an area to jump in on.

D

These tests that we have they will run against any kubernetes cluster. Won't they.

I

They will but there's things like, for example, storage classes that are uh different by each uh my cloud provider. So if you take a look, um go back to the distributed data generator, would you there joe um take a look, uh go up and uh take a look in the kubernetes directory and there's a bunch of old junk in here, but go ahead, go into yaml, and so I've set this up at the moment as kind of a customized thing so like in uh you can take a look in the base directory.

I

So this is basic thing. So there's like three different pieces that go in there's uh kabishi itself, there's the fcd and then there's a jump pad uh container that you use to log in and execute tests from. You can talk directly to egcd if you like, but this is workable and uh go ahead open up like the kibishi.yaml, it's pretty basic, but it uses.

I

For example, I just used like kibishi storage class here to pick the discs and now, if we go back and take a look, you go back up uh genus to yaml and then go into the aws directory.

I

So in here I've defined the aws storage class here, which is pretty basic, but it's different between aws and vsphere, and if you take a look at the customization dot, yaml please go yeah.

I

All it does is include the base stuff, and then it includes our storage class here and we can add other customizations for aws in here to override we'll kind of work our way through it as we go and the same same is true for vsphere. So this one is the csi driver, uh storage class.

I

That's what I've got so far, we'll see what else we need to do. That's cloud specific and then I want to do some stuff where we uh can easily change, for example, like the number of worker nodes. So right now it runs uh n containers, there's a stateful set in the um in the base. uh uh Yeah. If you go on base kaviche.yaml, it's a stateful set, so it's currently set replicas too, but we should be able to set it to whatever number we want and uh we can also change the amount of storage.

I

That's allocated so right now it's a one gig disk, which you know it's pretty small. We've done larger things during testing. So what I want to make is the the customization, the customized stuff. I think we can easily change the amount of storage allocated. For example, we can try different levels of testing so like one gig is good for a quick test. um There's scale testing where we might want to have a terabyte even and just see how long it takes. But you don't want to do that. Every time.

C

And one more question: I'm uh I'm just noticing the time I've got and I kind of loaded the discussion topics with all the design docs. um This doesn't does this do anything with rustic.

I

Yes, uh so the the current backup is using rustic to backup and restore and we'll need to modify it, so it uses the appropriate volume snapshots. But right now it's resting.

A

Cool. Thank you.

A

All right, any other questions, comments for dave.

A

Okay, diving into the discussion topics: um nolan. Is there anything you want to bring up first, or should we go from top to bottom here.

C

um I would say: go top to bottom, just out of respect for rafael and frankie for putting that in there first sounds good.

E

Thank you, um yeah. Just a couple updates on that api group version restore frankie, put the design dock thanks, everyone for the contribution we as part of the design dock.

E

Can you click on that link about the examples we work about four use cases, and for such I created this academic crd called rock band with multiple versions, multiple fields and schemas, and I coded um a controller with app web hook to convert back and forth, and if you can go down a little bit join us, but uh they saw all the cases we want to cover on this migration. I mean I'm on this conversion or back copper, restore. We.

E

I keep saying migration, because the whole point of this is restoring a backup on a different cluster that you took the backup. That's why I keep saying migration, so we have all these cases right where we have different preferred version and supported versions between the source and target cluster.

E

Please take a look on those cases. We put the rationale: how we're going to make decisions to do the restore certain cases already happening, because um such a case a right when we have the preferred version of the target and the source they're the same, that's already happening.

E

um There is other cases that are very unique case b is one of them um in case these. When there's no um the the preferred version, they do not match, but you have an intersection of the versions uh that are supported, which brings another update. There is indeed kubernetes has code for priority versions, so we're gonna. We added this on the design dock. We're gonna, follow the priority versions on case d.

E

um Another thing is with the those cases now we are in the position to start uh coding. Even if we decide to change the design, but the cases they do not change, they are what they are right. Those are situations that we're gonna come across, uh so we're gonna start to do some coding, and then we can always tweak now. Another update, based on the feedback from the last meeting, an option for the user, define what is the version to be restored so part of design doc. We we're gonna.

E

We are proposing a config map where the user can create arbitrarily versions and then it's going to bypass any priority. I was going to look at the config map. First.

E

So, there's a lot of work in flux. We welcome everyone to look at the design dock, but again the cases are the cases uh we're gonna start coding. um We can just change the sequence the priority, but um it's pretty much what we we have planned on the following weeks: the real code.

E

uh There is another contributor genting, um it's too late for him, but we're gonna work with him with his code. In our code we have already a prototype code. Frank is working on that, so it can come from a common ground.

E

Any questions about that any questions about my awesome, crd called rock band.

E

I I came up with the beatles, so if you create a crd where the with the first names it gonna convert for the phone names, that's my uh web hook, logic to make sure I got it right if you have a preferred rock band, um I'm gonna have metal point myself, but I decided to do beetles because it's common ground for many people right fab, four um anyways just being me uh and then, by the way, once we code that we can use that crd for unit testing, if you feel uh fit for our testing and I can change the syria now, I know cid back and forth, actually dreaming about them, so I can whatever you guys need want to do.

E

I can can help out. That's all thanks.

A

Awesome. Thank you. Oh any questions or comments here for rfl.

A

All right, thank you, rafael and frankie.

A

Next up we have fawn.

J

Hello, um so I think about few weeks ago I proposed the design for timeout for the backup item that we need and what we find out is that um so one of the feedback I received when I bring the desire to this community, I think I remember is maybe from from this that, instead of just passing out just control the timeout from the function call, we should also pass that timeout to the plug-in itself, so that the plug-in will have a control of when to terminate it uh by itself.

J

First, uh so I took a look at that uh idea to see how we can executing that idea, and I found out that if I need to pass that the the password timeout all the way to the server side where the plugin code is being executed, then we have to change the execute, execute message: the execute message being passed between the gpc, client, science and gpc server side.

J

If we do that, that means that I think it might cause incompatibility between the um the client side and the server side of the plugin if the plugin was compiled. With the previous version of the execute message, I I that's why I I don't have experience on that area. So I wonder if um I bring it here so that you you can see um what is the potential problem with that?

J

Adding the one field into the um the execute message.

C

So adding that field, as as I understand, grpc that's going to um in in val, so basically every plug-in of that type- would need to be recompiled and updated for the next version of valero. So it would.

J

Actually, work is actually worse than that. It's not only for that tie, but it's also uh the backup item. That's what.

C

J

C

Yeah yeah yeah that plug-in type, so anybody who's made a backup item action uh and wants it to be compatible with whatever version of valero that change would go in would need to be updating it yeah.

J

That is the problem. They have to be recompiled.

I

So without changing the api, how are we going to stop a running snapshot or what have you.

J

B

J

Original design was only controlled on the client side of the plugin. That means that when I send a command out from the client side, I just time out that one within a certain timeout that is not done, then I would cancel that and I move on so from there. It's probably gonna cancel.

K

It, though, right because it's going to ignore it.

J

Yeah, basically, the the the I I do as using go um go routine with contacts, and I just cancel that contacts that mean it's just terminate that um that client, you know no routine, that communicate with the server side. uh That's what I meant when I say, cancel. If it doesn't send out. You know the cancel command to the server side. If that.

J

Is and yes, I I think that uh I think this uh I think last few weeks ago you bring down that point. I think it's valid for that. If we don't pass the timeout to the server side where the truck is supposed to be executed, I mean it's kind of a kind of a missing piece in the chain.

C

So I have a question: you you in the design dock and on slack you mentioned passing in the timeouts uh yeah down here at the bottom. You pass in the timeouts at the backup command.

C

Is there I I'm not fully understanding why this is attached to a specific backup operation like is? Is there a reason.

J

C

Being a global setting.

J

Yeah, I cannot explain. I can explain that so the idea was that in the in our application, our backup solution, we allow a user to choose either class, consistent backup or app consistent backup right. So if it is a cross-consistent backup, we don't create a part, we don't. We don't hold our application in a lock.

J

Because of that you know the timeout can be, you know longer much longer, but if, in the app consistent backup the customer one you know do an app consistent backup, then we have to create the card, and that was a timeout needed right. If, if we close the party for a long time, the application will fail. So we want only the timeout when the customer used the app consistent backup for the custom user, backup we don't care about it.

J

So that is the context of why we want to specify the timeout only for when we need.

K

J

For the same name, space actually important for the same name say the customer can either choose between doing normal, backup and and and socket system. That sort of.

D

Thing somehow, it seems like you're trying to use backup item action plugins for this backup hooks. uh I I'm a full disclosure. I was away for a couple of minutes in when you were explaining, but it sounded to me that uh you it is possible to accomplish what you were trying to do using backup hooks instead of backup plugins.

D

Is that something that you've tried, because with backup hooks, we have timeouts and unquestion and qs operations, and that's what exactly? For what backup hooks are uh built for?.

J

um I think that is here, here's the sequence of action right. So if I, if I do enough conditions on the part first, I run the hook right now. The hook itself have the time out. I understand that then after the hook was executed, we created the part. Now I run the plug-in to backup this is the valero.

J

uh This is the vmware vsphere bus plugin. It.

L

Will be executed to.

J

Create a snapshot about pvc being used by the and.

G

That is the where the problem.

J

Is and then after the vcs plug-in was executed to create snapshots, then the uncreates hopes being executed. So the hope that you mentioned is only for the creation nps function. It do not apply for the vsphere plug-in, where we executing the snapshot, creating the snapshot for the pvc.

I

Trying to solve here is that we've got, for example, an app consistent backup taking place and we've quiet the application. So it's not taking any. um It's not servicing any requests, but then the snapshots are taking a really long time and that's holding the application uh quiest longer than we want. So exactly.

J

I

Yeah, so the goal here is to be able to bound how long the plugins take on doing their snapshots so that we can, for example, fail the backup even and let the application continue on.

I

um I think we really really need to get the cancellation into the plug-ins uh because, like one of the complaints has been like the vsphere plug-in is, is taking too long to timeout, and uh this is happening like, for example, we'll have problems talking to vcenter and the plug-in is retrying. We can reduce the number the amount of time that we tries.

I

I think it's way too high right now, but be then, as it may, there's always going to be some difference of opinion of how long it should be, but with this scheme right now, what would happen is we'd cancel on the on the valero side, we'd, say: okay, we're not waiting for that snapchat aim where we're going to move on, but the snapshot is going to wind up being taken and then on vsphere. If you have a snapshot on a vmdk, I guess hopefully we would.

I

We would move that over and uh delete it afterwards, but uh no actually in the powerprotect case we won't we'll just create the snapshot and return, and so, if you don't get the snapshot id, you won't be able to clean it up and if the snapshot is on the virtual disk, you can't delete it so we'll wind up with bugs. You know, like months later, when someone's trying to clean up something. It's like. Oh there's a snapshot hanging around that I can't delete and therefore I can't tear down my application.

I

So I think we need to be careful about clean-up.

J

On I actually understand the the seriousness of having you know, the the.

A

um The server side, at least.

J

Here server side to receive the timeout and receive some cancel, how we can do that because the the way we doing right now we can be executing the grpc client sizes, saying you know, execute, and and that's it that's the only thing that we have right now. How can we send a console.

G

For that, via pc.

J

Should we should we send a cancel, or should we send the timeout over? What do you guys think I'm asking you are premium here that are not familiar with this grpc myself? Well,.

C

My question is: are we solving only for vsphere like is this? Is this a is this something we're only solving for one implementation, um or is this applicable to more than one case.

I

um It depends on which, so the the cancellation, I think, is going to be useful everywhere.

F

I

um I think the cleanup depends on you know on your system right on ebs. You have a snapshot, that's lying around. It costs you money, but it doesn't block operation, I'm not sure about. You, know. Other systems.

C

I just want to be sure we're not we're not putting we're not making a whole system level change for one thing.

I

I mean we're going to do timeouts. I think we should do timeouts right. I think the plugin should be aware that it needs to cancel something and uh it should.

C

I

Opportunity to clean things up.

C

Right because I don't want the, I don't want it to cancel and have things dangling and not actually get like it. It needs to cancel all the way through down to storage system. um Otherwise, it's not correct um right, because then we don't have a full accounting.

J

It sounds like we cannot. We cannot just win now the the design in this meeting. Perhaps we should create another meeting and and then discuss about how we can properly cancel a plugin execution right.

A

J

Okay, I I would- I would create uh uh that I already talked about on the on slack. Maybe I would continue on that slack and pull in a few people for a meeting, because it does show to me up to this point that it's become bigger than you know. It's just a time out like it's it's more than that, so I left, let's create a new um meeting to talk about that. I don't want to take the community time here.

J

A

Okay, thank you so much.

C

Yeah my only request there is send the send the meeting to the project valero google group, um so we can invite the community to it um and uh yeah thanks for thanks for the discussion.

A

All right, uh we got three more discussion topics, so we're gonna time box. These um um the first one is scott and nolan.

C

So I wanted to bring this one up really quickly, because there was um some discussion um about uh doing a restore item, action with a namespace and if you go back to the agenda doc, um I have this other issue 1970 right now we have this problem with how valero restores namespaces uh namespaces are restored kind of just in time when the first object is inserted into a namespace and namespaces are created directly. uh They are not. They don't go through the full valero restore flow. um So I made a comment down at the bottom.

C

I should actually link these up via github, but basically, what I'm wondering is because um namespaces are like a global example of a dependency um right. You get the you get. The first item that gets inserted into the namespace will have a dependency on the namespace.

C

I'm wondering if this this additional items, design dock might be a way for us to revisit the design for namespaces making sure they're created. We may have to get creative in how we figure out, like we might have to dynamically insert an additional item to say: hey the namespace. Isn't there uh it's inserted as an additional item for the first item we get created so.

L

I was just gonna say what one way you might be able to do that. I'm not sure that relates to this necessarily, um but you know it perhaps it does, is that you could create another restore item, action plugin that matches everything which exactly.

A

L

Its only action inserts the namespace as an additional item um where.

G

L

Fit into this is, if there's a I'm this make, this may be one of those dependencies where if the namespace doesn't exist, yet the item won't create properly, and so you would. You might use this to have that weight, saying if the namespace.

G

L

Exist where it's just an existence check, um you know, so I I guess the point is it relates to this because you could independently this once this is implemented, create a new plugin that applies to every namespace. I'm sorry that every every item- um and if the item is a namespaced resource, find the namespace and include that as an additional item.

C

Yep, exactly that's, that's exactly it and.

L

I guess the other relevance is that if this weight thing doesn't exist, this plugin doesn't help you because it doesn't guarantee it's there. It's ready, but this this waiting allows you to rely on a plug-in and not a one-off. In the code.

C

Yep and right now we have a one-off in the code which the the um the original problem that this this um issue is talking about is valero. Won't rest. If you say valero, I forget the indication they did, but basically they were trying to restore only the namespace object and nothing inside of it. Valero couldn't do that um because name spaces are special cased and basically using this design dock. I think we can remove the special case. Well, we can move.

C

We can put the special case in the plug-in uh is a better, better way to say it, but basically exactly what you said: scott. We can move it into a plug-in that matches everything. uh If it's name-spaced, then we go gank the namespace make it a synchronous call and then continue once it's ready.

C

So that's the only thing I wanted to highlight here um that and and um the other problem with how we have namespaces being restored right now. Is you can't do restore item actions for them um because they're outside of the the whole valero restore loop, so yeah.

L

I think I hit that once trying to rights, we were trying to do something for that and it didn't work. um Yeah yep.

C

Yep, because they're they don't get matched on resources includes excludes or get anywhere in the indication loop. They just get created directly so.

L

I think this so this would this would because this would be treated as a normal thing. This would allow a namespace to get its own restorative action, um which would then get invoked when you made it an additional item. Yep.

C

Basically, I think this would 2867 would, by implementing this design, we could get name spaces into the happy path. They would still be a special case, but because not a whole lot of other things contain resources in kubernetes.

C

I think this might even support- I I don't know, but it might support the nested namespaces that people are doing now, um but I think basically, what I'm trying to get at is. This thing uh went from nice to have to like essential to me um for fixing that that name, space, uh weirdness yeah.

L

um I think that may one other thing they mention here, um this believability, it's the to the um to the pr review comments that were kind of back and forth is that there was a suggestion to instead of because right now, with my frozen, the only way that we wait for an item is to use a plug-in, specific funk pointer that we return that says, use this custom resource and the suggestion was, you know: why do we need a custom way of doing it?

L

Let's just check for existence using a normal get, um and I think my response to that was. I think we probably need both. I think that generic get is probably the maybe we wanted as a default, but we would. We still need that override in case there's a position in readiness, and I think what that means is. I need to right now. I have a single return value.

L

This is a function pointer that kind of serves as a boolean and the implementation override, and I think if we separate that out into a separate a boolean that says, do we bother to wait from this plugin or not, and then the optional function pointer. That says, if we are going to wait, if this is here use this. If this is not here, use a generic, you know get on the resource. um I think that solves both use cases.

L

It's still they're still up in the air, because this notion of timeout, you know what we want to use for that. The the placeholder that I have for termination is not appropriate. I understand that. Do we want to provide a default, do we want to make each plugin define it? Do we want to allow an override but provide a default? You know, what's the default, gonna be that's the kind of thing we need to finish, hashing out in the discussion uh on the pr, probably.

C

A

Do think yeah yeah just quickly. um I want to make sure that we cover the the last two discussion topics today as well. So uh if you want to continue the the conversation over on slack or or github, I think that'd be great.

D

Yeah, a quick closing comment on that uh scott. I agree with you. uh That's uh yeah, let's, let's take it.

D

L

Wait for your comments and then, if you want to comment again I'll update their design and to reflect the discussion.

D

A

Thank you. um Next up, we have pranav with the restore progress.

G

Yeah, I'm ready hello, everyone. My name is pranav uh just to give you a quick background for for the folks that don't know what this proposal is about uh here in this design doc, I'm proposing uh that we add progress, reporting for valero restore objects, just like uh just like what we have currently for for the backup objects, the backups here um high level.

G

uh This proposal, like this, proposes to break down the restore process in two steps, in that the first step just collects the items that are to be restored and saves them in in memory, and the second step is where actually the this. The call to the restore item uh is done.

G

uh So that's that's the high level design, and I have uh consider two approaches to do that right now. The restore process works by uh looping through the ordered list of resources and then the name spaces, and then we come come down to the items.

G

We can keep that as is, and instead of restoring the item, we just add it to the data structure and in the second loop we actually restore the item, so we call the same process. uh We actually look through the resources. Two times uh in, if you, if you can go down below yeah, that's it right, so we in in the first in the get order, resource collection function.

G

We are just looping through uh all the resources, then the items by namespaces and we are like storing the resources uh in a data structure and then we are using that to restore it. That's one approach. The second approach I've considered is that uh just like we are doing it for uh backup.

G

uh We can have a flat list of items which are again in order uh of the priorities uh that I I didn't document that but yeah. That was another alternative and we had also talked about having.

G

So, instead of doing this two-step process, just do one pass, restore the item and have something like the number of skipped items and uh the number of items restored so far, and in that that was another alternative that I considered yeah. That's all. I have.

C

Awesome thank you for for including the code changes. I appreciate that.

A

Yeah, this is great uh any questions. Comments for pronoun.

A

All right, um let's move on to the last one, which is handling multiple credentials and secrets.

A

This is kalisia or someone else.

C

um Since I added it without telling carlesi I can take it, I I just um want to say that this is so. I think I think the main thrust of this is going to be that we want to make sure that in a single cloud, provider or plugin can support multiple credentials and secrets. So right now, our current our current setup is not great and allows only for a one-to-one mapping.

C

So if you're on aws, for example, you can only use one well, aws is not a good example, because they allow multiple credentials in their file. Okay, so azure. If you're on azure, you can only use one file for accessing azure resources. You can't use multiple.

C

This proposal is basically going to say, let us say, um use multiple azure accounts for different for for the same plugin. um That's the overall idea, um I think, there's some some trickiness in filtering it down to rustic, um but the the overall idea is making sure we're not locked into a one-to-one relationship, so one plug-in can have n number of credentials.

C

That's the goal.

C

um I had some comments on here about doing short term credentials and rotation and stuff. Like that- and I think that is a different uh design dock- I think that is out of scope for this one.

B

Yeah rotation, uh it's it's a complex issue, and doesn't I mean if we were to tackle this now we should be tackled separately, but we're not going to tackle it now. So we can ignore that. Definitely.

C

And there's also support for things like um cube to im and uh I forget what the gcp and azure ones are called, but there is support for, like node-based authentication that you can do if you need stuff like that so um yeah. I I think this. This is largely for fixing the problem where we had we're. Finally, getting out of the one-to-one lock-in.

F

Are you going to make it so that um backups are tied to a specific.

F

uh Set of credentials then, or is it going to be when she set up the credentials, then any valero backup can use those credentials.

C

I don't remember the details of the design, but I think the main thrust of it is that credentials will be tied to a storage location, whether they're volume snapshot or a backup storage location.

C

So once a backup goes into that location, it'll have to use those same credentials that are tied to the location. If I'm remembering correctly.

A

All right we've got one minute left um nolan. You want to dive into the shout outs.

C

Yeah we can. We can address the question on the dock, um so the first one. uh Actually, uh the first one um was from nvidian. um He uh I'm I'm sorry, I'm not going to get that name right. uh My two videos matthias um ran code, spell against uh the valero core um and found a lot of typos and fixed them, and also volunteered to add a github action to check that. So thank you very much for that. um Abby updated our site to have a custom 404 with.

C

Links out to the various um main resources. um Thank you very much for that abby and then finally, um my ankh, uh I think, patel.

C

If I remember yeah, updated some code uh with the change pvc node selector plugin. um There was a mismatch between the plugin name and the label key uh on the config map, so that is now fixed to reflect our conventions.

A

All right awesome. That concludes our meeting today. Thank you. Everyone for joining, see you on slack and github and see you all next week have a good one.

L