VMware Velero Community Meetings, 20 Oct 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Velero Community Meeting / Open Discussion - Oct 20, 2020

Description

Oct 20, 2020

Status Updates
carlisia
Community support status:
done: triaged all recent (couple weeks) issues
wip: triaging older issues and starting on the other repos
PR reviews
tackle my broken test for https://github.com/vmware-tanzu/velero/pull/3004 (DownloadRequest migration to kubebuilder)
nrb
Fixed https://github.com/vmware-tanzu/velero/issues/2570, plan to get v1.5.2 & v1.4.3 out this week.
Planning our support-a-thon to reduce the number of open support issues this week, Wednesday to Friday
v1.6 checkup!
bridget
Made some changes to #2993 based on comments
Discovered a problem with approach in newest changes so need to revisit that approach and modify tests
Helped Ashish with internal release processes last week
Reviewing design docs
dsmithuchida
PTO last week
Working on e2e tests

Discussion Topics
carlisia
Should we add this to our list of supported plugins: https://github.com/Lirt/velero-plugin-swift/
How do we handle requests for a new plugin: https://github.com/vmware-tanzu/velero/issues/2988
Add them to https://github.com/vmware-tanzu/velero/tree/main/site/content/plugins/list
Also https://github.com/vmware-tanzu/velero/blob/main/site/content/docs/main/supported-providers.md
We are hiring a senior Product Manager to lead Velero: https://twitter.com/carlisia/status/1318304806784573440 (spread the word please!)
Jonas got a promotion, congratulations!! Well deserved title of Head of Community Management
nrb
What do folks think about moving from ZenHub to GitHub Projects?
Example: https://github.com/orgs/projectcontour/projects/2
Advantages:
Integrated (doesn’t need a 3rd party site/plugin)
Can create multiple project boards for different things
Disadvantages:
GitHub issues aren’t automatically added to the board (maybe GitHub Actions exist for this?)
https://github.com/marketplace/actions/github-project-automation
No concept of epics
One time switch cost
v1.6 priorities
Implementing e2e testing
Continue migrating to kubebuilder/controller-runtime
Enable the use of custom credential providers
Upgrade to restic v0.10.0
Phuong
Backup in parallel https://github.com/vmware-tanzu/velero/issues/2888: maybe just backing up pods in parallel
Enhance Backup with plugin timeout (https://github.com/vmware-tanzu/velero/issues/2922): pull request submitted https://github.com/vmware-tanzu/velero/pull/3018
dymurray
Follow up from previous meeting, restore progress design proposal: https://github.com/vmware-tanzu/velero/pull/3016

Community Shoutouts
@betta1: Fix BSL controller to avoid invoking init() on all BSLs regardless of…
@gliptak: Bump Go to 1.15
Arianit Uka for reporting https://github.com/vmware-tanzu/velero/issues/2570
Pranav Gaikwad for https://github.com/vmware-tanzu/velero/pull/3016

A

There we go uh hello, everyone and uh welcome to the valero community meeting and open discussion, and today is october 20th and per usual we're going to start out with some status updates before we dive into the discussion topics. First up we have khaleesi.

A

You're on mute.

B

Hello, I'm on community support this week, and yesterday I was trying triaging all like I was reading issues from the valero repo.

B

I was able to get through all of the this week's and like a cup, the ones from like leftover from a couple couple like up to a couple weeks back, and there are some that are older than that, just like maybe 10. I want to get through them today and triage the issues on the other repos.

B

I'm hoping that when we get our meeting we're going to talk about our like our extension of our support process, we're trying to make it better. So when we get to the meeting to do the planning for the supported time that this is what we're going to talk about, uh there was there'll be nothing to beat.

B

Basically, so we can just hop on dividing up the investigation for everything and there are, there are a bunch of pr's, I'm gonna get to review those, and then I have to go back and tackle my broken tests and that's it for me.

A

All right awesome, anything you need uh help with this week.

B

A

All right, uh thank you, felicia, uh any uh sorry, any questions comments for kalecia before we move uh move ahead.

A

C

Nolan hey so last week um there was a um an important issue with how valero brings persistent volumes and pvcs back into a cluster uh when it restores this would be new brand new clusters or, if you deleted a namespace and and pvs um in an existing cluster. So the fix is in main um need to get version, 152 and 143 out the door this week. So uh I may be working on that this afternoon at least getting the release branches updated.

C

um That will 152 will include the fix that I think anthony's on the call.

C

uh Last week we were, we were discussing some issues with um the the backup storage location, uh controller instantiating, the the plugins. A lot 152 will include that fix.

C

uh One four doesn't have the backup storage location controller, so 143 won't get it so um next, uh as carlesia mentioned, we're having uh what we're calling a supportathon um to expand on that, because I don't know that we've talked about this in a community meeting, but I am guessing a lot of folks have noticed our support backlog in terms of new issues that um may not have gotten a response yet has kind of grown across the core repo plug-in repos and helm.

C

So our goal here is to take some time this week, wednesday, through friday and uh categorize. These issues um sit down with a bunch of them and investigate them, or at least schedule them for investigation and hopefully start with a clean slate so that we have. uh We can go in with a refreshed support process and be more on top of these, and I have another topic to relate to this in the discussion topics.

C

um And then this is uh a v16 checkup. um So I have, I have a list of our priorities that I don't think was actually brought out to the community, um but our number one priority for the one six release is one six cycle is getting end-to-end testing up and running, um so uh dave is working on that she just put out a ping that he is having internet issues um so he's not on the call right now.

C

I've asked you sheesh to help dave with that um and then in terms of features we want to get uh custom. Credential support in and update rustic to v0.10, but mostly v16 is a tech debt release that we're getting and done testing in doing some q builder migration, updating rustic.

C

So that's that's most of what v16 is and in the discussion topics I have a list of what those priorities are.

C

That's for our team, if you, if you want to contribute in the december time time frame, we will do our best to accommodate um getting your feature in because I saw two design docs come in this morning.

C

Excuse me any questions.

A

Hey nolan, have we uh um have you shared the kind of the rundown of the priorities for 1.6 outside of the end-to-end testing.

C

No and that's what I was putting in the discussion topics. Okay, um um I basically took a took bullet points and put them in the discussion topics. So I I took our top four or five um the end down testing stuff that may end up, starting um with some internal stuff that we have for the vsphere plug-in.

C

uh It may not all end up in the open source repo yet, but that's something we're going to try to get on an ongoing basis, um but we're really trying to work on increasing uh the confidence, because valero is a complex piece of software.

A

All right awesome, any questions, comments for nola all right. Next up we have bridget.

D

D

um So during last week I made some changes to the pr that we discussed during last week's meeting, um which is to do with uh blur failing to delete the backup resource if the associated cloud resources are already missing from storage, um I discovered um an issue based on some of the some of the changes that I had made following our discussion comments from last week, um so I need to just revisit that pr, um and hopefully um I might have a chat with you on um on that issue or on that pr about and the approach that I'm taking there.

D

So um if I could get some feedback from you on that, that would be great. um Then uh last week, uh ashish um and I were working on some internal uh release processes that we had to do so. I was mostly shadowing what she was doing to learn about those um and I've also been reviewing some of the design docs.

D

um I've spent quite a bit of time reviewing the the waiting for additional items to be ready, using that as like an opportunity to really delve deep on some of the restore logic within valero, um so the design for that seems good. um My follow up and put a discussion topic just about or just kind of ask the team in general just about um the process for reviewing design docs at what point do you consider them ready?

D

I don't know if we have any um guidelines on the kind of process for reviewing design docs and at what point did they are they considered ready um so yeah? um I don't really have anything else. uh So that's me thanks.

C

As a very quick comment, the design doc process is basically the same as code review. It's giving getting to review two uh approvals and no um comments left.

B

Yeah, I would like to add to that it's uh we don't have like a solid guideline, but it's what nolan said.

B

um It's basically I mean at the least it needs two approvals, but if people have questions as long as we read it and we understand it, doesn't I mean that the idea of the design docs is uh we do have. We do have a guideline for design docs for how to write them. So if you're going to add an introduce a new variable, what what are you going to name that variable? So all that work is done up front.

B

So in the pr we for the actual implementation, we have less things to review because that's always all been figured out upfront, but as long as we okay is there going to be a very new variable. What's it going to be named, are those questions answered and if we understand what the design is doing and the trade-offs- and we don't have any more questions, then that's when we say: okay, it's good for me, so I approve subjective. But it's basically, if it's clear, it's a very, very vague definition, but.

D

No thanks, thanks for the extra information, that's very helpful. Thank.

A

All you any other questions comments for bridgette.

A

And we got dave.

E

Yeah, so um I was on pto last.

A

E

So I didn't get a lot done. I'm looking back to working on the undead tests, I'm trying to wrap my head around ginkgo and um just putting together a real basic test to start out with.

C

C

Why am I fuzzy? um Are the end-to-end tests that you have for the plug-in? I haven't looked at those in depth, but are those like based on python or are they some other some other.

E

um So what the what the rest of the uh tanza team is doing is go based stuff, so I just you know, I'm putting I'm trying to I'm trying to, like you know kind of fumble my way through this, because I don't know that there's actually you know what what the right way to do it is so ginkgo was suggested by the test team. That's what they're doing um I'm trying to wrap my head around the way.

E

Ginkgo figures, things out and works through things, so I'm just putting together a basic test at this point that creates a namespace um we'll create some data into it. Does a backup deletes the namespace, restores it and then verifies the data and then from there you know we'll at least have an idea of what happens: um the existing tests, the the valero plug-in tests, uh those are coded in a combo of java and python, and a lot of the there's a rather large uh framework for both of those that's used internally.

E

That's not open source, so we can probably leverage those in terms of um running our builds through them, but they're um they're not really.

F

E

To run like on aws or any of the other clouds and they're, not in a state where we can open-source them any part of it really.

C

Okay, cool- I was just curious about that um and I know we we were looking at uh ginkgo for as we move controllers into controller runtime. Although then caught wind, that the controller runtime folks are not abandoning geek ginkgo but like moving moving, not using as many features of ginkgo as I guess.

E

Yeah, that's that's the thing I mean you know it's like: what does the test framework really need to do for you and um I'm still trying to figure out all the things I'm trying to figure out exactly what ginkgo really does there's a lot of stuff about how you write your tests? You know in this manner, but it's like yeah, I'm not sure that that actually makes sense, um but you know what I really want is.

E

I want to have a framework that runs everything that tells us what's happening and where we know you know all the tests passed. Here's where the problem cases were, we can drill down into them and look back over things and I'm not. I don't think ginkgo actually does that, but I'm not sure there's anything out anything that actually does so.

E

My goal at this point is simply to write a test that actually does something and then look at it, because otherwise, I'm just an analysis, paralysis. You know trying to figure out well, geez is this. You know. Theoretically, the right thing is yeah.

B

um I I don't want to be that person who contributes to the analysis, paralysis, but I also don't want to not mention this. I don't know how like, if everybody is aware, I'm I'm myself and not I'm not recalling 100 but didn't the cappy people say like andy uh goatstein and say that they are moving away from ginkgo their reasons, and they also offered to show us what they are doing instead.

B

E

Be I looked at some of their stuff they're still doing the ginkgo stuff, the test people are wanting to do ginkgo, so I'm just gonna write. I just wanted to write a test, run it yeah and get my own view on what this thing does, but yeah. That's that's. The next step, I think is to you know, come up with some idea of what of what we're doing. There's a there's, a whole question of what is our end-to-end testing? Really?

E

What are we really trying to do and how much infrastructure do we build to? You know, run tests um like uh you know, I have a a test that I built for the plug-in that does like data generation and verification, but as far as I can tell, um we don't have.

G

E

That, for example, verifies the kubernetes resources at this point, and that's probably something we need to think about is how do we know that we got our kubernetes resources back to the state we wanted them to be in um and yeah so the test framework um I'm trying to figure out how much value ginkgo actually brings as a test framework. I'm not sure it actually brings all that much. You know there's a lot of talk about all the wonderful things it does and then you boil it down.

E

It's like um well, it runs tests in order and it handles assertions so yeah.

C

I think ginkgo in terms of what it does that's unique is its um bdd in go. um It brings uh cucumber like expressions where you can say like describe and it like. It's meant to do like a uh and uh their claim is it does natural language type descriptions of tests, but it's kind of awkward and go. I I find but and the.

E

What I'm trying to figure out is like what am I so like, for example, ginkgo, says, describe, you know an x, um so what am I actually testing? Am I testing a backup restore? Am I testing a particular valero command? Am I testing the whole process? Am I testing data generation? That's what I'm trying to wrap my head around. So I just I'm just poking at it at the moment and writing something.

B

May I make a suggestion, so might be helpful this I know you know this, but I could use a reminder like this. If I, if I were like deep into something like that, might be helpful to remember, we have unit tests, so we are actually testing. You know the internal of everything, so I think uh uh end-to-end testing like that is more like this is an input and output and not it try not to test like the interface.

B

So much like uh you know, because uh so that should be tested with the unit tests that we have.

E

Right and and so like, and so then, when you look at that, so if I say so like in ginkgo, I can say um what does my current test look like. um Let me share my screen. Real quick.

E

um So I'm basically at this point, I'm doing a test for a backup, so I say describe.

B

Back sorry, it's too small. I can't see it.

E

I have to resize my vm window and.

B

C

That that's good.

B

E

So so gingko, you say like describe the thing that you're testing is like we're testing a backup, but we really want to like back up and restore um so right now, I'm just you know, I've got some really stupid stuff. It's like backing up, name space. You know. When the backup successful there should be a backup name, um we should be able to execute the backup um and then you know I'm gonna check. Is it in the backup list?

E

You know we should be able to renew the namespace, restore the namespace and then verify the data, um but that becomes very you know just just a list of steps. Essentially here and I'd like to have something more like hey. You know we generated data, we verified data, so I'm going to poke at this and I'm going to get something running and then try to think some more on.

C

C

Cool thanks for that detail. um I think we can move on for this meeting but appreciate you going into.

C

A

All right, um let's move on to the discussion topics. uh Kalisia you got the first one.

B

All right, jonas, can you please bring up the yeah? Thank you.

B

All right, so we have um the community member on the call- and I forgot you knew I'm sorry.

H

B

H

B

Okay, that's I can do that android. um So thank you for hopping in and participating.

B

Basically, we had a request for ed for for adding for for developing or documenting if we, if it was in existence, a plugin, a swift plugin for valero, then a kind so hopped in and did some research and found this plugin and then the developer of the plugin, which is andre, hopped in and said, hey I'll, be glad to uh extend it, and so you take over and let us know the status of so what we would like to know, because my question is: should we include this in the list of supported, plugins plugins supported by the community?

B

So let us know where you are with this and where you, if you're willing to continue maintaining that please.

H

Yes, uh nolan did you want to say something.

C

Yeah, I just wanted to jump in um any person that makes a community plug-in can make a pr uh against our repo. We have got our lit. Our lists are in the valero website on the in the repo, so anyone that maintains a plug-in can absolutely add their plug-in to our repo in the website.

B

So actually that was the sec. My second item, so it has to come from the maintainer from the developer.

C

It doesn't have to, but I mean.

B

I mean do we do any due diligence? It's like anybody can put anything there. Even if it's tail do we have any rule of thumb that we check, because if we did, I I forgot, or I don't know we.

C

Don't currently have a vetting process right, yeah.

B

C

A

Yeah, so we um we have two different tiers, essentially on the the uh plug-in page. One is the uh the plugins that are being maintained by the valero maintainers and the others. Are the community maintained ones so far, none of them have gone stale, but if they do, I think it would be worthwhile to actually do a check every now and then and see if they are still working with with new versions of valero.

A

If this plugin has gone stale and doesn't work with new versions of openstack swift, for instance, I think that would be worthwhile to check in to look into at a regular time interval.

B

I think I think if we have a guideline, excuse me if we have a guideline like that, we should write it down somewhere argument.

A

Things all right.

B

Like okay, what is style? Is it two months two years you know what I mean. It's like yep,.

A

Yeah, I I I think that would be great.

B

And then we can use the same guideline to maybe vet at the inclusion of a plug-in.

A

B

I'm not saying we have to have it, I'm just saying: if we have have, if we're going to be tracking, then it makes sense that we also think.

A

If it's yeah, if it's a process, we should definitely document the process. Abby are you listening at all.

G

I'm here for documenting things.

A

Yeah, uh I think that makes total sense.

B

I'm all for documenting it, because I know two months from now. If this happens again I'll be like what what do we have a process so.

C

I just want the the current process. Is we invite people who made their own plugins to create a pr against here in terms of vetting and cleaning it up?

C

uh I don't want to decide that right now, the reason being, I think, that's a complicated process and there's geek keeping we have to do when we vet something and uh there's a rabbit hole that we go down of. How do we decide who gets in and and who's who's good and who's? Not so I don't want to define that process right now, but as it exists to to add yourself to the web page, it's as simple as a pr.

B

But anybody can do it doesn't have to be in the maintainer.

C

Right, correct.

A

When we first started out, though the plugin page, I actually reached out to the maintainers of different plugins and asked if they wanted to be a part of the web page, which I think is fair, and they said yes,.

B

All right andrei! So if you could open the pr.

H

Oh, this was kind of different because the pr was opened by someone else and I didn't see it. Then I got from one issue. It was linked to another and I saw this today so I wrote that yes, I am the maintainer and like I'm okay, with the listing.

H

uh You got it the point.

B

So excuse me one one question that andrea was asking is: uh if we, if we would like to have a no, was it a an object, store plugin, I forgot exactly that you would be willing to also develop that.

H

Yeah, I can share my story.

B

With this plugin, if.

H

You want just shortly uh well, I was looking for a plugin for the openstack suite. There was one by uh it was cisco sso or something like that, but it was uh last comment was two years ago, so I already knew that it wouldn't work. It's like with the open source, and there was no note about if it is maintained or not, but in there was one issue where this guy said that he is not longer working for the cisco, so he stopped maintaining.

H

Therefore, I copied I didn't do fork because I didn't know what is broken so I started to go all over and implemented this, and currently it has implemented only the object. Storage part. The block storage part, is not implemented yet, but as far as I know, it's not something complicated. It's just around 100 lines of code and doing some test automated tests, so that I'm sure, if when this is community used, it just works with all the versions of belgium.

H

This is one of like my my main concerns that a little about maintaining it's not just to implement something, but to make it work with all the versions and all the changes that happen. So I think I will have to spend some time with the testing and I don't need to automate it.

A

So um what the the exception there, um let me see how I want to phrase this so for for the plugins here uh when you do a pr, uh you essentially say what's supported and what's not supported so for here we got object, storage, supported but not volume, snap shutter, because we're not implementing sender right.

A

So that would just be say: yeah we got object, storage and that's cool, and then we got a yeah. That's it essentially, and then we will link to it.

H

Okay, so so it could be, we could start with object, storage and rest will be implemented later. Yep.

H

B

Okay yeah. Thank you so much for that um next thing on my list is: we are hiring hiring a product manager for the for us for our team. So if you know people um there is a the link to the job description. Is it's a bit generic, but it's for for the validity senders send them this link, so they can apply.

A

Or if anyone on the the call here would like to apply for for a pm role, please do so.

B

Oh yeah, absolutely anybody listening to this. If you have questions also about what the job is or anything you can think of skipping me. Definitely I don't want to volunteer.

F

B

I definitely I'll be more than happy, then clarify any questions regarding to how we work.

A

B

And uh the last item is jonas, got a promotion. Congratulations. He is now the head of community management.

A

B

The boss, people of our homes,.

A

I'm not gonna let the power go to my head, but thank you.

B

Yeah well deserved.

A

Choice, um any any other questions comments for kalisia here.

A

All right and we'll dive into nolan's discussion topics, we've got tons of discussion topics today. I love it. Yeah.

C

um So the first one is um probably tossing a hand grenade out there, because we could talk about this forever, uh but I want to get a temperature check for folks uh on whether or not they want to move. Excuse me from our zen hub count on board implementation to github projects.

C

um I was talking to michael michael, our current pm about this yesterday, he's, unfortunately not able to be on the call, but something he mentioned to me was that contour is on github projects and when we first looked, github projects did not support cross repo, linking which is really important for uh valero, because we've got all these plugins and the helm chart.

C

um Github projects now have cross project linking which is cool, and my motivation for proposing this is github projects puts it all in one place.

C

We have it all in github, folks, don't have to download a browser extension or go to a different website to get it. um I have not looked at other alternatives to this, yet this is the only one I've seen so far um in talking to the contour team.

C

um Some of the downsides to it that I I know of right now are that um a github issue that gets created uh out of the box does not automatically go on a board um and there's there's multi. Oh another advantage is there's multiple project boards, which might be useful.

C

um So one thing we do is uh we can see on our zenhub board whenever a new issue comes in, it automatically goes into the board without github actions. This doesn't do it. This probably doesn't mean anything to anyone, but the maintainers um and there's no concept of epics in github projects. So we can't have.

C

uh We can't mark an issue as an epic and like link a bunch of stuff to it as a to-do item um other we can do it in kind of a hacky way, but there's no formal support for epics and then finally, there's the one-time switching cost of moving stuff over. uh I don't think there's an automatic data import. um So um that is my question. uh I don't really have a pitch for github projects other than to say these are the two. These are the advantages and disadvantages. I can think of right now.

B

I have opinions um the having yeah, I know what you mean, because I I used it for a few other things. You do have to add like add a card like an ish like what you're saying that an issue comes in is up as a new issue doesn't get automatically put anywhere on the project. It's super annoying now with that said I'll, be on board with it, because I never look as in hope.

B

If we're not having a meeting where everybody is looking at the same hub, I never go there because it's not even so much that it's it feels like it's a separate tool. It's like it's taking me out of my contacts because I'm not a first averse to using multiple tools. It's just that it feels like it's a separate tool that doesn't help me, because when I look at through a column is like the issue, is there the related pr? Is there and then there's the epic?

B

Is there too, and it's like I have to contour like contour myself and change so many things to like get a drill down vision. That is helpful. That is not worth it. So between having something that's totally useless for my day-to-day and trying something that has one annoying thing but then once that's in place can be very useful. It's useful to see the lanes and see what what you know what you did and see actually like a little. You know psychologically rewarding, so I I we I'll be very on board to.

A

A

Anyone else have issues with zenhub or issues with github projects.

A

Not issues in but issues.

C

With I'm also curious for folks, not in the maintainers team, do you check xen hub at all? uh We we broadcast that that's what people should be checking, but do folks not inside the maintainers team, actually look.

F

I've been mainly just tracking release, dates and milestones and stuff, but mostly it's just the issues and the pirs. For me.

A

C

Feedback, I think um also with the the minor annoyance of uh issues not automatically being put into a board um that product project automation the github dash product project, automation, action. The first example they have is moving new issues into a project, so we would have to install this on all relevant repos, but there's already a uh stock action that we can throw in there. So um excuse me the this is.

C

That's something that can be worked around with actions. It's I've looked around and github has been getting a lot of requests to just do it by default, but so far they've been resistant to it. So it's not a thing um that they're doing yet.

A

And this would be a project board on under the valero repo or the org.

C

uh This would be on the core valero repo, and then it would point out to the um the aws gcp azure csi exam example plugins and the helm chart repo. I think, okay, I think that's everything uh so seven repos in total.

A

And I'm just wondering if, if this would work to automate putting um an issue in another repo onto a board which is tied to a different repo.

C

We would have to test it, I don't know, uh I know you can make a project that has multiple repos in it. But I don't know, that's a really good point. I don't know if you can target a board in a different repo. With this we may have to look around um right and I asked cappy. They do not use github projects mostly because they don't own the repos, but that makes sense nobody really owns to get uh or the kubernetes and kubernetes repos.

C

So um but yeah part of part of another part of the motivation is, I see.

C

Folks, don't tend to use zenhub, so if it's not, if folks aren't using it, then we should probably look for something else instead of trying to force folks into it. I put myself in that category too. So.

B

Anybody would have a problem moving away or or concerned moving away from zenhub.

C

Let us know we're not going to do it now, like not it's not going to happen today, but it I'm not hearing any strong objections. So I'm going to double.

A

Yeah just an easy one, open up an issue and then give a thumbs up or a thumbs down. If you want to do it or not,.

C

Yep, I I already have uh mike michael as a pm michael's vote is uh all of his other projects are on github projects, so uh he wants it, but he that was his opinion. He did not we're not dictating this.

A

All right, I want to move over to uh 1.6.

C

Yeah, so um the number one priority is implementing some form of end-to-end testing. From my perspective, I I like the thought that dave's putting into this in terms of what are we testing, um I have an issue out there. That's like a very, very, very, very basic test, like I think I was looking at starting at the level of did the command successfully happen and then building from there.

C

But that's like the the the number one priority um after that we looked at continuing to migrate over to controller runtime for our controllers.

C

I don't remember the three controllers we picked for the 1 6 release we may not have, but we um we should do that soon um and and get get working on those um for those who don't know we're, taking a phased approach to moving all of our controllers to controller runtime and cube builder and with the intention of once we get them all over.

C

We will eventually cut over to using the controller runtime client for valero uh resources and that will likely trigger valero 2.0, simply in terms of api compatibility.

B

um So the sorry so addressing the question of which controllers we picked, I don't think we. We definitely didn't pick uh I'm looking at the epic, but we can address that. Okay easily yeah.

F

B

The download one is almost done. It's one stupid test.

C

Okay, so then really we have.

B

I tested like the cli I tested everything is working. I just want actual automated.

C

Tests we were planning to get on one five. We.

B

Were planning yeah it didn't get in so.

C

Three more.

B

Really yeah, I just said he had started working on the backup one. I don't know where he is with that. So we could maybe pick two more. I don't know.

C

Okay, yeah we'll check with him when he's available, yeah and then the next one is enabling use of custom credential providers. So we had a basic design dock out um that karlisia had started back in april and we kind of.

C

There there's like a couple things wrapped up in this, but the overall guest is there's there's things that would be really useful to have so things like uh multiple accounts per provider. Right, you want to back up from one aws account to another uh in the same region. I don't. I don't know that we're going to tackle multi-region in a provider on this uh or um uh time limited credentials, those kinds of things.

C

So uh this is like the big feature we want to get in the rest is from our perspective, the rest is going to be maintenance, and I encourage folks to read over this and make comments on their with their thoughts. um I don't think we're going to be. I had made some comments about adding more scope to this, but I think I also walked those back later because I think they're actually different problems so um yeah and in full disclosure. This is something uh the tonsi mission control team wants. um They.

C

They may end up coming to us and contributing this, but we don't know um so. This is something our team is going to be tackling um and then, finally, uh for those who haven't seen rustic has released 0.10.0.

C

um We need to, at the very least, get updated to v0.10.0 and make sure there's no problems, but we have. um We have a lot of of rustic stability issues uh and by issues I mean github issues that have been filed. So what I'd like to do is upgrade rustic to v0.10.0 and then recheck. All of those uh see, if there's still issues are they are they rustic issues or are they valero rustic integration issues? So um yeah uh rustic went about a year between updates um we've.

C

We've also had some talks about maybe moving rustic into a fully fledged plug-in model, so people could take, take a take rustic out and maybe use their own plug-in or their own file system plug-in.

C

Those are really early stages. If you would like to see that, please let us know, um I know, there's there's other folks out there using copia, and I know she wanted to play with a prototype of that um since we're talking we're talking about december time frame for one six, the immediate term is going to be rustic zero 10, but we may start talking more about a plug-in implementation for file system level backup.

C

So those are the the four highlights for one six: uh that I've got.

A

Awesome any questions here, long.

G

Okay, uh I'm next okay, so I have two issues um I want to discuss, so one of them is the backup item in parallel and and the other one is the plugin timeout, so the backup in parallel. um I I looked. I looked into this and I found that most of the backup item that we back up is pretty fast. It made a data array, it's pretty fast, so we're not really concerned much about them, except for the persistent volume the precision volume is when we actually send out the command to take a snapshot.

G

uh When we, the plugin, like, for example, the vsphere plug-in right, we we sent out uh to the center to take a snapshot of the pvc, even though it takes a second but sometimes got stuck and stuff the main problem, and when we want to execute them in parallel. We also want to um these things. You know being executed.

H

Very fast, so we can.

G

um Snapshot these pvc as close as possible to each other, so this backup resource in parallel. I think I can mainly focus on just the pvc, but when I look at the current way, we doing our backup part, and I found that when we back up a part, we look into the pvc belongs to that part and then we snap and then we execute.

G

We also back up the pv bound to that pvc. So that part pvc pvs is linked together in one. You know like we can look at it as one action so in order to we, we want, uh if we we ultimately want to. You know, snapshot these pvc together, but the way we currently um implement the backup in that data in the data path right now is.

F

G

Do the per pvc per part so because of that I propose that instead of doing pvc in parallel, we can backup post in parallel and when we execute the part, we will execute the matching of the pvc and pv belong to that part together in that one chunk, and then we can run them in parallel.

G

So that way it you know it satisfy what I need, but also it keeps the it keeps the the current workflow of the backup data fast almost intact. We only need to manipulate only one small chunk of the code so that it, when you do backup ports, they just back up these parts in parallel and that's it. So that is what uh I propose and I I think I'm gonna write down a desired sometime this week or maybe next week, so we can discuss in detail.

G

But that is my number one item on the discussion list.

G

It does anyone have any opinion on that or we can pop it offline in slack.

E

I think I want to avoid special casing things, so you know pvcs and pods and all these other things are kind of handled in special cases, and I don't want to add more special case code, where you know we we do pods in parallel, but nothing else, and so I'd like to have a design that kind of looks at all kinds of different items um and they have a generalized uh solution because otherwise we wind up like oh pods, are special and now you know, there's other special things and it it gets weird.

C

Yeah I agree I I would rather start trying to do it general and like we need a really really really good reason not to do the general one. The other question I have is how this affects the plugins, like with the plug-ins run in parallel or are you? Are you spinning up multiple plug-ins to do this work?

C

I think you would end up doing it if you just like in a naive implementation, if you just spin up a go routine, uh to run the functions as a as they are, I think that's going to end up spinning up a plug-in per resource.

E

I thought it. um I thought I'd check to see if there was already a plug-in running.

C

It might oh yeah it does. I I need to look to see if those plug-ins can handle multiples, though.

E

Yeah because they may they're going to get a grpc request and I'm not sure.

C

Right they were not designed for this, I mean they. If they need to be updated, they should, I just think in a naive implementation.

G

I think that's think, that's a good feedback. Let me let me uh to be honest, I need can each sound like. I need to dig into this plug-in a little bit more to.

F

G

If they support, you know multiple running in parallel, if.

E

If you're really concerned with that one I mean that's that actually has a radical.

I

E

Yeah you can uh check in with, um or you know we can. We can take a look.

F

E

It the code's all there in the repository um I haven't done an architecture doctor yet, but we had to do a lot of changes in order to support the project. Pacific architecture.

G

I work with sima and sukana in this vsphere plug-in and from what they update to me. They did move they plan to.

F

Move accordingly,.

G

They're doing the positive volume uh plug-in and they want to move it into the backup item uh plug-ins, but.

F

Besides that, I'm not really understanding.

G

The detail for that yeah.

E

G

E

There's a lot of big changes, so you can you can you know bug me a little bit on that, but you know we're trying to get the release out for that one right now.

G

Oh, thank you in that calendar. Let me talk to you offline.

E

J

Make this a generic case.

C

um We definitely need to make sure all the all the plugins like plugins can.

F

Fan out because.

C

We could be running it against anything since.

F

I

C

I

Match against anything, and the other thing is.

C

uh Since they're fanning out, um we may bump into uh and they run in the valero podge, uh you may run into valero pod limits, so we might have to look at as part of this. We may have to bump up the valero pod resource.

E

Yeah, we may also run into uh limitations from the underlying storage system.

G

Yeah, so I I I will take a look at these um items and I'm gonna work on that item in the next, uh maybe one week or two all right. Thank you. uh The second item, like I want to talk about, is also related to this, which is the uh the timeout right. So when we um so I have a I have a design for that, and I uh it is in a pr right now.

G

I can talk about quickly here, but basically the the the pr is about, instead of just focus on the timeout, from one plugin or for the plugin. What I really ultimately think I want is that when I run a backup, I'm gonna run a backup on on that positive volume, and it have to run some plug-in these all of the on that object had to be finished within a certain amount of time.

G

If I, if I put in the timer for this plugin but forgot to put in the black camera from the other plugin, but both of them running on the same object and the adding together, they can exceed that timeout. So, instead of specify for hov individual plug-in, I just want to specify that, for example, for pv, I just gave it two minutes. No more than two minutes every two minute time out doesn't matter how many plugins is running on that tv.

G

So that is the general idea, and I also um the idea also behind that- is that not all the application having the same timeout requirement. For example, if I run a custom distance back up, I mean it takes five minutes to do.

G

The snapshot is no problem, because I don't have any uh consistent requirement on it, but if I run it on an afternoon, then before I run back back up, I s the application and that caused the application to block all the user interaction in such case, then, I need to um take this and I'll run this plugin within a few seconds, a few milliseconds that the application required and this application might have different uh requirement for uh for each of these. So I I just that is the design that I propose.

G

So uh I just want to bring it out here so that it, if anyone have any opinion on that one. Otherwise, you can put it in the review for that.

E

um Are you passing the timeout down you're, not passing the timeout down to the plugins, yet.

G

Though, are you nope, I'm gonna? No, I will not passing the timeout to individual blocking I just when we execute a function called execute uh action. I should wrap around that with the timer.

E

So the whacking the function isn't a bad, isn't a bad thing, but I'd like to also have timeouts go down to the plug-ins so that they know when they're supposed to be done. And if there's a bug in the plug-in. That's one thing, but you know so, for example like if, um if you have a snapshot operation, that's taking a long time and you cancel the function you're going to wind up, but the snapshot actually succeeds. You're going to wind up with an orphan snapshot lying around and there's no way to clean that up.

E

So um it'd be nice that the plugins were participating in this and they knew that the timeout that they'd exceeded their timeout, and maybe they should cue up a background task to delete whatever they just created or whatever.

G

You know you meant to pass the plug-in time to the plug-in itself right, yeah.

E

Yeah, so the plug-in is the first thing that's responsible for for satisfying the timeout and then, if the plug-in exceeds the timeout by some amount, you know you can take 10 off or something and then you'll actually whack the function.

G

Okay, that's actually a good suggestion if that involves to change the plug-in api to pass that parameter, not sure. Currently, I need to take a look deeper into the uh the spec of the plugin right now to see what part of matter is accept.

E

Yeah, I don't think there is a parameter at the moment, but that would be good.

A

Just a quick time check uh we're almost out um and uh yeah. Please continue this discussion up on the issue, the pr and on slack um dylan. Do you want to talk about yours uh quickly before we wrap up here.

I

Yeah this will just take a second um last community call. We briefly discussed how we were interested in taking on the restore progress, design proposal. So um we went ahead and did that and then I I think it was brought up that we kind of discussed bringing this up into phases, but um as we went through it last week, pranav kind of came up with a good way of solving this, um taking the two-stage approach that we discussed so um we're just simply sending feedback.

I

If anyone has an opportunity to take a look at that- and um I don't- I know that one six sounds like a tech um tech debt release, but um if there is any possibility we get this in there. um That would be cool, but I'll leave that up to you guys just throwing that out. There thanks.

A

Thank you dylan all right quickly. Before we wrap up nolan, do you want to go over the community? Shout outs.

C

Yeah um really quickly, thank you to anthony bett for providing the fix for the backup storage location, to avoid spinning up object, storage, plugins bef when uh the back storage location wasn't actually due for checking, um and uh thank you to. I do not know this person's name um uh gliptak uh for uh gabor gabor elliptic um for updating go to uh 115 on our build image.

C

uh That's super helpful.

C

um Next uh ariana um reported uh this issue back in may uh and it it admittedly took us some back and forth to identify the issue, um but this uh was a very important bug that he identified and think we thank him for providing replication cases and sticking with us and then finally, uh thanks to pranav for getting the um design doc up for progress reporting and thank you to uh fong for the design dock for the timeouts.

A

Awesome well, thank you all for for joining today. uh This is it for, for this week, have a fantastic rest of the week. Everyone and see y'all next time, bye.