VMware Velero Community Meetings, 30 Jun 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Velero Community Meeting/Open Discussion - June 30, 2020

Description

June 30, 2020

Status Updates
carlisia
BSL controller to handle validation + update BSL status phase - PR ready for review (this changes the behavior of the Velero server): https://github.com/vmware-tanzu/velero/pull/2674/
WIP: add webhook validation to BSL create/update
PR reviews owed
ashish
on community support last week
worked on some internal build setup and testing of the build artifacts
Opened this issue last week Issue 2673
Currently working on some load testing for restic
nrb
Was out for the end of last week for family health issues, didn’t get everything I wanted done.
Reviews!
Updates on docs, getting Jason S up to speed for implementing new commands
On community support this week

Discussion topics
[nrb] Moving code from backup & restore controllers out into the pkg/{backup/restore} directories
Helps with kubebuilder/concurrency
[stephanie] Scheduling a backup and restore focus meeting Monday July 6th, more info to come here

Contributor shoutouts
Vishnu Itta for the volume snapshotter progress design doc!! Let’s get that merged this week (for real!)
Marc Campbell for the restore hooks design doc!
Phuong for the work on the ordering design doc!

A

Hello, everyone and welcome to yet another episode of the Valerio community meeting and open discussion today is June 30th 2020 we're going on week umpteenth of the the COBIT stay at home regulations, so we're all staying at home. So let's get together and talk about something fun. Let's talk about Valero, so I'm gonna share my screen. If you want to add your stuff into the Hat can be. If you have any discussion topics or anything else, do you want to talk about? Please add them.

A

There I'll share my screen and let's get started all right, so get a few status updates here. I'm gonna, let Carly Zia go first today.

B

Hello, so I have this PR ready for reveal. That adds controller for the backup, storage location, and this does change the way the letter behaves. So currently we write right way and the lair is installed. We validate that there is the all backup, storage locations they exist are valid and if even one is invalid, we've we halt the system. So we had a request to change that, and so this changes that so the level will never halt.

B

It will validate any backup, storage location and if it's, if it's, how good it will mark as available and if it's not it will mark, is unavailable in through an error, but it will continue and if everything is unavailable, if you have no backup storage location or they are all invalid, then pretty much you can't you can create a backup and you can't sink any backup.

B

So you can't do anything, but the valar server will still be up and running, so you wouldn't be crashing and making you wonder direct crash because of this because of that. So you just have the error message there and you know exactly why you can't do anything. So I think it's a pretty cool improvements and appreciate reviews. I have to add a web fog validation to to the creation updates for for the backup, storage locations and then I'm, gonna yeah.

B

That's it's in and I'm gonna be working on and not converting another controller to DQ builder control, runtime controller stuff. That's it and I need to do PR reviews.

A

Sounds good any questions, comments for Alicia and the PR here.

A

Alright, moving on to Ashish I.

C

Was in community support last week, so a lot of github issue, triage and stuff. The apart from that I worked on some internal field set out band like verifying the artifacts generated by the build were actually usable, so I was doing that I.

C

This other issue came up last week: issue 2, 6, 7, 3 e, which is an ask to Auto, include CR DS when backing up custom resources, even if during backup and restore actually even if, like the include cluster resources flag, is not provided so I opened this out and I had a discussion with Andy about this on slack I know put this issue up and then currently I am working on some load testing for rustic other than that.

C

To back up working on load testing for a stick, if any of you have done any sort of load testing on using restate, we would love to hear from you and hear what your.

C

Experiences were and what what are? What is the load that you're on and what was the outcome? So if you have anything to contribute to that, please let us know we would.

D

Yep on the CID thing: are we currently then restoring the C or D? Then the CRS as part of the restore so.

C

No, the order is not the problem, it is you need to say, I want to include cluster resources because CR DS, our cluster resources, you need to say, I- want when I put from this restore I want to include cluster resources during restore. So even if your backup has CRD in it, if you don't run the restore with include cluster resources to see our DS may not be applied. Sure.

E

Did you did you confirm this yourself because, like I swear, Steve fixed this for one point for like I was positive? This was a bug that was fixed in one point for I.

C

I did not see the code that in the place where I thought I should, but we can, we can double check. I did not run tests to confirm this. It was just cold inspection and and discussion over slack.

D

Yeah I know the the VMware supervisor cluster. They want to restrict the creation of CR DS and that's probably going to be a headache.

C

D

We'll have to come back to that when we start looking at supervisor clusters, some more, we may have to slip supervisor cluster.

D

More restrictive create cluster, maybe restrict the use of it, create cluster resources flag. What else would be in the cluster resources.

E

Persistent volumes cluster role, but can't remember cluster roles with cluster role bindings, which one is in the clock.

F

E

F

E

A cluster resource.

E

This year,.

G

E

These themselves or the cluster resource, some CR DS, can have their custom resources defined as a cluster resource as a cluster level resource.

D

E

E

Lots of things yeah.

D

I know I, think we're gonna have to have a sit-down and see how these things intersect. So currently it would quote unquote work because we're gonna run bolero with the cluster admin role. What you normally don't get, so we can, for example, slam CR D's in, but it's probably not the mechanism. They want at least not at this point, because it would really let you backdoor things in by creating a backup with CR DS in them and then restoring the backup and normally you're restricted from creating CR DS. So that's going to be painful and.

C

Names Vickie Barton's, if you're not able to create it, I'm, not sure how you how it ended up in your backups.

D

No, so the user isn't to the whole.

E

If you could get, if you could do a get operation, then you should be able to put it in the backup, but the create would be the failure on restore time. So.

D

There's a special path for creating CR, DS and typically the user doesn't get cluster. Add them there's another path to the app operator that allows you to create, get the cluster admin role and create see our DS also namespaces need a little special vSphere stuff in supervisor cluster, so they actually get their resources allocated. Otherwise they don't work. So we were planning to not do namespace automatic namespace restoral for supervisor cluster, yet yeah I think well after well after work this one through yes,.

C

Yeah I'm, before starting working on that maybe or whoever starts to work on that, we should verify that what is in the issue is actually true by running some tests. I'm going to add that to the issue comment.

A

Any other questions comments for Ashish.

A

E

So, just to let everyone know, I was out at the end of last week's fame and health issues, so that was kind of abrupt, so I know there were some reviews and and some design documents that we wanted to get in last week, and that did not happen so I apologize there, so I was doing a lot of reviews and updating on Doc's, for example, I was trying to get Jason's grano up speed on implementing some new commands.

E

It's also doing some investigation on the rustic stuff that that she was talking about and some internal build stuff for our vSphere plugin see. I was doing some thinking on prep work for concurrency or worker pods that I'll have in discussion topics I, don't think. I have a whole lot else, I'll be on community sport this week, so I might have a little bit of a backlog from yesterday.

E

I think that's mostly what I've got.

E

A

That sounds good. Any questions comments from for Nolan.

A

Anything you need with regards to the updates on Docs or and your reviews.

E

Honestly, I need to go: go back through I need to go through my email and all my get up stuff. Still so I don't know so yeah right now, I know, I know it's in the contributor shoutouts but I know I owe Vishnu a reply on some things, but from my perspective, like for the volume snapshot of progress, I've left out a word: there volumes d'etre progress, design, I'm ready to put my approval on that, and we can move that forward and I know. Excuse me we intended to get the.

E

Restore hooks design meeting going this week so I'd like to get that scheduled when that's a good time for everyone. Those are the two things that dirt that came to mind. I'll stop my head: hey.

H

Nolan for the restore hooks design meeting it looks like Monday morning is gonna, be the best time so I'll be sending that out and coordinating with Jonas for getting that out on Twitter and the community meeting, and then we can just go from there. Awesome.

A

All right, I think we are ready to move into the discussion topics, so we should.

C

Maybe add that to the meeting notes that we have scheduled. So if somebody is looking at the notes, as an afterthought can I'll.

A

Do that exactly.

C

A

So no one you wanna, take it away with the discussion topic. Yeah.

E

So I was thinking about how we can make moving to worker pods or even.

E

Moving to our new moving to making Claire a little bit more distributed, whether that's more distributed controllers or worker pods or something else, make that a little bit easier and also possibly making our libraries easier and moving making the move to queue builder and controller runtime easier and as thinking about there's an issue out there, which I need to go, find thinking about moving the code that sits in the back up and restore controllers into the backup and restore packages.

E

So package backup and package restore there is an old PR from Steve die, lifts I think it was the backup controller into package backup and he also had a demo PR or maybe it was a demo branched.

E

I think he might have shown on the community meeting a while ago that showed a command that ran inside worker, pods and so I wanted to float the idea of starting to maybe not doing huge moves, but starting to move little bits of code from backup and restore controllers into package, backup and package restore as as we're designing the worker pods and the the controller or or the concurrency, so that when we get to actually doing that work, it's a smaller lift and we're doing the code cleanup.

E

Now, since our 1.5 release is going to be mostly focused on restore hooks I'm willing to take on that work because I think it's mostly going to be shuffling things around.

E

I'm curious what other people's thoughts are on that Nolan.

B

The do you think that logic needs to people the little users or obviously I'm talking about developers, wouldn't need to have access to that logic.

E

Maybe III did wonder about making whatever that is just making package back up and back into restore public and I would have to think about what that API actually is, but I I'm, not sure I, see the downside, but I I don't know I since I, don't know exactly what the API will be.

B

So if you look at the PR that I listed there on that went out on today's agenda, I put the the logic for the backup controller for the for the sorry for the backup.

F

B

Location I put any logic that was that did not belong in the controller I put it behind internal in a package called velaro. Okay,.

F

B

I, don't see that anybody would need to use that logic now going forwards with moving I was going to make removing the logic part of the computer conversion.

B

So we definitely should I mean whether with visual parallel or make a part of that conversion, or whoever does it does the conversion to. We will figure it out, but we definitely should settle on in the place and maybe it might be two places. My personal opinion is we put, should put all the logic behind the internal directory, and if people need to use, they should tell us and tell us why. So we can properly design things design the public API, because then we're going to go in to help support that API going forward.

B

So until we know what people are using this stuff for I'd say: let's make it our private and in in yeah, just just expose what we have to what we know people are using in when we know how they're using but we'll be able to properly design the interface yeah.

C

That's fair that'd, be the move alone, be the cooperating change for people if somebody's already using things. If.

E

It's sitting inside the controller, maybe.

E

A lot of the stuff I'm thinking of I'm, literally going to lift it out of, can the controller methods which most of those I think are private, they're, not exported and I.

E

Think a lot of it is to get get them out of the controllers, because we're gonna need in a worker pod design. You're, not gonna, run the controller on every worker pod. You would be running it on. You would be running controller.

E

That would there be a controller grabbing the backup and then basically assigning it somewhere, and then that worker would be doing the backup logic. So we would have to extract the backup logic anyway, which okay would be some sort of breaking change and that's that's kind of where I want to go with that, so that it's kind of ready- and we don't have to do that later plus.

E

If we pull that out now, it makes moving the backup and restore controllers to control a runtime, easier, I won't say easy, but it makes it makes that work. A smaller left, India.

B

It might be in my brain sometimes, but it goes goes back to the issue we've been talking about, which is. It is not none of that as part of the packages that we intentionally export and we need to start surfacing that a lighting people know but and.

E

And I can start with the stuff that is literally and uh I. Don't view this as one big PR I view this as like small steps, so I can start with stuff inside of controllers, private methods, I think so.

D

In terms of parallelizing things out into multiple pods, what's the current bottleneck, you know I mean we're, you know, because we should just be walking across the namespaces in the API server. So is that really a CPU intensive thing? The current.

E

Honestly, the current bottleneck is, we have no leader election or any anything to prevent race conditions and grabbing a new backup. So when a new women- oh that's.

D

Why that's why you might have a trouble implementing it, but in terms of like running things inside a single pod like oh, what's the push to get this pushed out, I mean I, know I went to like a scale out data mover because the I/o stuff got crazy, but are we really seeing that much in terms of doing the control path? I.

E

Think the big motivator there would be rustic.

E

Which is not managed on these pods anyway, yeah.

D

Because rustics already scale out yeah.

E

So the the other thing I want to do is revisit so the other thing I want to do is revisit the worker pot or the concurrency and scale out design dog and not just focus on worker pods, but focus on what do we want to do and what are the approaches? Because I think you you're asking a good question Dave?

E

What is the problem yeah? Because if.

D

Because you know, if our problem is, you know throughput on on traversing the API server we ultimately we're banging on the API servers. Typically its thing, though it isn't it it's not really distributed out.

D

Then I'm, not sure on me.

A

D

Single node with failover right, so you know we distribute out and bang on anymore.

G

E

Comes down to tuning the API requests and stuff well,.

D

Yeah but it also, maybe, then we have to push some of this out in the stack we have to say hey in order for backup to run efficiently, we need for API server to become more threaded, more parallel, API.

C

Server is like, if you have multiple instances of your control, plane machines, you can hit any of the API server to get what you want.

D

Okay, so, but we can check and see how much the actual load is and those all go back against you I.

C

Can get it on there.

D

Going back again PTC, you know it so be interesting to.

C

D

Bottlenecks are here and really characterize the bottlenecks right I mean if we're, if, because the control path, if the control path needs to go faster, you know. That's that's a completely reasonable thing to do it. It's like, but is paralyzing it onto multiple worker notes. Is that going to make it go faster? Are we gonna hit another bottleneck as soon as we do that.

C

So I'm not sure how is the discovery shared across backups or is it put back up.

C

Their discovery, cash, let me have pink.

E

Discovery cache is shared right now.

G

Yeah, so that would be one that's.

E

When one part will have to pull out.

E

Well, currently that we're only running one backup at a time right, right, I, think I think right now, the client, the current limitation in Valero, is one backup in one restore at a time and then the the thing we want to the goal is remove that restriction and get to do lots more I, don't I, don't know the number, but basically max out the API server.

D

Any other word yeah.

E

You can't, you can't run any other work, but.

D

Backup backup seems straightforward, multiple restores at the same time, yeah.

E

Especially I was thinking about that and like especially if you hit the same Nate namespace, dumb boy I was thinking about that and the wave Valero currently works. If you restore. This is turning into like an ad hoc design session on on the concurrency. But if you, if you restore let's, let's take a namespace but I think this can stand in for a lot of different resources, assuming employers current behavior, where it doesn't overwrite anything the first. The first thing, the first restore job that restores something will put it in it'll, get assuming it successful.

E

The second one that comes along will see it already exists and will fail. No.

D

It's like a theory only right, but TVs, for example, are being in stored one by one and if the two are running, one might have door different versions and I other.

C

Is the other aspect is to support to support like concurrent executions? We might not just Valero has to support it, but like also the case of hooks right. If there are multiple backups running, you will have to be running multiple hooks or you will have to be have the ability to run multiple hooks and those hooks internally should coordinate amongst themselves to make sure only one of them is running.

D

E

Needs to go on yeah yeah, and this was like there's, there's a there's, a design document that Steve had kind of closed out, because it's kind of old and it was specifically designing around worker, pods, I kind of want to go higher level and say what are they? What are all these kinds of problems and then say what is what is the design that could accommodate them versus? This is a design.

E

We've been talking it about like the the concurrency model, as worker pods, but I'm not sure. If worker pods are the right design, they are a design choice but I'm, not 100% convinced. Yet you.

D

Gotta hear that you're maxing out, CPU and and I/o before peril. You know multiple pods really makes any sense right.

E

Like I could I could see that we just if we can get the Valero like we could use literal like something like a leader election that grabs the backup and assigns it. Then we could just say: hey, set replicas on your deployment and be done with it yeah we could make that work. Yeah.

C

E

But then we still have all these other issues of restoring at the same time, concurrency on your hooks, all of those things I think are still applicable, no matter what design may be, which use so.

C

Yeah, the worker pods not only make sense for CPU intensive, but even for I/o right. If you have multiple pods, you are hopefully scheduling them on different nodes and thereby not maxing out like the disk, you lands on just one node and causing I/o rates. Well, so we did that for the for.

D

The vSphere plugin right, we have the scale out data management, and so that's that's running as a node set, the one per node and then they try to grab because they're, actually moving data and most of the other implementations, don't actually move any data, so even know a large API server. Your actual backup winds up being 100 Meg, maybe not even oh yeah,.

E

Right because it's it's mostly the the manifests end up being that the normal, the normal max limit from @cd is 1.5 Meg per object. You can tweak that if you control a CD, but it like, if you're on a managed system, you can't so g ke ke asks those kinds of thing. I don't know if the VMware TKG solution allows you to to mess with that, but you do that and then we compress it so 125 Meg times I, don't know couple hundred and then compress it.

D

Yeah I, don't think there's any does that many large objects at maybe.

F

In general, you know I would.

D

Connect backups being the certainly less than a gigabyte right. I mean you're, not anywhere near maxing out IO, because you got to spend most your time retrieving from the API server, not rendering.

E

Power file right- it's HTTP, calls most of the time, so I'm curious is on I'm curious. If anybody else has thoughts on this I think to pop up real quickly to the other topic, I think Carly CEA's point makes sense that we move it to an internal package. For now until we like design package, backup and package restore because I have heard requests for hey, wouldn't it be awesome if we could just run a Valero Valero as an API, but I think that warrants design time and people can run it as an API.

E

If you just submit CR DS where's yours right.

D

That's actually the current approach, we're taking with say, for example, Veritas yep. It's Provera toss to wrapper Valero.

E

D

Then we want to have an infrastructure layer, and you know some of that was supposed to be the astrolabe layer and looking at how we bring that up.

G

D

Valero and bringing some some parts of a letter down into that I think where we want to look to in terms of like straight-up snapshotting.

E

So I'm not hearing objections here.

E

Does anyone else have any any kind of any comments concerns.

E

Look like it: okay, otherwise I think we can. We can move on. I was mostly mostly wanted to bring up this topic, because I think I think this code move kind of hits two things we've been thinking about, which is this this concurrency thing and the the overall code cleanup that we've been thinking about so.

A

Does anyone else have any discussion topics for today that you would like to bring up.

C

Backing up to the cold move stuff. How does that affect something like the volumes, not charter progress, because we want the plugins to be able to access some of these api's and how do we restrict access between what the plugins can do and what somebody else is called kingdom, because plugins are in fact, like third-party code right right, I.

E

Will have to get more details on what I'm actually moving yeah I think my thoughts were like I I need to find the exact issue, and maybe Steve's example PR that he had yeah didn't it. He had a PR open that was like moving stuff and I think this was maybe towards 1.3 1.4 time frame. They just closed it. Cuz it got stale I, don't think I would do anything as big as what Steve did I think. What I would aim for is like in let's say in package, restore there's like huge methods, yeah.

G

E

Like maybe condense, or at least take them and name them, but that is a really good point. Most of this, the volume snapshot. Our progress should be external to velaro. That is like yeah. We need to include the CR DS. We need to create them in some instances. I think I need to reread the review. Read reread the proposal, the.

C

Plug-In actually creates the CRE Valero has to include them lots.

E

C

E

Secrete the series with the CRS does: he are sorry. Yeah I see that over time.

C

E

So Valero puts the definition on the API server yeah and then the the plug-in would be responsible for managing the actual instances. Valero would be responsible for doing any of the backwards compatibility checking in marshalling between the tarball.

E

So, yes, that code may be caught up in the marshalling code may be caught up in this I.

C

Guess this is the thing things to consider when we move things so yeah yeah.

E

And we can, we can navigate that yeah.

C

E

And I think like I, think that marshaling code is in package persistence but I'm, not 100% sure you may be, but but we can. We can also talk I, don't think fish news on the call, but we can talk to him about that. Yeah.

C

E

Something I was going to mention on on his when I, when I dropped. Another comment on there was I would advise that when he, when, when we get to implementing, we should use queue builder for creating those new CR DS and including them yeah. Now that we're now that we're getting on board with that and we're, including a new CRD, we should be using that, even though it's it's it's an existing data type that's getting promoted, we should still be thank you builder,.

E

But yeah you're you're right that I think as part of the backup and restore some code will touch here. But exactly what I don't know, yeah.

A

A

Anything else with large the best topic.

A

Alright and I don't think we have any other discussion topics for today. So Nolan do you want to dive into the contributor shout outs, yeah.

E

I did a check in it doesn't look like we've got any external contributor PRS merge this week, but I just wanted to say huge things to Vishnu and eita for the volumes down. Try our progress, design, doc I know you've been patiently waiting for this, and thank you for the back and forth so maintain errs. Let's, let's get that merge this week, yeah yeah, let's, let's read through that and make sure we get anything addressed also mark Campbell. Thank you for your patience on the restore hooks design.

E

I know we're gonna schedule a meeting for that on next Monday, this one's a big one and it's it's complex, but thank you Mark for your patience and the willingness to drive that and also I wanted to the.

C

E

Think long yeah, yes, I, have not had a chance to look at it, but thank you for for submitting that long. I will get to that this week. Thank you for.

F

You I I, do have a book fort ready to reveal to what I need to have the desire to go. First, I can dip. You know, I know that if we like next in their now next meeting, but it it's very soon, I mean it's. A the total code chain is about 100 live cool, it's very contem, isolated into one area, so the regression potential is very long.

E

Awesome, let's, uh let's get you put in for for a demo. I'll, be honest.

A

All right, I I, think that's it for today right so with that, thank you all for for joining today's call and have a fantastic rest of the week, see you all next week and bye. Everyone.