VMware Velero Community Meetings, 26 Jan 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Velero Community Meeting \ Open Discussion - Jan 26, 2021

Description

Jan 26, 2021

Status Updates - items working towards roadmap
nrb:
Reviewing Frankie’s PR for version negotiation at restore.
Reviewing PRs that multi-creds work depends on.
Tech debt items for v1.6.0 will be trimmed from the board this week and moved to v1.7.0
bridget:
Continuing work on multiple credentials support
Reviewing PRs
dave
e2e tests across multi-cloud
e2e test for multiple namespaces

Discussion Topics
Ritesh Patel from Nirmata will demo Kyverno

Pranav Gaikwad (@pranavgaikwad) Restore Progress feedback. Should the excluded resources be included in the count of items restored? Not included as per design.

Have a question? You can ask in the Discussion Q&A

Community Shoutouts
@MadhavJivrajani: Raise logging level for PV deletion timeout
@a-mccarthy: Update docs to clarify backup location and relationship with other data
@cpanato: [velero] Helm2 fix
@cpanato: add stable repo to the new location

A

Welcome everyone, and to the valero community meeting and open discussion uh today is january. 26 uh 2021. we got some status updates uh to go through and then we're gonna have a live demo by rutesh patel from nirmata he's going to demo cuberno and some really cool features on self-service backups for valero.

A

So that's going to be super interesting. If you have any other discussion topics for today, please put them in the in the hacking video as well and uh with that uh nolan.

B

Yeah, so I'm going to keep my update focused on items on the roadmap today, so I'm working on uh getting frankie's pr for inversion negotiation at restore time uh figured out.

B

um I've already got a review in for the code itself, and I've asked frankie to kind of split up that pr, so that some of the new testing scaffolding is in a separate one.

B

um Then I've been working on uh reviewing pr's that our dependents for the multiple credential support um so to get there, we've identified a few extra things that we need to implement first, so trying to get those knocked out and then finally, I'm going to try to trim up the uh v16 board, so it's more accurate and what we're actually going to try to deliver um and I'm going to get most of the tech that items moved to uh the 1 7 release, which were a big thing.

B

We're gonna try to get done in one seven is the valero debug command, um which will gather logs and hopefully be a one-stop shop for information. So um I think that's it. For me.

A

Sounds good and um did you want to go over the board next week.

B

Yeah yeah, we can do that um see where we're at on the the multiple credential support um and yeah.

A

Absolutely do that all right. uh Any questions for nolan.

A

All right, uh bridgette.

C

Hi everyone, um so I've been continuing to work on multiple credential support. As someone said, there are a few different pr's I've been trying to refactor some of the codes that I'm going to be modifying up front just to kind of make it a bit easier. So I don't end up with one giant pr: they'll be really difficult to review and that's coming along. Well, um I've got it working for backup storage locations. I've got a plan for how to do it in wrestling, and I've also um got some work in place for volume snapshotting.

C

So I feel I'm pretty confident with how that's going um aside from that. I've just been trying to make my way through the list of pr's and trying to get as many of those reviewed as possible. um I still have more to do but uh trying to get there. So thank you for everyone for your patience there. While we get through that- um and that's me.

A

Awesome uh any questions for bridgette all right, david.

A

You're, muted, okay,.

D

Unmuted yeah, um let's see so and then testing stuff, uh still working, uh let's see put in a uh a test that actually backs up like a large number of name spaces, just nothing in them really. But uh we had some issues with that or some issues recorded and looking to get all that stuff integrated with some of the internal ci cd pipelines yeah. So that's pretty much it for me.

A

Cool any questions for dave.

A

All right, uh then, we're going to dive into some demos here um so retesh from uh nirmata I'm gonna hand it over to you, so you can introduce yourself and uh share your screen.

E

Awesome, thank you chance. So hey everybody, um I'm ritesh patel from nermata, and um you know we're a cloud-based management platform for kubernetes. um Among other things, one of the areas we've been focused on in the last um year. A couple of years is around. uh You know continuous compliance and policy management for kubernetes and we open source a project called kiverno, which is now part of, um is now a cncf sandbox project, and I would like to share kiberno and also discuss.

E

um You know how cuberno can be leveraged for automated backups. um You know uh through valero and we have been uh you know as nirmata as our sas platform. We've been uh incorporating and integrating with valero for over a couple of years. So pretty much since uh I guess pre-100 days, so we are already.

E

uh You know we have uh deep integration into valero uh from our platform, and you know some of these use cases have emerged based on um you know, customer uh discussions right so in terms of uh you know just brief on policy management, and you know them in terms of what what it means and what it does um really.

E

uh You know in kubernetes there's a lot of configuration and there's you know various requirements for from different teams to to simplify that configuration management or maybe automate some part of configuration generation enforcing best practices making sure uh you know the security policies uh are not violated or enforcing those policies, and so on um you know, and in the community there's uh there's.

E

uh You know uh policy engines like oppa oppa gatekeeper, which addressed some of these, but we, you know, we saw a need, as we were talking to con to customers to to to to you, know, build a policy engine which was a lot easier and kubernetes native. So, if you're familiar with opa uh to to write policies, you would need to learn. uh You know a new language which is rego um in case of kiverno we've as kivono is a kubernetes native, uh a policy engine.

E

Everything is defined as kubernetes yamls and, and it can do three three things it you can. You know use kubernetes to validate configuration, you can use kubernetes to mutate configuration and you can use kevin to generate new configuration based on certain triggers and that makes it really really powerful to address certain use cases which in the past may have required, like maybe uh you know somebody to write and write an operator right so that it makes it really powerful and we are seeing a ton of different use.

E

Cases emerge uh in the community, um so you know, in short, it's it's giverno is, is not a general purpose engine it doesn't. uh You know you can't use it for uh you know uh just for, like you know, aws or any you know enforcing policies on cloud. It's purpose built for kubernetes and it uses constructs that are familiar to kubernetes users. uh So the learning curve is is pretty um you know low right. The bar to entry is pretty low um and here's a quick you know snapshot of how it works.

E

Basically, it acts as an admission controller and you know, sees all the requests coming coming in and and then based on, uh the policies uh uh applies. Those applies policies that kind of match. um You know the the configuration and essentially either you know, validate or mutate or generate a new config and there's uh in case of um policy violations. It can. uh It generates a violation report.

E

So so I guess what I'll do is next jump into. uh You know what we, how kubernetes can help with automated backups, and uh really I mean this is just one use case that I wanted to share. The idea is to share this with the community so that if there are other such use cases that emerge uh you know we would love to kind of. You know enable those and uh help uh you know, work work with the community on those. So uh quick on kiverno.

E

Well, you know, kuberno, like I said, is already a cncf sandbox project kiwano dot. Io is the website you would go to. uh Obviously there are links to you know documentation.

E

um You know all the policies that are these are you know coming from the community, so we keep adding these policies over here uh and also our uh the github page right and uh the last thing is, um you know we did a uh so we already have a blog. We did a while back on how nirmata, which is uh the sas platform that that uh you know, we've built that integrates with valero for disaster recovery and then recently we posted this um valero self-service backups.

E

uh You know using kubernetes just posted a couple days back, and this is what I'll go through in a quick uh demo.

E

Any questions before I jump into the demo.

E

All right all right, so I have a cluster over here. I already uh have uh kiwerno deployed.

E

And uh valero also deployed and configured so everything should be up and running. Let's just check that all right, so everything is running and you know we also make sure all of the crds are are already deployed right. So we have a bunch of crds, including um you know, valero crds and then you know, quebec cavanaugh has a whole bunch of crds around for policies.

E

So the first thing uh just uh will. Let me check if we have any uh backups.

E

So there are no backups here and- and we also check if we have any cluster policies, so cluster policies are uh in the short form of sepal. These are the policies that are uh that can be applied and that will be used by kiverno. So we don't have any policies here and what I'm going to do is uh I'm going to show you a policy that I'm going to apply to kiwan over here right and just kind of take a couple of a few seconds to actually explain what's going on here.

E

So we have a policy over here which is a match policy. It's a basic! It's a generate policy. Rather so what it's going to do is going to match any name space which has this label called, auto backup enabled right and it's going to just match that namespace and then generate a valero schedule uh with uh with uh you know some parameters right, like the uh it's gonna, take the namespace name and all the information that it can get from the request.

E

But then it's also just going to be right today, you're just saying: okay, it's a daily backup, and I have my storage location and volume snapshot. Location set up. So really that's kind of the simple use case where, if a developer wants to actually do this, uh you know today they would you know they would need access to uh to the valero namespace and they would have to.

E

You know, create the schedule themselves or ask somebody with access to the valero namespace to create a schedule for them now here what they all they have to do is add this label to their namespace right, okay.

E

So so what we'll do is we'll apply this uh policy, uh just you know, create minus f, auto backup policy, and we can see uh this policy should be simple, so the policy is already applied right and it's ready to go so now, let's, let's actually try to create this backup so I'll first create a namespace, nginx and I'll.

E

Add my nginx yaml to it to just you know: oops do that and I need to specify the namespace, so it's created in the night right, namespace.

E

Okay, so my nginx is running now uh next, what I'll do is? I will uh I'm just gonna copy this command from somewhere else, just so that it's faster, but I'm gonna just and add this label do this uh namespace, I think auto, backup enable uh the label is any added, and basically now I'm gonna, I'm gonna check.

E

uh You know if my schedule is created and you can see my schedule got automatically created and I can.

E

The backup is already the first backup is already created. So basically this is you know this is what in our in our product in their mata. All of this, you know happens through uh a user interface. So we have, you know built-in. uh You know, policy management to you know be using githubs, so all these policies can be applied and then we have this add-on. Where we have this, uh we have the valero uh integration.

E

Where some you know where I'd set up all the you know the backup locations, snapshot and all the schedules, and everything would show up over here right, so that's kind of in our sas platform. But what I just kind of did is everything through cli, that everything that I did through cli could be could could happen through the sas platform. uh But uh this is what I wanted to quickly cover um and and and you know there may be other use cases.

E

One thing to kind of note: is uh you know uh if, if you delete the namespace, obviously the schedule will automatically get deleted, so there's no kind of from a operator standpoint. They don't have to deal with. You know these schedules kind of um you know sticking around, so I think those kind of things are already built into kievano and uh and and and yeah, and it's very flexible. So if you want to label your schedules and backups and all of that can be done very, very easily.

E

So so that's that's it from me uh what you know I would like to kind of you know we'd like to get some feedback. I mean kiwano is an emerging project. We are working and engaging with different communities to kind of address. These similar use cases, whether it's around uh automation or whether it's around you know best practice, enforcement and we'd like to kind of get feedback, and you know build out that set of policies and use cases that would help the broader community.

A

This is great, thank you so much rotish. I know we've had multiple discussions on how to enable uh developers and and other users to do valero backups. So.

E

Yeah yeah. Actually I remember there was this: um there's this epic multi-tenancy and self-service. That's you know I kind of went through that. Then it seems to this use case seems to be addressed through kiberno pretty well. I think in most cases.

D

So um yeah this does look good. um How would a user do a restore.

E

So restore today uh uh you know: we've been we've actually added that in in nirmata itself in our in our product, I'm trying to see if there's a simple way for kiverno to do that. The way this. uh What what uh one thing? Maybe I didn't I'm? Maybe I went through too quickly. uh We actually when we have the backup, uh we are actually tagging these backups uh with uh with labels.

E

So we are tagging these backups, it's backup, type, auto and the namespace is, is nginx and in nirmata in our product we are able to then display these uh backups uh in to the to the user, so the user in nirmata, we have a concept of what we call environments and these environments are are essentially mapped to namespaces.

E

And here, if automated backups are enabled, the user would see those backups over here and then they could restore it so that integration we've done through nirmata. At this point, it's not something. We've explored with kevorno yet, but certainly that would be an interesting scenario. The the challenge there tends to be. You know the user. The developer does not have access to uh backup, unless you know you set it up in a way that they can access.

E

The thing is, how do you set up access to specific backups only for their name space and not across the board, and that's one of the things at least we are solving through nirmander mata at this point and not using kiwano for that.

D

Right right! Well, that may be why we're looking at pulling that inside of valero, but it's kind of scary.

E

Right yeah, so this is an interesting scenario I mean. Ideally, we would love full everything else kind of to to go through. You know kievano, and you know just automated, even the restore of the backups, but I think we we had to kind of take this approach because of access control.

A

Yeah having having self-service, backups and then asking an admin to to restore it, uh I think that's part of the way.

E

Yeah, it's part of the way: it's not all the way there, but you're right. I think at least uh it gets us. You know something right, because backup I mean restore is not something that again it's a more like an exception condition. It's like you know, not something you will do very frequently but backups, given how dynamic, like you know, application applications are created deleted.

E

I mean you may want just backup, for you know, even if it's a temporary kind of scenario, where it's not a long-lived application, you may want to have periodic backups for various reasons. So that's what we are enabling.

D

Yeah, so um so the way I'm hearing about nermata and the way I'm saying it's working, I'm a little confused because most of the um the the intro seems to introduce it as being like a security mechanism.

D

E

What you're actually doing.

D

Is yeah it's more like policy as.

E

I would think of it so policy I mean nirmata is a sas platform and policy management, and you know, is one part of it right. That's weird,.

D

E

Sorry, given are you talking about kiwano.

D

Yeah, so the way you introduce policy seems to be more talking about uh admission, control and security, but then the way we're using it.

F

Is similar to like what we do for like storage policies and vsphere correct.

E

So policies is broad right I mean today I mean most of the policies that everybody's familiar with are around. You know security and best practices enforcement. You know, part security policies. You know, uh policies to kind of ensure that your um you know configuration is, is compliant and so on. So that's what most people are familiar with, but what we are seeing emerge from the community and through adoption is, is this kind of automation where even for like using you know through for developer self-service?

E

You know triggering automation through labels through annotations, uh that's what we are seeing through through community, because and and the reason we we are seeing. That is because kiberno has this capability to mutate and generate configuration which other policy engines do not, and so that's kind of it's just something that we are seeing more happen more in the in the community, because it takes away the need to write for for writing specific uh operators and controllers to do this stuff. So it's.

D

E

It's it's design.

D

I'm trying to figure out like the the box that we can work in or out of so, for example, like the the policy that you generated there, um it's available to pretty much everybody right, so anybody who puts a label on the name space will get a backup into that. Using that policy correct, would it be possible to say you know, is you know what would like the user control or permissions on policies? Look like? Could we control who? Yes, we put a label.

E

Yes, because it's an admission control, we know we, we know the user, we know the group, you can control. All of that. It's up to you how? What kind of controls you want! You can even control name spaces. You can say: okay, only these name spaces can do this. So all of that is part of the machinery that has been built and it's available out of the box.

D

Right so, for example like for, like a self-service restore, I could see us maybe putting a label on that says, restore from backup x.

D

Is that something that would work and could we, for example, control it so that you can't access other people's backups by simply writing the label.

E

It should be possible, I I'd have to think through it a little bit more if we know the user, if you know the group we can, we can trigger it. Based on that, so I can take that back to the the kimono community and maybe come up with some of these scenarios. But these are great. I mean we'd, love to you know.

E

If there are more such scenarios, just let us know, and we can we can see what's the best way to to accomplish these, but I think it's certainly possible because the certain metadata, like the user, the group, the name space, all of that is available to us when we actually process the policy.

F

Cool any other questions.

F

I had one more but I'll.

D

Let anybody else go first, if they have something.

B

Yeah, I I just wanted to double check when you enable auto backup on the namespace. um It's mostly just enabling schedules right. It's not doing anything more intelligent like trying to look for diffs or anything in the metadata.

E

Not not today, I mean it's it's again it it can. You can configure it. However, you want, we can do more, just need to understand the use case and what how it can be accomplished.

B

Okay, yeah- and I just asked because we've had people ask for that in valero, but this seems like a really good first step, um doing diff changes and watching for events and stuff. That's a lot more complicated right.

E

D

um So the other thought I had is just so caverno is automatically generating, for example, the new schedule, that's good as the valero admin, this new schedule, pops up and go hey where's that coming from and I go and delete it. And then I assume caverno is going to create it again. How.

E

Do you manage that and then that's why? What we did is we added in the schedule we added a label said it said auto right, so the valero admin I mean we're using labels and automation annotations to actually let the admin know that hey this schedule was automatically generated, and you know you don't need to worry about it, because you don't need to do anything about it, the assumption being that the valero admin or the cluster admin themselves have set this up. So they are.

E

They know that this is happening right, because they shouldn't be completely surprised that the schedules are automatically being.

C

E

If, if because then, whoever whoever kind of actually one thing I quickly skimmed through here is you need in kiberno we have a. We have a role, cluster role for generating uh resources, and you have to add valero schedule in that role, so that kiberno can generate the schedule right. So there is actually an admin step and uh to to to be able to do that.

E

So if, if the assumption here is your admin uh or this valero admin or the cluster admin, is setting this up and knows that this is happening and we can tag everything through labels so that they can quickly identify which schedules are manual which schedules are created automatically and so on.

E

Cool, I think this is this is a good discussion and you know uh uh I'll. You know I would like to continue this. If there's any use cases around kiberno, please, uh you know reach out I'll share my email in the chat and I'll be also. You know, following the the community valero community, so uh yeah as things come up, we can. We can help out more.

A

Thank you great. This is great. uh Thank you. So much for for the demo and answering all our questions here attach. uh This is really cool. Thank you. Thank you uh all right. Next up we have uh pranav.

G

Yeah, uh this is about the pr. I have open about restore progress, reporting uh so the question, uh so I I got a feedback on the pr saying that there's a huge difference in the number of resources backed up and the ones showed as restored, uh which comes from the fact that the current progress reporting doesn't account for the resources which are excluded from the resource uh restore.

G

So uh I was just wanted to make sure like. um Is that what's the expected behavior.

D

um If anything I mean, the actual number of things are restored is the number I would want to see. um It might be useful to add another field for excluded, so that you can make sure that they all add up.

D

But I would expect that if I look at my restore I'll know, you know, and then I look at how many resources actually got restored, that the two would match up.

G

Yeah, okay, so originally so so when, when I had this uh design dock uh submitted, we we decided we would not keep the excluded resources in the account. uh I think uh I just comment on the pr and see what other folks think right.

B

Yeah, like I, I yeah, I think I agree with dave that we could add an extra field to make sure it all adds up, but um we, I I'm also fine with like having a pr that um does this first and then follow up with another one like as as long as it's eventually feature complete, I'm not too picky there.

G

All right, yeah sounds good. That's all. I.

A

Had all right, um we don't have any other discussion items. So does anyone have anything they want to talk about today?.

H

Yeah, I have a question related to the store. I'm not sure if this is a good place to ask, but um I have the discussion that when we restore and then say, for example, they could have existing objects in it and the item that we supposed to restore also you know in already exists in that native space.

H

So, for example, if I have a like a deployment and deployment, let's just say that it have scale equals five, but then, in the backup image I have deployment with the scale equal three um when it behaves like the way. I I think it's experienced.

H

If the object is already there, it's uh the valero seem to skip it. It is just like ignoring it, instead of restore it to the deployment with scale equal three, as is in the backup you just just ignore that uh object um completely.

H

I wonder if that is, um is it by design or is it a bug that we are experiencing or if it is by design? It definitely way we can force it to happen like you know, if we force it to being restored to the um the image in the in the backup image right. So that is my question. If anyone know about in on top of the head or otherwise, I had to dig into the code.

B

B

um So we used to just restore uh the controller owner, so we used to just restore deployments or staple sets, or um things like that. The problem came when especially with rustic, when you have things directly on that pod, um we needed to restore the pot as it was.

B

But you do run into this issue where, if it's already running, uh you can't restore it right now. The advice we've had so far is just to make sure everything is deleted, um but we may have to look at more options or be more intelligent around certain resources right now, there's no force for any restore.

B

We've had requests for that for like arbitrary items that you can say just have valero restore it no matter what, um but I think that goes back to a discussion we had a couple weeks ago about doing patch we'd either have to like patch it or we'd have to delete it and restore it, um which gets tricky, because somebody may have a workload running.

B

So I don't have a good answer for that right now. um Do you have a github issue that we can look at.

H

um I will I will create one, I will get a detective and we can continue this test. Yeah.

C

H

Found that situation, then, when we restore to the original namespace like the application, is running. You know it's right now and it has some problem and we want to revert it back to the situation like five days ago and.

B

We hit that problem that it sort of saw the object.

H

B

C

Use for a recovery.

H

So that is something uh not desirable. That's behavior! Sometimes we want to see what's going on as a temporary, you know solution we just created to. uh We respond to a new namespace right uh that that is something that uh uh non-desirable for us. Okay, I will create a github page.

E

So I had another question: actually an interesting one that came up in uh one of our recent conversations, um and this is uh you know how does valero kind of how would valero work in in a world where customers are using githubs? And maybe it's something that's discussed in the community but in terms of a restore you know, is the approach to kind of in githubs? Are you it to restore?

E

You would go back to that previous commit or if you use valero, I mean that would kind of break git ops in some sense right, because it's restoring something that's not latest but then get uh you know upstream you're pointing to kind of is different from what valero is restoring. So have you guys discussed how valero lives and how coexist, rather with with gitops.

B

So you're right that a restorer wouldn't necessarily match the commit. um The one thing that there's a gap in get ops with your persistent volumes so get out. You almost never put the pv uh data in there. Yes, um so even rolling back on git ops, you're gonna have a little bit of disconnect and valero would have.

B

Valero includes the pv snapshot um so in terms of reconciling that um valero doesn't provide any extra tools. For that. um The closest I could think of is you would get you do a valero backup get and um pull that that um yellow or the json out and convert it to yaml and put it in your get ops repo, um but yeah. Git ops gets tricky with this, and I don't have a good answer here. um I think the the biggest thing is.

E

That definitely makes sense but yeah I was we just. I think it came up in a recent discussion. We are having uh you know where customer asked. Why do I need valero if I'm using github? So do I need valero? If I'm using git ops, and I guess the answer is yes, you do? If you want to back up your data, you know your persistent volumes, you do need it.

B

Yeah and there's there's also kubernetes controllers like valero, is an example of that, where their state is stored in crds that aren't necessarily ones. You would keep in your your um in your repo, for example, uh you wouldn't necessarily in the valero case. You wouldn't necessarily store all the backup camels, um so there there are some.

B

I I think it's actually fairly large number now, but there's there's controllers that store their state in crds that you wouldn't always keep in your repo very similar to like you, wouldn't keep the pv file in your rainbow, but you kind of need it when you're reconstituting stuff.

E

Right, okay, that makes sense yeah thank.

A

You all right, any uh any other questions comments.

B

All right, uh then, I actually have a thing I wanted to toss in on the discussion topics. That's sure it should be the last discussion topic on. Let me pull up the notes again yeah, so I wanted to uh jonas. Could you share the the notes? Real quick, um so we've talked a little bit about moving to github discussions, and um something I wanted to highlight is that we're going to be using this discussion?

B

Q a board, so we've got a link to it there on the notes and I've added it to the header of this document.

B

So this is a place to ask questions and get answers from the community. We invite people to come in and answer here too um we're going to going to use this so that, hopefully we can get questions answered that aren't necessarily bugs or or feature requests it's just kind of. How can I do this thing, or this is a weird behavior? um Is this expected?

B

um We also? um I don't I didn't see if it got merged yet or not, but we're we're adding to um our issues tab. When you create a new issue, it will prompt you to start a discussion um yep, so you can go open that up and ask questions there. So um you'll see a lot of entries there that we took um issues that were mostly questions and move them over.

B

So that's why there's some with with older dates on them yeah. I just wanted to to give people an update on what we were doing with this and um yeah. That's! That's. All I've got.

A

Awesome and uh we're also um archiving our um meeting notes over there and youtube links and so on. So if you head over to the discussions uh tab and you go into the community meetings, you can see the community meetings here. So the notes for the last one and also the link to the youtube recording.

A

So you can always find that there.

A

All right nolan: do you want to do uh community? Shout outs.

B

Did you say kubernetes shout outs or did I miss community? Shout outs.

A

B

Yeah, so um the first one is uh here from madav.

B

um This is a small one, but in a very important one that we had a log message that was at the debug level, um and it would actually happen quite a bit um and people really need to see that information, so madoff changed the uh log level on that message from debug to warning so that it will always show.

B

So. Thank you very much.

B

um The next one is from our own abbey um mccarthy, for uh updating our docs to clarify um how backup storage locations relate to other data. So you can have other data in your valero object bucket. But it's um you in order to do that. You need a prefix uh to put the valero data in so abby clarified that for us and then the last two are related.

B

Helm had some repositories hard coded, but they moved the url, so carlos fixed this in um two pr's we've got one for the helm2 updating and then also updating our pipeline to use the correct location so, basically fixing up some things as helm itself moved um moved his hosting around so yeah. Thank you. Everyone for contributing, I'm also hoping to include the um community q, a answers in this uh as well. So.

A

Sounds great well, thank you. Everyone for for joining today uh with that have a fantastic rest of the week. Everyone and see you all next week, bye folks, bye. Thank you.