VMware Velero Community Meetings, 23 Feb 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Velero Community Meeting / Open Discussion - Feb 23, 2021

Description

Feb 23, 2021
Status Updates - items working towards roadmap

bridget
Multiple credentials support
BSL support in PR 3442
Restic support in draft PR 3489
Plugin support for AWS, Azure, GCP
Currently running tests and need to write docs
Added fix for restic restores when using namespace mapping: PR 3475
Released Azure plugin v1.1.2
carlisia
Addressed code reviews for the DownloadRequest controller - https://github.com/vmware-tanzu/velero/pull/3004
Looking at an issue with a capi cluster
PR reviews
nrb
Reviewing blockers for v1.6.0
Continuing to refine roadmap out to v1.9+
Testing an update to restic v0.12.0
dave
Working on road map
Open review/discussion for Upload Progress monitoring design doc
https://github.com/vmware-tanzu/velero/pull/3416
Zoom - Feb 25, 2021 9:00 AM Pacific - https://VMware.zoom.us/j/94331975442?pwd=V2Rmci9kMjlOMWlHQUZqSkZKQ0ttdz09&from=addon
ashish
Finishing up plumbing work for Velero e2e tests on internal platform.

Discussion Topics

Any rotating topics we want to add here?
What was the outcome/progress from last week’s discussion topics?
nrb: Started a design proposal for a backup manifest data structure
@codegold79 / @brito-rafa: Findings for what happens when a resource is in multiple API groups when processed by a custom plugin. Related issue: https://github.com/vmware-tanzu/velero/issues/3491
Plugin authors should specify fully qualified names in the ResourceSelector type
Action: Document ResourceSelector type
Action: Resources that have multiple views with shared UUIDs - define how to handle these properly on restore
Example: Deployments which move versions from v1.15 - v1.16
Example: customresourcedefinitions.apiextensions.k8s.io v1beta1 - v1 introduced different fields, but the object was the same one
dave: Using an outbound proxy - Any experience from the group on best practices?

Contributor Shoutouts

@MadhavJivrajani: added documentation for how velero handles encryption
@aslafy-z: fix typo in item_hook_handler
@aristosvo: Security configuration Azure Storage Account with AKS

A

Hello, everyone and uh welcome to the valero community meeting, slash open discussion today is february 23rd, 2021 and per usual. uh We get a bunch of status, updates we're going to talk about the roadmap. A little uh well talk about status, updates against the roadmap or rather towards the roadmap. There we go english, not my first language and then discussion, topics and contributor shoutouts.

A

So with that, let's get started. um Bridgette.

B

Hi everyone, um so my focus for this week has been uh trying to wrap up the multiple credentials support. um This is something that we want for one six, and so I have a number of prs open. So the first one is for any support um for backup storage locations.

B

um Then I have another one which builds on that which is specifically for adding support for rustic commands, and then I have three pr's open for each of the plug-ins. um At the moment. I'm currently running tests against those, and I need to write docs and I've verified that all of them are working except for breastsick on azure. So there's some things. I need to check out there um and I'll try and uh get some antenna tests as well.

B

For that um uh there was also um an issue that came up um on the discussion boards around uh rustic restores when using namespace mapping.

B

So we got a fix in for that this week, um which will go in one six and we'll, um as we'll mention on the pr we'll try and get that into one five four as well, um to fix the one five uh release series and then we also did a release for the azure plug-in, um which included a fix um that I think I mentioned last last week that I wanted to get a fix.

B

I want to get a release out for a fix to do with uh um the azure plug-in, not listing everything that was in a bucket so or in a blob storage container. So we've got our release out for that um and that's it for me.

A

Awesome: lots of pr goodness uh great stuff any uh questions, comments for bridgette.

A

All right uh next up, carlesia.

C

Hello, the download request, controller, I think, is in pretty good shape.

C

He has one thumbs up from bridget nola had given a thumbs up, but bridget found the problem with the test, which showed that I was a bit still a bit confused about some of the making some incorrect assumptions and I fixed the test uh set up and that's it and that's it for that, and I am looking at an issue with a cappy cluster.

C

Then I think I'm going to um maybe pause that for for a couple hours and do reviews on those pr's from bridges or at least half of them.

A

What what what's the issue.

C

C

So when when people are making.

C

When people make a backup of one of the cluster the measure, the management cluster is stopped. Working, I'm still trying to find my way around to even like a happy path for valero or in that environment, because that's new to me.

C

So I can try to replicate and take into work, possibly because they are using velar 1.4.

C

A

All right, yeah that that sounds like something that needs to be uh investigated for sure.

D

At least you uh is it just a matter of time that you have not got into it, or do you want another advice or help with this? I.

C

Do I actually, I was thinking about asking you or dave yeah. I can help.

D

C

Okay, yeah, let me know well, we have a cup, I have another meeting, but let's touch base after maybe you can.

D

C

A

No problem awesome any other questions, comments for kalisia.

A

All right, nolan.

E

um I've been trying to focus on getting reviews out for all the blockers on one six. um That link shows.

E

We've actually brought this list down pretty far, at least on the valero uh core repo thanks bridget, for getting that work started across the plugins, um I'm continuing to work with dave to refine the roadmap um from 1.9 out. We may need to adjust stuff and and we're.

E

um I don't think we're quite happy with it. Where everything is at the moment, um some some things may need to be moved around, but we were hoping to get that out this week, but that did not happen um and also uh something I've been taking up that I'd like to get into one one six, um but given time constraints, I may not um it's testing rustic 0 12 uh on valero. um This uh we're a few versions behind on rustic now and rustic development has picked up significantly.

E

um So uh we want to test that out and also there are some dependencies they have uh in their go code. That could be a security risk, so we're trying to get that updated to basically get rid of those dependencies.

A

That sounds like a good idea yep, um so you, you said: rustic uh development and contributions have picked up. um That's great to hear, because I I know we were looking into this last year. uh We saw development that slowed down uh quite remarkably, so um does that change any uh anything with regards to uh making rustic a uh kind of a modular part of alera.

E

No, I don't think so, because there's there's lots of different engines that people want to use, um and so we want people to be able to swap those out. um There there's certainly been traditional backup vendors that uh want to use bolero, but they want to put their own engine in because they've, you know, invest a lot of time and effort in there. So um we're looking at rustic incopia to be candidates.

E

um I think dave's played with copies um and I think ashish did too, but I think our our general direction will be moving all of those types of things into a data movement layer and hopefully that could work in sync with csi snapshotting.

A

ah That sounds good cool. Thank you.

A

Any other questions comments for nolan.

A

All right uh dave.

F

Yeah so let's see so as nolan mentioned, we've.

B

F

Working on the roadmap, that's pretty much that so as part of our movement towards getting uh data movement into valero. uh I want to get upload progress monitoring into one seven, and so I have a design doc for that. There's a pull request for that and I have a meeting scheduled for the 25th, which would be, I believe, uh thursday at nine am and uh so everybody's. Welcome to that. The zoom link is here in the uh hack, md and I'll, send it out on the mailing list as well, and that's it for me.

A

Awesome, uh can you send out an uh invite, so people have it in their calendars if they want to join yeah. Okay, thank you.

A

Nice um any questions, comments for dave.

A

All right, ashish.

D

uh For me, I am right now looking into plumbing our current t2e tests that we have into our gm web so that we are running those tests that are regular cadence. So I am doing that.

D

I will also be spending some time today, working with carlesia on that issue. She was talking about.

D

Pretty much everything.

A

Sounds good any uh questions, comments for ashish.

A

All right um moving into the discussion topics, uh so I added two questions there, and these are questions for for the maintainer teams, uh the maintainer team. Rather, you are one team uh and the the community here. So uh I know previously. We we had some um uh consistent topics uh in the discussion topics. We we talked about um larger breaking changes.

A

Things like that, and I was wondering if there's anything like that, that we want to add in here that we want to mention um not sure if there is, but I want to do a call out here and see if there is something that we should add in here, that we should continuously talk about uh during the community meetings.

E

So we we did have the api changes proposed and and we've kind of that's kind of taken, a back seat to some of the other, uh the other priorities. But I think that would be a good one to add here. um Road map discussions when we can get a public would be good too. I know our status updates are supposed to be map focused, but we can.

E

uh I think, jonas you and I had talked about- do a big look at the road map at the beginning of the month, and then we kind of update towards that.

A

Yep sounds good, so um roadmap, beginning of every month, first first meeting of every month. Well we'll take a look at the roadmap and see where we're heading uh for for the current release and also, if there's any any work being done on things that will affect other future releases. So breaking changes I think, would be great to bring up there and then remind people of them as well.

A

Okay, um another any sorry before I go ahead, any other comments there anything else. Anyone would like to add as a rotating or repeating topic.

A

All right, so um we do have a lot of uh great discussion topics uh uh throughout the meetings here and I think it's worthwhile to kind of look back at the the previous week's discussion topic as well and see if there's any any outcomes that we should bring up uh on the meetings, because we've been having these discussions and then the rest of it happens in in github or on slack in code is merged and we're done um so.

A

We we rarely bring up the results of these unless it's a it's a longer discussion, so I just wanted to make sure we didn't actually have any discussion topics last week, but going forward. I want to make sure that we just take a look at the previous week's discussion topics and see, if there's any old business, that we need to um revisit and kind of uh give a little status report on.

A

What's that sound good, all right cool, uh all right, uh so we're gonna. We've got three discussion topics here. If anyone wants to add any other discussion topics, I definitely think we have time for that today. So uh please do so in the hacking d here. First up we have nolan.

E

Yeah, so this one, I'm gonna keep real short, because this is not like an urgent thing. um I I envision this work not being done until 1 7 and even then it's like a building block, um so this design document is aiming to provide extra metadata for valero to figure out what's in the backup and using that to do a little bit more sophisticated logic on what we should restore or what we should back up.

E

um I listed in this uh design doc some of the the things I think this will enable, so I think it will enable better cross uh or better parallelization of backups. So if we can use this manifest data structure to say, hey, what's the overlap between objects on these two backups?

E

If there's none, we can just go ahead and do both. At the same time, um it will enable dry runs by building up the manifest structure uh and then, when you actually do a backup, you just use the manifest structure and it will, in the future, enable stuff like graph based logic for restores so right now, valero sorts all of the kubernetes kinds or api groups in a specific order.

E

At the front, and then everything below after that is alphabetical, this would allow us to say what objects does. This object need uh to be restored and we could restore those first and then uh kind of work through that way, instead of just one at a time.

E

So I'd encourage people to take a look at this. This proposal is just for the data structure, and so we can like start populating it and bump the backup version, backup format, version to v 1.2 and then in future releases we can start actually using the stuff in this data structure, to make this decisions.

D

Where does this fall under the roadmap.

E

So I think this would fall under a dependency for um all of the uh concurrency stuff, the dry run and um getting support for cappy, making support better for cappy, because cappy specifically was something that kind of triggered my thoughts here.

E

um So I I view it as a dependency for those three features, and maybe more so I'm thinking this is a getting. The data structure would be a one seven or a one. Eight.

E

But I also, I also don't wanna implement it until we like have a good discussion around it, because I, given how many things I want to, I think could use it. I would like to get the data structure right.

G

I have a question, so will this enable to uh run concurrent backups in valero the data structure? Will let us have the information.

E

We need to do that, um but this design itself does not bring concurrent backups, but if we can build this manifest of each of each backup and then say: okay, what what objects are in each do they overlap if they're, if they have any overlap, then we'll do it serially. If they have no overlap, then we could do uh concurrent.

E

E

This is- and this is all data we have now in the the backup tarball, but it's not in a convenient and usable format. We'd have to like read in every single json file. um So this basically lets us collapse. A lot of the the identifying information about objects, so name namespace, api group group version labels and annotations so that that's all we should need to identify an object. We don't need the whole.

E

A

um I see you're talking about uh dry runs here uh with this. Enable dry runs.

E

Yes, so my thought for dry runs is, we would simply build the manifest and return it. So we would say: here's all the things I matched, maybe a super long list, but by returning that we basically um dry runs fall out of it.

E

The thing I uh I have a question on the design still is whether or not we want to enable something like I run, a dry run and then okay, just back up that dry run, but I think that's that can be answered in the implementation phase for for the dry run, um I think just getting enhancing the valero backup code to start building this manifest to step one- um and this is like inspired by uh what some of this is inspired by what pranav uh did with the restore procedure.

E

Historic progress- um and this is the thing that I've been thinking about for a long time with valero is how do we make it graph based instead of just doing things one by one, uh because that's bugged me for a long time.

F

And actually, thinking about this, something we could consider here is like the output of the dry run would be like a manifest or a json.

F

We could also have an option to do a backup based on a json, and then we can cut down on the number of filters and so forth that we're building into valero and let people build their own filters around the manifest and the same on.

E

Restore yeah I I could see that that if we do a dry run, we just return the manifest to them uh and they could provide that as input. And if you want to do client-side filtering on that, you could.

H

All right, this is raphael. um Are you envisioning having a new crd for that, like a backup, dry run.

E

So no um this will be exclusively for valero's internal use. So I I envision this as a file that just gets put in the object store. um We have. um We actually have a file there right now that you can get. If you do valero backup describe details that has like a a list of everything in there, but this in time would replace that file um and that file is built kind of differently than what I envision here. It only has.

E

It doesn't have enough identifying information to serve as what we want. It doesn't have labels, it doesn't have owner references things like that which we would need to build a graph.

E

E

And I envision on talking about this more like I, I just want to bring this to people's attention. um I I'm sure we'll have some one-offs on this and talk about it, some more, um but I just wanted to let folks know this is like this is early. Please come at it ask questions poke holes.

E

Some of the naming I'm not super happy with, but uh yeah, certainly looking for feedback, because I I view this as being an enabler for a lot of other stuff that we want to do for valero.

A

That's really cool any other questions. Comments.

A

All right, thank you. Nolan uh moving on to uh frankie and rafael.

I

All right- that's cool you've, memorized, my uh github user handle, of course very flattered. So for this one, it's very very easy to get into the weeds, and I don't really want to do that in a community discussion um but to understand what's happening.

I

Imagine two different scenarios, um one where you have uh one resource, but it fits in multiple groups. All right. Let me do a review on what is a resource and what is a group before continuing, because I always get this confused. I think the example I like to use is deployments.

I

You know what we know that deployment singular is a kind it is in the uh it is a deployments resource right and then the group that deployments is in is called apps all right. Well, when, if deployments was in two different groups, instead of just apps when it fills in something else. Well you um this can have. This actually happened to us with ingresses ingresses is the resource.

I

Ingress is the kind, and then the group that ingress is in currently pre 1.22 is extensions. That's one of the extensions is a group. It's also in the group called networking.kates.I, so hopefully everyone's following, we got one resource, that's in two groups. So how do you deal with that?

I

And by the way this can happen with crds too, as soon as people start creating their own resources, they could name it whatever they want when, if so, you know raphael, he has this. He created a resource called rock bands. It's in the example.music, sorry music.example.io!

I

Well, I liked raw file's idea. So I have my own domain. That's called frankie.io and I call- and I create a um a resource called rock bands as well same thing and then I say mine is but mine's different, mine's, frank rockbands.frankie.io.

I

uh When so you know what how does valero deal with uh this resource being in multiple groups and the reason why we came across this issue is because you know here thanks jonas, for bringing up this bug, which I wonder, if it's a bug now it actually, we have a workaround that turns out. It works pretty well, um but maybe the wording should be changed, but in the plug-in, when you make a plus custom plug-in, you can create a section called apply suit. But before I continue, is there any questions or raphael?

I

Should I have mentioned something else.

F

um I do have a question and uh it sounds like there's two different cases here. So one is it's actually the same object really truly, but it's been moved between groups because of like one group has been deprecated.

I

F

I

F

That's case a that's.

I

F

Right and then case b, you have two uh resources that have the same name: for example rock band, but they're not the same object. They're.

I

Completely different.

F

Yeah, so those are two pretty different cases.

I

Exactly and the only thing they have in common is that resource and kind names.

F

Yes, but the group I mean the group, is there specifically to distinguish the two differences, so the.

I

F

Phase, do we need to worry about that.

I

I I we haven't really played with the second case and and that's one, and that was one one of the reasons why we brought up in discussion is to see if, uh if valero, if the valero team might consider this a big problem, maybe it's not. I'm not priority-wise, we're not quite sure where it fits.

J

How do you tell the difference between that case and the previous case.

I

H

Yeah, let me let me um jump in um frank. I think explanation was good. um The the case a you, have the same group, the same object which happens to belong to different groups. It's the same uuid object and the way that valero uh took care of it. Let's use the ingress, because ingress is a native object of kubernetes, so kubernetes is in charge how to deprecate api groups on that object or not.

H

So what we do see in the valero backup two backups of the same object in different uh folders right, because the first folder we have is the resource and the api group.

H

So for our case our plug-in case, our workaround or our solution on the plugin instead of just say, ingress. We specify ingress dot, extensions and ingress dot network dot, kubernetes rio, so that we can manage um the logic. So the plugin is gonna, be executed twice for the same object. We're gonna handle that, so this is kind of under control.

H

I guess because it's an object that belongs to is the same object belonging to two api groups. The restore gonna kick in twice, it's gonna see the object existed, it was restored on the first pass on the second pass. You're gonna ignore.

E

H

Good on that one, I I.

E

Think go ahead. Well, so what I was gonna suggest when jonas had the code up was fully qualifying the name. So, like you just said raphael you you specify the name the uh you know all the way up through that I think, but I I need to find examples. I think the valero resource uh selector is smart enough, that you can list multiple uh groups, um but I think in your case you really only want to do that if the objects are equivalent.

E

So if I, if I have this ingress and it's in two groups, I want to make sure it's ignored, um but what about stuff? That's not in both groups right like so what? If there's an ingress and only one should we also ignore it.

F

Can I ask a question here so yeah? I think what I'm hearing is. We have an object that is identified by a uuid, correct yeah that object in kubernetes in the api server actually has two resources that point to the same uu.

D

F

Yes, so it's not as though so from the kubernetes point of view. There's two objects.

H

In perspective, yes, so it's very similar what the work we've done with the web hooks with the multi. Multiple api groups is the same object, but has different views of the same object.

F

Yeah so from the kubernetes point of view, there's two resources. It's like if you made, I don't know if you could even do this, but probably you construct a you. Can you can do things like construct? Two pvs that have the same volume handle um that'll, probably do bad things, but I don't think anything actually checks to make sure you can't do that and that those would actually be, and you could, you could probably can construct two ingresses that are not different groups but that have the same uuid.

F

The controller might uh prevent you, they're gonna,.

H

Bump you, if you try to create so you can create an ingress dot extension and one name space to a name. Then you try to create uh ingress dot. Networking on the same name, space with the same name, gonna bump.

F

H

But forget about the different.

F

Groups make it in the same group. I think you could construct it where you have two objects, pointing at the same.

E

Uuid you cannot use, you cannot submit an object to the kubernetes server with a uuid. It assigns those except like pvs, with static stack provision right.

E

No, it will it'll make the uuid the.

F

Google uid the volume handle, though right. Oh.

E

Yeah yeah the volume handle is different from the kubernetes.

F

What so do we get two objects with the same, so is the uuid like the kubernetes.

H

So the the view, the view is almost like- you have different api group versions right, remember that that you can create kind of web hooks and this behind the scenes web hooks about ingress, particularly.

F

H

Of kubernetes and.

F

I think I'm trying to understand, because I think nolan is talking about a api server uuid, which is unique within the api server, and my thinking would be that that would be unique for each object. That's created unless there's some magic going on in there.

E

There's magic going on so uh what rafael's describing is. This is a kubernetes object that has the same. It's one uuid, it's one object, but it can be presented. You can query it through both api endpoints. Exactly so.

F

Yaml is going to be the same.

E

No not necessarily actually.

F

E

F

E

F

Owns this actual object, that's returned, so it's not simply an fcd instrument.

E

H

Related apis. So all this logic back and forth at extensions and networking is handled by the kubernetes api.

E

There's compatibility layers.

H

E

Compatibility layers in the api server.

J

It could also be an aggregated api server.

E

Yep, an aggregated api server, which is not the kubernetes api server right, so it could.

J

Be something that outside of that scope, I guess, is web just wanted to throw it out there.

H

But one thing that I haven't seen yet in this particular case: what is the preferred group, if it's an extension or networking? Actually, when you do cube city I'll get ingresses without specifying the api group, it returns extension so kind of give a hint extension. Is it for the lack of a better term preferred group name.

F

Okay, um so going back to our original thing, we had two cases we'd identified. One was that there's like a deprecated thing and uh yes, something it's.

H

A great segue yeah.

F

Two different: you know people just happen to name their objects. The.

H

F

H

I share my screen very quick.

F

Yeah is the second case possible.

H

Yeah, so we just tested right now. Let me share my screen, um so we have these rock bands. Okay, so um I created, I had the rock bands dot music example: dot io I tweaked and I create rockbands.cbgb.io, I'm a big fan of punk rock, so cbgb is a temple used to be, um and then now I have if I run cubectl api resources and I grab for rock, I got two crds okay.

H

They they are completely different, different schema, but the kind happens to be rock bands, but they have one cbgb, others music.exempl what happened when, after I created my second rock band, you find.

F

What uuid, on those two objects.

H

Oh they're different, oh actually,.

F

They haven't even.

H

They're completely, I I should hope so I think we've got two really different cases right. I thought we were done on the first case.

F

Well, the second case doesn't seem like a case really I mean there's going to be name collisions. That's the whole point of group and kind.

H

Yes, yes, but the the the raise awareness of the point of this conversation after the name collision. When I just do get rock bands, I don't get anything else right. I was getting hey, get me rock bands. I got my beetles here, but then after I create my other rock band rock bands by itself, my my command line doesn't find beetles anymore. So that's the it's awareness conversation here and and for the community always use the fully qualified name of your resources.

E

Yeah, that's kind of what I'm getting out of this because because this will happen sometimes on. I can't remember to fix this, but there was a case where, if you did qctl get backup like for valero, uh it would say: oh I don't know what that is, uh but then you say: keep cl get galera or backup.galero.io.

E

Then it gives you everything. So this is. I think this boils down to uh some of the stuff that the api, server or aggregated api server does behind the scenes. Crds can register to say, hey. My short name is rock band, but if two of them do it cube, ctl just says: hey, I don't know what to talk about uh which I kind of expect um built-ins like ingresses, that's not really built in, but it's close. um I think they get slightly special rules um but they're trying to follow more rules so yeah.

E

In this case, valero observed the exact same object through two different filters. Essentially um it saw an ingress.extension in an ingress.networking.

E

Kubernetes can store different fields for those um I'm not super familiar with that, but in valero we basically manifested it as two completely different json files and then, when we were going for matching, uh we had the same issue that cube ctl said where, if you just gave me ingress, okay, I'm gonna get everything that says ingress.

E

Is that summary helpful or more confusing.

H

I think was was helpful.

H

It was more like a raise awareness to community and you know.

F

Well, the second case so like on restore. We have two objects with the same uuid so, but we can't specify the uuid so when we create object, a that, presumably like if we created in the uh deprecated space that presumably also creates the same object or the same object suddenly appears in the new space.

F

And then we create the object in the new space, but that's actually a different object because we didn't specify uuid. So now we get two in both namespaces.

E

Oh yeah you're right on restore valero rips, the uu id out. I.

F

Mean you can't you just. We just said that we can't specify it right.

E

Right we, when we do rip it out because yeah so uh this grab. This scrap api may help us.

F

E

F

We'd have to sort it and find in the backup items with the same uuid and like pick one to restore, would be nice.

E

Right so yeah we could use some of the logic that frankie and rafael have developed for what's the right version, what's the right group for this version of kubernetes um like if you're restoring the old thing, the maybe the deprecated ones preferred if you go to a newer version of kubernetes, uh the new form is preferred.

E

If you have both.

I

Yeah that sounds good um now. My dilemma is: should I close this bug um bug ticket because it's actually working pretty well for us once we fully use a fully qualified name. um It just wasn't obvious when you reading the documentation or even reading the, if jonas, if you could show the that that uh bug ticket again, if you look at the that applies to, if you scroll up there uh applies to the func function uh applies to you know, we put the word.

I

First of all, we put ingress in there, which is a kind, but it works, it works included. Resources seems to indicate you should just make a string, a slice of resource strings, but really it could be fully qualified names and that's what will fix it.

E

Sorry go ahead. Oh.

I

No, that was it. That was that that was it.

E

I I think, at least for the immediate term. We can better document the resource selector um because, like I'm looking at included resources here and yeah, it's a slice of strings, so you could make the same. Plugin apply to both fully qualified names.

E

Since it's a slice. I I looked really quickly in our code and we don't do that too many places. Oh.

I

We tested it out, you can absolutely have two two strings there and that's what ended up working for us is. We had the first string, be ingress, dot, extend extensions and then the second one was actually it was networking, ingresses.networking.com, kate style so having both it worked for us. That was the work around.

B

I

It wasn't clear to us in the beginning that that was allowed when you, you know, under the term, included resources, so.

E

I

Everyone know just use the fully qualified name if, if you're having problems.

E

Sounds like a good developer, documentation thing, because I don't think I don't think resource selector. uh That type is very well documented.

F

Yeah, we might even want to just say you know you got to use the fully qualified name.

H

F

Use distributed apr.

H

Just to change on the documentation and there we go right.

I

Yeah, maybe the fix, maybe the fix for this bug is just update. Documentation.

F

Yeah, I think, ultimately, we're going to need to be a little. It's really easy to write yourself, a backup item, action, we're starting to recommend these and try to make this easier for people and saying yeah. You know just write one for yours, if you, if you make one and it works, but then somebody else likes rock bands and all of a sudden you're starting having problems in the field. That's really not fun, right, so yeah.

F

I think we should document it and then also put in like change this to you know fully qualified names before there's too many existing plugins out there that rely on behavior.

H

F

Fully qualified names.

H

Yeah, frankly, rock is dead, so hip hop. I think that's gonna be a lot.

A

Nerd core all the way.

I

And by the way, what happens here when you, when we specify ingress for applies so it does not address both groups? It only just it chooses one, the extensions and moves on and- and it mattered to us, because we asked that the skip restorer was true for ingresses, but yet it kept being restored. Well, the ingresses. extensions was being skipped as per instructions, but the networking dot. Sorry, the ingress is that networking that case.io was not being skipped and getting restored. That's where our problem, that's where we encountered our problem.

I

So that's it forever. For me in this discussion,.

A

Awesome. Thank you. Thank you for bringing this up um so the end result of this is to improve the documentation. Yes,.

I

A

I

I'll write the work around in this ticket and then in this issue, and then uh I'll I'll go ahead and update I'll go ahead and write documentation for it. In order to close this issue,.

H

A little comment.

I

H

Example, I don't know you know, maybe regular documentation and a couple comments over there on the template. Yeah. Actually, the shared.

F

Uuid actually has me more worried than this one.

E

Yeah we can talk about that some more separately, I think yeah. I don't.

F

Want to lose it.

E

Yeah yeah, it is definitely a thing. Kubernetes supports and there's multiple use cases for it, but yeah.

A

All right awesome, um thank you, frankie for writing out document documentation for this.

I

A

uh All right, uh we have one more discussion topic uh for dave.

F

Yeah, so um we've been getting some requests of customers who are setting up like a full-blown man-in-the-middle proxy mitm proxy on the way for everything going out of the network and they're having some issues with setting up the certificates. They found like a bug in the valero code, where we're not using the certificate the specified certificate everywhere to override things. I just wanted to see if anybody had any experience with using outbound proxies in their environment, you know what they felt were best practices.

F

um You know it's just kind of looking for some if anybody, if anybody's doing anything with it, if not, you know muddle through.

F

So it sounds like nobody's.

H

Crazy enough to do that. Well, I have a lot of experience for inbound trap at proxy, but not outbound.

F

Yeah so they're like routing everything through this proxy and then like the http certificates. Of course, don't work because it is literally a man in the middle attack, and so they want to override the certificate um and I'm just trying to figure out how common that's going to be for people and what you know whether we should support like in this case.

F

One possibility is: I'm I'm probably going to point them in the direction of using the s3 urls, rather than trying to use the domain name setup, because we test the s3 url uh certificate override with minio.

F

And if this is going to be a really uncommon case, then I don't want us to add, like another test case, that we can't test or we'd have to build environments that have this outbound man in the middle proxy to test overriding the certificate everywhere, because it has to work not just in the object store plug-in. It has to work in the uh what you might call it and rustic. I might even have to work in the volume snapshot or depending on what they've managed to do.

J

um So when you say that they're pushing all the traffic to the outbound proxy, does that mean they are overwriting the urls to go through that proxy? Like.

F

No they're intercepting everything on the way out, um so I'm not sure exactly how it works, but the idea is that all network traffic will go through this thing.

J

So it's not that they're using the yeah! So it's not that they're using the http and https proxy variables and pushing them are using those that you're saying that they actually are gripping all that traffic and are capturing all that traffic and then sending it on and okay, interesting.

F

Yeah, it sounds like a bad idea, but you know customers right.

J

huh I figure you should be more explicit than that, but okay.

F

Yeah, you know, that's, that's that's why I'm looking to see if anybody has any experience with you know real world like hey, you know yeah. We see this all the time. Oh nobody ever does that because to me is like it's like a. I don't think, that's a good idea, but I can't say from actual experience you know I mean if all the.

J

Fortune 500.

F

Companies do it we're gonna have to deal with it.

J

Yeah, like I know, for openshift at least we have a global proxy thing that you're able to set up, and that will uh you could also set up like an ssl cert um inside that global proxy. That will allow everything to go talk to that global proxy with that ssl cert to verify like so it could be a self-sensor it could be. You know what I mean, um and you could add that to the to the cluster to to verify these things, but actually mending man in the middling yourself.

J

I have no idea how to get around that. I don't know if we, I don't know if we have anybody doing that. So.

F

Yeah, I think what they're doing is they're rewriting all the search, so they have their own private certificate authority, essentially where they would write a cert for aws.amazon.com. And yes, I said this is the real cert. It's like that just doesn't seem like a good idea, but.

E

Yeah, like that almost to me sounds like I mean that's a that is a weird networking issue that I don't really know a whole lot about, uh but like to me it sounds like they should start using custom certs, uh but I don't know how the the translation would work.

F

Well, they did do like uh so they set the cert, they set the custom cert or the the certificate of authority, but there's a fir there's a part in the code right now in the object store plug-in where it goes to like automagically, retrieve the region and we're not passing the search surgeon.

F

And it's like okay, that you know we could probably fix that, but where it's going to be whack-a-mole with you know things that don't work. So that's what I'm just trying to figure out. You know how much whack-a-mole should we play.

F

Okay, so thanks sean- and you know anybody else- has any input- would be very useful. Thanks.

A

Awesome um with that we're uh we're out of discussion topics for today, uh so we got one thing left to do and that is the contributor shout outs, nolan.

E

Hey so um first we got a really helpful um contrib contribution from madav, uh where this um specifies kind of what encryption is present within valero in the backups. um So it's just clarifying a question: we've gotten semi often uh so this just gives us a a good place to point to um very short version: valero doesn't encrypt stuff itself, it relies on https transfer and uh your object store to do encryption at rest. So it's just clarifying the current state of things um as uh asla fi. I'm sorry, I'm zeal! uh I'm sorry!

E

I'm butchering this. um I don't think I saw this change so.

E

A small comment uh fix. Thank you very much.

E

um And then um we have aristotle. uh That is updating. um I believe this is documentation. It's been a little while, since I've looked at this uh yeah, it's updating some security documentation uh for azure storage. So thank you very much. uh We appreciate that contribution.

A

Awesome, and that is it for for this week. um Thank you. Everyone for for joining the uh community call today um have a fantastic week, the rest of the week. Everyone and uh see you all next week. Bye thanks, see ya.