VMware Contour Community Meetings, 8 Sep 2020

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Contour Community Meeting - Sept 8, 2020

Description

September 8, 2020

News
Contour 1.8.1 was released with a performance fix for status updates

What have we been working on?
[stevesloka] go-control-plane xDS Server
InFlight PR
Config File
[stevekriss] internal/dag refactoring

Discussion
[Chad Cravens] External Auth Providers, how do they work?
https://github.com/projectcontour/contour/blob/main/design/external-authorization-design.md
https://www.envoyproxy.io/docs/envoy/v1.15.0/intro/arch_overview/security/ext_authz_filter#arch-overview-ext-authz
https://github.com/projectcontour/contour-authserver

A

Welcome everyone to the contour community meeting today is september, 8th 2020.. The time is 101 pm on the east coast. um Welcome everyone and we're gonna start off with some news.

B

Yeah, so um we released contour 1.8.1 uh last week. I think we talked about at the last office hours real quick, um but this is the um just for future. Folks might be watching this, so matt moore works on k, native and as part of the k native deployment. You can use contour as the ingress controller to route traffic around in all the candidate of inner workings.

B

So matt was doing some some testing of upgrading the version of contour that ships in k native from an old one like 1.3 or something I believe to to latest, and there were some issues he found with performance where the performance got a lot worse in later versions, since found out that there was sort of an issue that we could solve in how those status updates.

B

So contour is in charge of writing status out to different objects, so things like http proxy objects as well as ingress objects, so it writes status for those updating you know, is it valid? Is it invalid all that sort of stuff? There was a an issue, how we were doing that as updates. Basically, the api server was rate limiting us, um the updates would stack and it caused um envoy to get slow. So that's in that was the reason of the 1.8.1 real quick.

B

um So so, if you're up, if you're running contour, this is a great recommendation. Now this only really affected large large clusters. I think the smaller ones didn't really notice it as much just because the problem wasn't as apparent but yeah, so it's cool. So thanks shout out to matt for his work and nick for doing the change really quickly. Last week,.

A

Awesome. Thank you.

A

um Next up, we have. What have we been working on.

B

Yeah, so I had the same thing I feel like I've been talking about for weeks now, but it's actually almost done so um there's one more pr there in the list that I think jonas is pulling up now to finish, which I think we'll land soon there's a couple little things um I need to fix up for comments, uh but basically, what happens now is there's if you actually go to the second link there.

B

Jonas, what we have now in the config file there there's this xds server type option, and the idea here is that um contour ships with its own built-in xds server and the idea was that we wanted to have a way that you could switch. This go control plane type, but not actually have to do it all at once right, so maybe you weren't ready to switch, um and we want to you know test it some more! So now you can toggle between the two. So contour is the default one.

B

I think you can choose envoy, which does the envoy xds type so now now that that's in there once this pr lands that we had on the other tab um that will have full support now for um swapping out the the contour xds server with the upstream envoy go control plane.

B

um So in theory you know no one's gonna really should notice a difference in terms of how control works or performs, but once we have this go control point now we'll be able to easily switch to like ads and those sort of things which are which are kind of neat.

B

I think our other plan was to start the switch now to the xds v3 types and start implementing those, so that will come here. um I think next after we get this in.

B

So that was all so yeah just let you know that once once this is landed, this nothing will change again unless you buy into it. I guess: what's the big thing I want to get across.

B

And then we'll watch that I think eventually we'll switch we'll have it just you know we'll take out like we did with ingress route, we'll deprecate the built-in xes server and then switch to switch over to it fully.

B

uh That's what those two things yeah.

A

Sounds good any questions comments for steve.

C

No, I'm excited about that. That's cool.

D

Awesome uh next up steve chris yeah, not too much of an update, but I talked last week about uh some of the refactoring work that I took over in the dag package around how we build the dag, which is contours kind of internal representation, of of the various ingress apis and and all the related objects.

D

So I'm continuing with that a little bit to continue to to sort of improve the internal interfaces around this and really the goal here is to make it easier for contoured developers in the future to add new code that that adds objects to the dag. So as we're looking at implementing, you know additional apis, so things like service apis or ingress v2.

D

um You know. If this is done. Well, then it means that that adding those things will be a little bit simpler. So that's that's kind of what I've been looking at for the last few days.

A

Sounds good any questions comments for steve.

B

Yeah folks are interested in that. I know that the last office hours we went through a big walkthrough of how contour uh takes data from kubernetes and then processes through to get to envoy. So that's a good uh once that gets posted up on youtube. That'll, be a good thing to watch anyone's interested in understanding a little bit more about what steve's working on um so yeah, and that would be nice that'd be nice to break those things apart.

B

I'm gonna be able to um then we're gonna hit full charging the service apis work as well, and those are starting to get finalized, so it'd be great to have that work um test them out in a real ingress controller.

A

Speaking of the uh the office hours, um there were a lot of people that joined in on thursday last week. So thank you. Everyone for for joining in uh it was fun to see a full house and um you, um you talked about a bunch of different things uh in there and you covered a lot of ground in in two hours. I think I believe it is on youtube already, so it should be viewable for those that didn't watch it if it isn't it'll be up later.

B

Today, where does that go? What is there a link somewhere on the community site? I guess I should. I should know this.

A

It should be linked there. Yes,.

B

uh Youtube playlist, okay august 20th is the latest one huh all right, listen to playlist.

A

I'll fix that yeah, I thought I uploaded everything on friday, but apparently well thursday, so I'll get that done. Yeah no worries cool thanks uh all right. Next up we have a discussion topic. Chad,.

C

Yay, okay, yeah, oh thanks, jonas, and also real quick on the youtube. um I do remember that there was uh a conversation we had, probably two or three weeks ago regarding setting up a dev environment. um So I I haven't, checked the youtube: if it's there or not yet, but I just wanted to uh also throw that out there too, that I think that would probably be helpful. um Hopefully, for me.

E

C

Other people that are trying to do uh contour, dev on how to kind of get that local environment set up so um but yeah. So thanks jonas um cool yeah, so um you know so I've been looking a little bit into kind of auth providers and how they work with contour and kubernetes.

C

You know so our last conversation from my perspective, anyways really cleared up the purpose of contour in terms of kind of it, acting as like a translation layer between kubernetes objects and events and envoy configurations, essentially right, and I think overall, that's the theme that I got out of kind of contour being that middleman right.

C

That translates kubernetes events to um to envoy uh configurations and um one of the things you know, obviously like the purpose of contour for kubernetes cluster is so important, right, handling, ingress and routing, and one of the things that I'm working on in my own kind of cluster is: I want to be able to manage authentication and authorization for pretty much everything that goes into my cluster right. So if there's a route, that's being directed and contours, essentially you know configured that route um yeah.

C

How does how does this external auth stuff kind of work- and I was just wondering I thought it might be a good- you know, conversation topic, so you know just yeah and and and how does that work? You know with kubernetes and envoy, and you know how far can we take this essentially so.

B

Yeah sure so um there's a couple things I guess so um a lot of the services in envoy they get their their network filters that you add onto the envoy configuration and a lot of times they. What you do is you configure some external service, so in the case of external auth you would have stand up and auth service and that service would talk to envoy over grpc.

B

So one thing we have to configure- and this is the design that jonas has up here- that james worked on of how we're actually going to implement this external auth in contour but anyway, but the high level bit. Is that what you first do? Is you define this thing called an extension service and what this is?

B

This will be a crd which will define what that service looks like that you're going to connect to right so you're, going to configure envoy to say hey when a request comes in, go reach out to this external service over grpc and ask it you know basically thumbs up or thumbs down. Should this request be authorized or not, so what happens is envo will receive the request once this is all configured it'll pass it off to that external service and that external service will do whatever magic it needs to do you know it?

B

Could it could reach out to an oidc provider? It could do an internal check of a database whatever it needs to do. It can do um and then it'll return a result back to envoy and then we'll respond back to the client with you know, like you know, you know: 200, it's okay, keep going or, or you know, four or three redirect whatever whatever the case it may be um so ideas and contours, and this the same thing works for like um the rate limiting service.

B

So I think, a year and a half ago I wrote a bunch of rate limiting work and it kind of fell off the wagon, but um we're gonna that works the same way right. So you stand up in external service which implements how rate limiting should work, and then a request comes in onward. Asks that service you know. Should this request go through or not, and then that happens um so in that design dock that that we had there in the notes um james walks through a couple pieces.

B

So the first piece is that extension service crd and that's going to describe what that external service looks like so the service that onboard needs to connect to. We need that for a couple reasons.

B

One is we have to bring that we want to bring that service into the envoy cluster, so we can then do metrics on it and do all the things that envoy can do to watch that service, and then we want to be able to have an extension point to implement other bits, so things like rate limiting um and there's like some logging things that we could do as well, that this extension service can be reused for um so that's the high level bit. um If that makes sense, I know there's another link.

B

We could send you that has the um james has started a um an auth uh repo. um Where is it first of all? Does that make sense, and I ran through that pretty quick, but.

C

Yeah yeah, so I think it does maybe a couple questions I could. I could just run real, quick, so, okay, so when we're creating this extension service, essentially it's going to be a pod, that's running a grpc server right and we register this extension service with contour in our co.

C

In our manifest right, like in in in some sort of manifest, we tell contour, hey, register this external service and I guess we give it a service name and then contour is going to tell envoy hey connect to this grpc service as an external provider and then once contour's configured envoy. To do that, then contour is no longer kind of in the picture right. The request comes into envoy. The grpc goes to that external service.

C

We respond with a yes, no to envoy and then how to essentially respond with, like a you know like a 301 redirect or something like that right is. That is that the overall yeah.

B

Yeah, that's that's about it! Yeah, okay,.

C

Cool go ahead. Oh yeah is.

B

This a feature.

C

Complete thing, or is this something that's kind of been more in in the process or there of.

B

It's definitely in flight right now, so so there's some bits here so um yeah out here in contour. There's a couple pr's that james has um so the first one is implementing the support for this external service through some of the internal bits, this one's just about ready to go. So this one will allow the the thing that steve was talking about that that dag processor bit.

B

um We need a way to include that those external services now into contour to let them like contour, know about how you know what they are, because right now, contra just watches for kubernetes services, um and this new type is going to. Let us watch for this new crd type once that's in. I believe james has another one here, which adds uh this implements the bits into hdb proxy.

B

So this pr here builds on the last couple, so this is going to actually have the bits to actually configure your proxy to actually look, for, you know to add the the off pieces to your ingress object right to your hp proxy object to actually map it and say you know now that I have all so once you do all the plumbing to wire up all of these services and stuff, and then you have to tell you know which route do you want to be have auth on and how it should talk to what service that sort of thing.

C

Yep yep, okay, so that crd is actually defined by contour. Yes, yes, right, okay, and then we just deploy that crd right and then is there like a container. So is there like a pod that gets attached or like a service that gets attached to that crd?

C

So do we have to deploy? Do we have to deploy a deployment? That is that grpc service, that you know talks it off provider whatever and then um and then kind of like register that service with the crd is that kind of how that works.

B

Yeah you're gonna have to yeah you're gonna have to deploy um some sort of service. That's gonna be that external bit, so here out in project contour. I think james has this uh auth server here. So this is a generic auth server that james wrote that you could stand up to be that grpc endpoint, and I believe this was just for testing. This is going to use. You know passwords.

B

You know a basic auth password thing, um but the idea is that you can you've got to put some sort of service in to talk to whatever provider. You want to talk to right so say, you're authoring with you know off xero or google or azure or whatever. What again, whatever your external service might be.

B

You stand up some service to to do that part. So I guess this is the part that was confusing for me as well, but this design here this external opposition support, is just designing the plumbing to hook up contour to some sort of authorization server right. That's all the plumbing bits to get everything wired up to talk to something the actual implementation is still. You know at this point you know we have to. You know, make this a little more mature, uh but up to you to get that integrated properly.

B

So I think we've talked about adding uh you know an oidc provider, auth server to maybe extend this auth server project to do oidc as well or something I like to talk to dex or something generic like that.

B

But there's always going to be, like always has a specific service or other kind of you know things around it that make it difficult to make it generic you know without having it be, um having a thousand command flags and stuff to configure all these different parameters and stuff.

B

um So I guess we got to figure that out and understand from the community and you folks that hey you know, what are you looking to talk to? um Do you need you know some sort of generic service that we can all contribute to, or is it specific to your environment? Or you know, however, however, that might might look and feel.

C

Absolutely and I think, from our perspective right we're going to have probably several authentication mechanisms. I think we're going to have some kerberos like in internally we're going to have an external auth provider. Now, from my understanding, our external auth provider is not actually like oidc like fully compliant. um I think I've got andrew andrew. uh Are you on here, because andrew did a lot of that integration? He knows kind of.

A

C

Yeah was it, was it fully like oidc, or there was some kind of special stuff we had to do right with the with that external auth provider. We were working with.

A

uh Yeah they have a different flow. Okay, slightly different yeah.

C

So I so so I think the fact that it's generic enough um steve, like you, said that there's like so many kind of like yeah, we don't want to have something where it's like. We've got to try to fit something. That's so kind of pre-baked. For us, I think, having the flexibility is, like you know, just point: hey, here's, the auth provider. It's going to give you the you know. The results you need, I think is, is great. So that sounds awesome. Yeah.

B

Yeah and we've got a lot of work in um everyone heard of gangway um this thing this was from heptio. uh This is the happy labs thing, but this was the idea of this was to let you get some credentials to talk to a kubernetes cluster through oidc or something so you would configure gangway to then talk to whatever provider that was, um and then you, this you'd log into this website, and it would, it would generate you.

B

A config file then talk to a cluster, um but this is sort of the same thing we just talked about where there's a whole bunch of config files um to do all the configuration I'm trying to see where the I forget where it is, but yeah each one has a different like setup, setup and stuff, um because gaming is a little different than google, which is a little different than azure, which is a little different than even though there's standards and air quotes. um There's always these little itty-bitty things that are different.

B

So I guess one of my point with this was we can pull a lot of stuff from this as well. If we need to to be able to have you know some a lot of work, that's done so um yeah, it's definitely definitely moving along.

B

I know james um got blocked up a little bit on some of the um other plumbing bits internally, like we had to switch to um some to do a single envoy set of cluster load assignments for endpoints and stuff, but now we're over the hump for all those a lot of those bits that we can now. You know, move forward quickly with this stuff.

E

Fantastic, hey, steve. I had a quick question about about this as well. So I know this has been in development for quite a while and uh about two weeks ago. I guess um a new iwolf filter landed in in the main branch. So it's not yet in a release, but it's something that uh the folks that pinterest uh one of the guys at pinterest who left pinterest, but it was finished off by the pinterest folks and it's a it's.

E

A built-in oauth 2, filter, okay, and I was wondering what impact that has and whether that would be an additional functionality to sort of turn on that filter and whether that would make sense. So obviously this can call out to an auth server, but there may be an opportunity to just leverage your filter directly inside envoy versus externalizing. As a as an additive approach.

B

Yeah, we'll have to look. I know during some of the design stuff that james did. um We looked at that. um I like the idea of not having these extra bits running around right, not having to deploy this off bit and configure it and have another thing to manage and upgrade and run. You know, I think, is great, so um yeah. I personally think that'd be a great move to allow that so say you are just wanting to do oitc.

B

You know just turn on the filter and keep going. We just have to look to see how that might fit. um Yeah.

E

B

E

B

I mean, as I said it.

E

Is brand new, but you know we've got teams that are currently doing a bunch of you know: custom lure, coding and stuff, and we've been following this, our filter for a long time and it finally landed, and so the our internal teams are pretty excited because they think they can actually just get rid of a whole bunch of additional scaffolding and custom lure code and just completely get rid of that yeah. It should be nice. Oh.

B

For sure yeah that'd be great yeah um yeah, so yeah we'll have to I'll take a look at that. It says: oidc excuse me I'll, go find that um or if you have a link to it, that'd be great just to look at it.

B

Is it this one? Maybe I've been here before this might be it cool, yeah I'll. Take a look at that.

E

Yeah, it's landed and there is documentation on the website as well now, so it's all sort of been pushed through but, as I said, I think it's in one one. Sixteen zero div, no, it's not in our release. Yet, okay.

B

Yeah it looks like here I added a link to it from yeah there you go from this root, one yeah, all these references all over the place, yeah so um yeah, we'll have to see, see how that might fit. Yeah cool thanks for sharing that yeah.

B

Does that make sense to chad anyone else, andrew shadow, whenever any kind of.

C

Yeah, no, that sounds great. I think uh you know I was just talking to kind of andrew and kevin a little bit about it and I think it it totally makes sense and we're gonna.

C

uh You know it's gonna be a few months right before we, I think, uh kind of already that we've got you know something we can say: hey guys, we you know, we got it and it's it's working or it's not, um but definitely we're going to be doing a lot of uh you know, um custom, I wouldn't say customization, but um you know uh external authentication and authorization. You know at essentially every request level right. Every request level uh on in.

A

C

Case essentially needs to go through some sort of uh authentication authorization and I think, being able to do that kind of at the ingress level. You know really lets people know that yeah we've got a secure. You know, we've got a secure environment um without having to you know, try to manage that on each individual deployment, essentially so yeah for sure, really cool, yeah, okay, um yeah and uh yeah. I I don't think once I start getting in this time. I have more questions, but thank you. This is this is awesome.

B

Yeah, no for sure, yeah and I think timing-wise the folks are looking for that. um I know that, like james james is working pretty hard in this and it's it's middle of the night for him. So he's not here to comment before he is, I know he's got a couple pr's so once that other pr runs or lands, then this one will come in and then um you know we can. We can probably wire it all together being. He already has the sample off server out here, uh which one this one.

B

uh We should be able to wire up something, and then you know, take a breath and see where we are with everything and you know run from there. I think all the plumbing will be there then to do this, so it doesn't look too far away.

A

Cool awesome, um any other questions or discussion topics that anyone wants to bring up today.

C

I have one other, but I can just be quiet for a second, let other people uh you know go, but if nothing, if nobody has anything, I have probably one more that I could probably talk yeah, if you guys aren't for it.

A

I say: go for it.

C

Okay cool, um then I might actually share screen real quick, um if that's okay, sure um and what so, what I was gonna do is. Maybe just talk really quick, um so I've got a I've, got an issue, an open issue and uh there's just kind of some interesting updates that I thought I might want to share um and just kind of maybe get some feedback from everybody on um this. So uh let me see so this kind of started us back in the day.

C

um I would say, probably a month or two ago where we were uh I put in an issue regarding the ability to kind of modify, headers right and essentially what I was looking for is you know: we've got these internal applications that do these three or two redirects right, so you may say: okay, go to you, know app.example.com, and it's going to redirect you it's going to send a 302.1 redirect with a location header of like you know, app.example.com login right essentially, and not all applications can be essentially configured to say.

C

Okay, it thinks it's running over http, because it's behind our cert manager, you know, and all the proxy stuff over here and not all applications can say: hey respond with. You know, https right, and so when we, the problem, is when we embed with iframes.

C

um What you'll see is you know over the browser the request gets sent over https. You know it. Does the search manager ssl termination here then it goes http at the pod. Then the application is what sends that 3021 redirect with that location, header and then it gets fed back and the problem is we sent the request over https, but the response came back to http right um and then that's when the browser goes and error.

C

Protocol mismatch right and what we would need for to happen is that you know goes https http, the application things that's running over http, so it does a three or two redirect and then it's actually envoy. That would be essentially um you know uh changing that location. uh You know header um from an http to an https right and then I kind of talked about you know. We've got these um add these set and these remove functions right in envoy um and then also in contour right, where we can essentially specify.

C

But then I put here you know the problem with like setting right so like if I were to try to do a set on like a per app basis or something you know, then we have. This uh issue happen where it kind of like does a circular circular, like you know, redirect redirect every time um and then uh you know so anyways, I kind of brought. You know that up.

C

But you know long story short, so I was doing a little bit of research and then I noticed that there's actually an envoy issue um here that somebody submitted a pr for right. So somebody actually just uh about the same time that I submitted my issue submitted this in envoy, and this is before. I knew that really there's a dependency on envoy to have the functionality and then contour just does the configuration.

C

But obviously I still think that there's something that we could talk about within the contour in terms of like okay, you know what should the manifest look like right for this type of you know functionality and we're kind of having this discussion. I think now on the envoy issue here, where somebody kind of um you know uh chimed in and said um that uh you know they they you know, look, we've already got a set and remove you know. Are we really just doing the same thing, but I don't think so right.

C

I think that there's actually a need for like a replace functionality, um and this isn't actually the feature that I was looking at. It would be. um It would be another wait. Actually is it this one here ah here we go okay, yeah so was it submitted as a pr uh looks like it was yeah, so so this, so this guy actually submitted a pr regard regarding this right and he was looking for the exact same thing.

C

I was on the location header, um which is the exact same example that I provided in my contour issue here. um But then you can see that we got some feedback this morning, right saying that you know, we've already got api fields to add or remove response headers. um You know there's discouraging from introducing a redundant mechanism, um but I feel that there's actually a need for a replace mechanism where you know would be.

C

You know something like this right, like okay for the location header, uh you know, find everything, that's http colon and replace it with an https colon. I don't know how we would be able to use the remove or I'm sorry, the add or set the set or remove functions of envoy.

C

To actually do this, um you know and and and solve the use case, the problem that we're having and it looks like- I'm, not the one, the only one that's having it um as well, but so I think my I think my you know way forward. Is I'm gonna continue to talk with the folks over on the envoy side and see?

C

If maybe we should add these functions to envoy, and then you know get these introduced in the contour as well, but it I it just looks like there's other people with um you know that are looking for this type of functionality and originally originally.

C

I think he submitted it as a you know, location header rewrite and my suggestion was well: let's just make it a little more generic so that we could actually just rewrite any sort of uh http header um and you know just specify which header we want to rewrite, um because we we already do things in envoy, like um rewriting paths right, so they've got like path rewrites, you know.

C

So um you know we already kind of have this functionality in some regard, but we don't have it for http headers and I do feel it would be useful. But I wanted to maybe stop there and and see if this makes sense or- or you know what other people thought about about this. So.

B

Yeah for sure I I forget now, if on I don't, I know envoy has added remove.

B

I think we made it set to do that, but I don't know you can you can replace the header, though I think if you, if you added a second one that would you'd have two headers which still felt weird.

B

um I had to double check that to see I mean, I believe you know harvey on here would know.

B

What's up we have ad removed but not set, I don't believe yeah. Sorry, I don't think I'm making any sense. I need to check to see if how this replace might work. Dude I mean in envoy itself. um If it doesn't, I don't, it doesn't seem like it does that so from what you need, we have to think about how else we might be able to do this. um Maybe external auth would solve you because you wouldn't have to have your back end app. Do the auth bit then.

C

Yeah, well, this isn't actually really regarding the authentication part, but more just like, for example, you know he has. You know kind of some examples here like okay. If I want to set, you know like service name and then there's some sort of like remove some sort of internal secret header. You know that gets passed around internally. You'd want to remove that before it kind of goes outside of your.

C

You know outside of your cluster, but then something like this right where there's some sort of like okay, I want you know, there's a header and it has some sort of value, but I want to do some sort of, like you know, find replace uh of that value in that http header um and I was looking at like the path rewrites, uh but obviously that's for http paths right.

C

So that's going to look in in the path part of the http uh payload, but um there's nothing for like headers right, and so I think this would actually be a pretty easy thing to do in envoy. I think it's mainly just you know communicating properly. um You know what it is that we're doing and why it's necessary, um but I think it should be- and you know we've already got this. uh You know this pr here where it looks like they've, they've, already kind of done.

C

Some work right in terms of like you know, adding the different uh parts within um uh you know like the protobufs, and you know some of the different functions and stuff. Now they call it location, header rewrite. I would just call http header rewrite right, and then you just give it the which header what you're looking for and what you want it to be like replaced with essentially- and so you know um just my overall perspective, I think this should be actually doable.

C

um But again I just wanted to bring this up to everybody and you know get feedback or you know just kind of. Let you all know that um this is something I definitely want to. You know keep looking at these guys.

B

Cool yeah for sure yeah. Let me go look at that issue again um and I'll. Look at the look through all these as well and see. If I can comment- and you know, help clarify as well. Okay,.

C

Yeah yeah, just I just wanted to bring it up and I'm not necessarily saying this is the way or something but definitely just bring it up and kind of make my case, and you know, get some good feedback on it. Yeah yeah for sure.

A

This is great, thank you, so much chad yeah my pleasure. Thank you. um We are at time so uh with that um we're gonna close out this week's meeting. uh Thank you all for joining, and um we do have a meeting next week again uh on the later time schedule uh for our uh australia folks, and we also have a uh a contour office hours uh next week as well on thursday, the 17th.

A

So please join us then, and uh with that have a fantastic rest of the week. Everyone.

B

Yeah take care thanks. Everybody bye.