VMware Contour Community Meetings, 19 May 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Contour Community Meeting - May 19, 2020

Description

May 19, 2020

What have we been working on?

@youngnick - Address Status Updates for HTTPProxy:
#2371
At some point, we’ve introduced a race #2522
@jpeach - xDS certificate rotation. @tsaarni did the underlying work, so plumbing it through the deployment and tooling. Will be easy for operators to replace certgen with cert-manager. Certificate rotation support makes the secret format transition possible.
#2538 certgen doesn’t have enough RBAC permissions to update secrets
#2537 Stop using ImagePullPolicy of Always
#2524 Update deployment to use rotatable bootstrap config
#2494 Certgen should generate secrets that are compatible with cert-manager
@stevesloka - “ingress/v1” ingressClassName

Contour 1.5 release
Currently scheduled for the 29th May, PDT.

If you have landed a change that needs to be discussed in the release notes, the draft is available and can be updated now.

Issue discussion

#2483 Are we all agreed that we are going to take a Lua dependency for 1.5? (note that taking this dependency does not imply we will give the ability to configure arbitrary Lua filters)

A

Good evening, good morning, good afternoon, everyone and welcome to the may 19th a 20/20 contour community meeting. Today we got a bunch of things that we want to talk about regarding what we've been working on, and we also want to talk a little about the contour 1.5 release that it's coming up. So let me share my screen, so you all can see what we're talking about here so Nick. You want to kick things off.

A

B

Thank you much, but um if you could pop up in those two issues, I've got it that would be great I've been working on taking the address status stuff, that I added in for English status, lasts, release and put and bringing it to HTTP proxy as well. So that's covered in this one 237 one. If you could scroll down, join us, there's a in two three seven one, there's a summary comment: there, I'm working on doing this in three phases: keep down you go it's that one, the one and three check boxes, that's it!

B

So, um there's a basically I'm, currently a step! I have the pr open for step. One of this that is and I hope, they're ready for the second one. Once we do the first one, and so basically the plan here is to get in order to implement this I found once I started. Doing it just the same way that I did the ingress.

B

One turns out that, having in the way that we were doing the status updates having there were two places that would update the status for HTTP proxies when you do that, with the method that we were using, where you use the version that you received from an event from from the API server, then it can lead to your update being rejected, because there were two concurrent updates happening at the same time, and so this, whichever one happens to come in second gets rejected, so are as part of this I am building a new ingress status, update pattern for all of contour.

B

That will be useful for further down the road, but it also fixes this issue and a race that Steve founder. That's two five, two two, the other issue, Jonas thank you yeah, and so this issue that they found is cause at some point since contour 1.3, we've done something that has made it so that you can get this. You can get this issue.

B

The changes that I've had to do for status address on HTTP proxy are going to fix this because they're gonna effectively serialize the it's back to the status of it, so the teal they are here is these two issues will be fixed. I really want to get this done before we relate to 1.5.

B

So we'll talk about that more in a bit but yeah. This is what I've been working on: I'm, hoping that this will. This will mean that then it will unblock us, adding we're going to see if we can there's another issue that I'll find them linked in afterwards.

B

There's another issue for adding HTTP proxy support to external DNS, so that, if you are using external DNS and contour, then you can use fish to be proxy and have external DNS pick up the records and do the DNS magic that it does yeah I think that's probably enough for making one so.

C

Nick in the PR, you have the check list of checkbox items to work on. If you move that into the first comment on the far superior, then github shows you they honest, eater, a list of tasks in any issue list yeah.

B

I'll do that yeah.

C

It's super weird, but it's the only way that it's the only place that we will show you like, okay, that one of for two or four kind of thing, yeah.

B

All I'll do that I'll copy that I'll copy that list into into the first comment. Thanks.

A

Awesome any other comments, questions or Nick.

A

Alright, let's move over to James.

C

Okay, so certificate rotation. So it's only landed all the changes in contour and envoy to be able to in theory, have the XPS certificate rotate and everything just work so I'm working on taking that theory into practice for our deployment example deployment and plumbing it through, so that we can just rip it by default.

C

There's an interrelated feature here which is generating the default certificate if a default XTS certificate for contour in the same format that cert manager does which basically L makes it a lot easier for people to switch away from using search in to using something like cert manager, which is tidied which could be tied into you, know some authoritative certificate vending source. So if you, because of that transition that this couple, because to transition the secret, the TLS secrets format, we also need to automatically pick up the new secrets.

C

So that's why these two things are kind of tightly interrelated. So, as of yesterday, I I have everything working. It's it's like just a bunch of changes all across the copay all across the deployment and in surgeon. So I've got four issues that we'll need to fix. There's our backup date for search in we need to chain in a default deployment.

C

We have image for policy of always I guess, there's some historical reason for that, but it totally make it totally breaks test things, we've kind and I'm struggling to imagine you know deployment that really want that there. Third one 25:24, is updating the deployment to give contour their arguments. That will tell it to generate a rotatable secrets config for envoy, and the final one is having teaching surgeon to generate secrets in the set manager compatible format. So once all these things are landed, you will deploy your new y ml. You'll have a new search engine.

C

What I'd like to do eventually is have each deployment generate a unique, a uniquely named search engine, so you'll get a new search in job. Every time you deploy contour, so search in will run, it will read it it'll rewrite the secrets that and everything else will restart, because the everything else will we start, because the secrets volumes have changed a bit and by the time everything picks back up. It'll work again, so it's really reasonable I think you can do a slightly.

C

If you deploy with something that's better than Q control apply, then you can make it a little bit more slick, but could you control apply does actually work for this case now.

C

The idea is eventually that I think what I would like to get to is for our deployments that every time you do, if you're, using search in every time you deploy you'll, get a news. You'll get your certificates rotated, so that will also resolve the long step. They're kind of that grant issue. We've had where people have been saying where we've.

C

Also, we have this background issue where search in tickets have a fixed lifetime and because we haven't had rotation there's like a year. If people installed contour at the end of last year, the end of this year, their certificates will be starting to run out. So if we can, if we rotate this certificates, every time you install contour, then people who are keeping track with current releases will never have a certificate rotation issue or certificate lifetime issue. Sorry, zip.

A

Awesome any questions, comments for James I.

C

Don't wanna say extra, thank you for sorry to working on this because I think the work that he did there has really let us kill like a bunch of birds with this stone, so I think it's been really really really good, enabling capability.

A

A

All right next up, we have Steve I.

D

Don't wanna be left out, I put something silly on that I was working on, so it part of the the ingress v1 work, there's some new fields that I decided to poke through so in 118, I. Think if you click on that ingress class name, Jonas, there's a link to the blog post, there's two fields that they released. One was the ingress class. So now in the spec, you can define the ingress class and not make it an annotation and there's also a.

D

What's it called implementation, specific type for path types which we haven't talked about yet, but I started poking through just the ingress class. The interesting bit is in the which is funny in the the comments of the struct. It tells you that the annotations should take precedence over the spec ingress class type.

D

So, basically, if you define both the annotation as well as the spec ingress class name, the annotation takes precedence which is backwards, but I. Don't know why that's about migration pause, I would guess. Yeah I think they doesn't want to break yeah break historical users, but at the same time, yeah anyway. That's just that's.

D

What's in the comments um there's an interesting bit is that there isn't I, don't see an aggressive, e1, API class, yet at least even anywhere, so maybe the one night team that goes ta that will show up, but this is still than the v1 beta 1 tapes, so yeah yeah.

B

Yes, I think that's how they that's how they're doing this thing is that they are adding these additional they're. Adding these additional spec fields into interview on Vita one and that's why I got pushback to 119 actual geoglyphs is that they wanted to have a version first.

B

C

Steve sometimes when, when they spec happened some time ago, I filed issues to cover all the changes that I think I thought we needed at the time when I looked at it. So we should have github issues covering all the features that we would need to add to get to take a look sure.

D

I found that so the ISO, the one I, did sign the one to welcome.

C

D

Classname yep, that was very, very helpful, I think there's four features in total I think these two are in 118 and I. Guess some, the other ones will land in 119. You see them yet. Yeah.

B

That that will be that that's another thing where, when we do move to the v1, but when they actually move out code to the v1 is another that draws a hug version line on the 119 as a requirement, because you know the v1 beetle one day if the API server has both versions installed and you request for you, one be the one you'll get the one object you just translated. Just with the haven't been called v1 v2. One note: that's why they've added these fields back to they won't be the one so that they.

B

So you can keep using the v1 people and stuff for a while, but there's another when we actually move our code to use the v1 version, which will be a pretty big change set, and then then there will be double draw a hard line under 119. So we won't be able to do that for a while.

D

Yep cool- that's all I had interesting things with the with ingress.

A

Any other questions comments there.

A

All right and we'll dive into the next release the 1.5 release. Nick. Yes,.

B

On the controller 1.5 release is currently scheduled for 29th of May PDT, which is during the day Friday 28 to May 4 for us I think, actually, that's actually the it'll.

A

Be around that.

B

Time, there's a hot um the, so the key part is all of the stuff that we just talked about should make it him and for those of us who have got changes in the release, notes draft is available so that we can start writing that up and then you know, one of us will be able to write up the usual announcement blog post.

B

If there is anything that people in the community you know need fixed super urgently. Now is the time to speak up. I, don't believe, there's anything that needs super urgent fixes, but you got a week in case. There is something that you really really need. Tell us now.

C

It might be worth someone quickly landing the pier which was filed last night, so the issue which was filed last night to allow multiple addresses for egress low balance, multiple ingress load balancer addresses to be specified on the command line as.

A

C

Useful for the since the first time we shipping this feature, and it should be pretty easy to do. Yes,.

B

It should be reasonably straightforward: I will look as well. It's pretty closely.

B

um There was that I don't think they are online, but yeah person who logged that one. They logged another one about doing a similar behavior to what nginx ingress does for for picking up actual node addresses and using that as the servicing that in the in the status address so yeah I'll I'll have a bit more. That one is the chocolate in there. Yeah.

C

I think that makes sense overall, but you know this close to a release. It seems like it's best to push it out. That's.

B

A pretty big deal, there's a lot of shenanigans and go on there.

A

All right, any any questions regarding the 1.5 release.

A

Okay, moving into the discussion part of the community, meaning so issue, you 2483 talking about a Lua dependency. Okay,.

C

So this is for the 41 responses, with Braille HTTP, to browsers being very aggressive about coalescing connections when you're using wildcard certificates. So we can fix this. People hit this in the wild. We can fix this, but only if we take a lower dependency on an envoy.

C

Now this isn't us supporting lower filters is just us taking advantage of the lower support in the Envoy in the Envoy, but that way programming.

C

I wanted to raise this so there's like some official discussion that we feel that this is okay, it'll, be fine for anyone who's using the official envoy container images it'll, so it'll be fine for anyone who's using the contour deployment llamó, because we use that there's potentially some performance impact, because if you really have sweetly strong latency requirements in you know putting lure in the middle of that it's gonna add a little bit of jitter I'd.

C

Be surprised if anyone who's running this kind of configure in a cloud environment would really be out of measure a difference. I, don't think they're cloud environments have consistent enough performance at that layer that level to be out of bitchest only but I wanted to make sure everyone has an opportunity so that we cool agree that this is something that we're cool with.

B

Yeah I think you can say on the issue that I'm okay with it as a as a as a temporary fix the but yeah I, do I would like to see something done with with I'm Stream envoy to fix this problem. I think it's. There is an issue already like I said, like James said, though, I think that we're probably gonna have to do it. Yeah.

C

I think to get out of this I, don't see anyone upstream there's no-one upstream working on it, so I think. If we, if we want to get out of the lower dependency for this, then we need to have someone in our team. Take it on upstream yeah.

E

That does the temporary Lua solution potentially open up any new attack, vectors nope.

B

Yeah cuz it we're not accepting Lua from muses we're just like doing our and Lua photo defeats. This exact specific.

C

It's super super simple, so, basically, when we generate the conflict in on in contour, when we generate the config, we know the virtual host that we're generating at 4 and so on the routing table for that virtual host. We stick a lower filter that checks the hot virtual that checks the HTTP Authority, and if we get a request for a HTTP Authority, that's not the virtual host we're dealing with we respond with a 41 and then browsers and browsers will like retry the request with a new session. What.

E

Is 41, what's like the short mr.

C

At misdirected request yeah, so basically it means it means. You sent a teasing, a request on a TLS session that doesn't have this host name. Is that supposed to mean, because yeah there's less verbiage in HTTP 2 spec, which allows browsers to be what allows clients to be pretty aggressive in how they coalesce connections? With regards to the TLS server name?.

D

Is there any other like reason, someone would not want to do this. I know, I know it's. It's fixing an issue that I don't know, I guess if you didn't have one to lure dependency, what happens if you don't compile that into Envoy disease? Is it just air out or I'm, not sure? So, if you don't have this.

C

What will happen is envoy will not accept. They envoy will reject the config change and then contour will not be able to recover yeah.

B

You'll get into the situation where, on where envoy has to restart. If Omer ever rejects a complete change, you have to restart the Envoy and to have it. You have to fix the problem and then how to restart the Envoy to have a connect back to code, but.

C

In fact, it's not fixable, because contour will always send in this particular case. Contour will always send the lower filter. Sorry, unless we add it, unless we had a feature flag to this as I was going to know it wouldn't be, it wouldn't be recoverable. I.

B

Don't so the feature flag would need to be a disable. This functionality feature flag like which it should be enabled by default a short, and you know, I, don't like I'd, prefer to wait and see. If, if anyone needs it disabled, then I'll add it like my feeling, but James I mean I'm willing to defer to you on this one, but the I mean, and the other thing we got to make sure we do is update the list of olive oil extensions in the in the Envoy documentation thing that we have yep.

C

Yep, the current patch I think has that change.

C

B

James, what do you think about that? Would you like their for disabling this book.

C

I'm, putting my sre hat on and I always want a feature flag. Whenever I wear that hat, we don't have any precedent in the contour project or feature flags, and it's not so you know and.

B

C

Flags, isn't really a contour thing historically, but adding a feature flag for this would make me feel more comfortable. So.

B

In my mind, that's wacky yeah yeah, the pros pros dating a feature flag. Are that if someone really doesn't want this, they can turn it off. I kind of would be surprised if anyone really doesn't want this. The only reason that I can think that you really wouldn't want this is, if it hey, you do really even want to make sure that you don't have lower, enabled on your envoy or they are or you or you run into some performance problem but like if someone.

A

B

A performance problem and that's something that we would probably need to look at anyway to see if you know, if you're running into a performance problem with this, then everyone's gonna run into a performance problem with this.

B

The thing that I worry about here is that, by having a feature flag for this one, we may lose out on knowing why people would be using it if you said I'm saying like the yeah having having like, although because the only reason I'm willing to accept that is because I really don't believe that it's very likely that people will need to use that feature flow.

B

C

I think it's unlikely, the the one, the one user who has expressed interest in running in to pawn their own contour, build, told me that they at that point yet they're still running upstream contour, so I don't know of anyone. Yes, Pacific Lee, who would get bitten by this I'm Pro feature flag in general, I think, like eventually, I would like to propose that contour gets feature flags, but maybe also it's not something that I want to stuff in at the last. Second I, don't.

B

Think you know the last second right now and I do want it to be that when we do start adding you, we will need to do some. Something like adding feature flags at some point, but I want them to be in the config file rather than on the command line and there's a couple of other things that I want around that. So I.

B

Don't think that this is a good time to do that, and and because of the reason that you know I kind of want to hear from somebody if they, if it doesn't work for them and why it didn't work so that we can evaluate if this is the right way to be doing it.

B

And you know if there's something else we didn't know about so I guess to me that, like wearing we're in the tech we've had like I'm happy to make the call that we don't that we don't put a feature flag in at this time. But you I do appreciate having the discussion James. Thank you for raising it yeah. So.

C

It sounds like for now for the one book 5 release, we'll just make sure it's really clear in the release. Notes I'll actually take an item to file an issue for feature flags so that we can so that we can revisit that. There's a time discussion, yeah.

B

Okay, I think that's a good idea.

B

I think it's it'll, be you as I said, as I've said previously in those meetings, I am working on a sort of a policy change thing update on about how we configure contour part of that is then I want to start deprecating having so many command line flags and move to just putting all those settings into the into the config file, and so that we so that the because, then the config file is to change their command line flags, because removing command line flags means that your theme no longer starts up.

B

So the company file is safer in that respect, um so yeah. So my feeling is personally I'm, okay with us, taking a dependency for now on the walk as long as it's in this limited fashion, and as long as we have as long as you tell everybody that you now have a little dependency cool.

C

Is anybody else on the call have any feedback on the Lua dependency.

E

What's the level of effort for actually changing it inquire on voice, so we can remove the lowered dependency. Any idea.

C

I'm gonna say a month elapsed. It's at.

E

C

That's so this every.

E

C

That there's no it's mainly around process right, there's, no and I think the actual code is will be that big, a deal it's more about. Okay. What is the right? What is the right design? Can we get agreement on the right design and then working through there?

C

You know contribution contribution steps, so it's just those things tend to take time.

B

I think we got like one minute left, Jonah's right. So sorry what I was just asking. Does anyone else have anything.

A

E

A

If there's nothing else on the agenda, then we're gonna close it up for this week and with that, thank you all for joining and have a great rescue every week. See you next time.