Kubernetes Network Gateway API Special Interest Group, 13 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG Network Gateway API Bi-Weekly Meeting for 20220613

Description

SIG Network Gateway API Bi-Weekly Meeting for 20220613

A

All right, it is the gateway api meeting for june 13. 2022. uh we've got a few things on the agenda today, uh but before we get into all that, just as always, uh this is a kubernetes community meeting, which means we follow the same code of conduct that exists elsewhere throughout kubernetes community.

A

That also means that you're very welcome to add things to the agenda. Please feel welcome to uh to talk to ask questions to provide feedback.

A

I want everyone to feel welcome and it's been a while, since uh we've we've said this, but is, if there's anyone new to the call that hasn't had a chance to introduce themselves uh anyone that hasn't introduced themselves and is interested we'd love to hear from you.

A

No pressure, if not, though, hey.

B

I think so I'm joining this call for the first time, so my name is sneha. uh I actually work at microsoft and uh we're definitely interested in looking at the gateway apis. So we currently have a solution called agic out in the market for our azure cloud providers and uh it's ingress space right now, like the english apis. So we're definitely looking at the gateway apis and if that's something, we should uh invest our efforts into in the near future.

A

That's uh that's awesome. It would be great to have some additional feedback. uh You know. Obviously I represent a cloud provider uh lewin on the call has also been involved in this discussion too, so great to have more more involvement and really as you're going through the api or, as you see things here, really appreciate your feedback throughout and welcome yeah. Anyone else.

C

And I joined after a long time so hello, everyone.

A

Welcome back harry.

D

Yeah, I haven't seen you for a while.

C

Juggling upstream and your employer, you know yeah yeah,.

E

Yep we've all been there.

A

Cool all right- uh well, I think, uh that's great uh great to have new people and some people. Returning um with that, let me jump right into the agenda. I'm not sure who added this. I have some guesses, but uh I think we've all been. We've all had various discussions around the appropriate status code. Here uh it is almost hilarious. How much time we spent on the you know determining which of these numbers is appropriate, which of these status codes.

A

um I went through and added this rfc information, but I think I know someone other than me added this agenda item first, so maybe they have additional contacts if they're on the call.

A

E

All right well, I think that I think actually sorry it has been a while since last week. I think I added this at meijer's request to talk about like what we're supposed to be what we're supposed to be doing. Thank you.

E

Thank you, yeah. That's my sorry.

A

That that's all good, uh so I I wanted to provide a little bit of context. uh You know it just. It just happened that last week I was able to schedule some time last night with tim. I want these to be a more regular one. I want to have more signet api reviewers involved.

A

We have some people out of office that are other sig network api reviewers, so tim is basically the last person on the hook uh and it's notoriously hard to schedule something where everyone's available, but uh this has led me to a broader themed idea of if we could get to a kind of standard release sequence where you know we know every four months. Let's say we're cutting a release. Then we can schedule time with these api reviewers months out and block off this block of time on their calendars, where we can all meet and discuss.

A

You know this release candidate and any little things, so I I want this process to be better, uh but with that said, this is the feedback that I I got from tim as we were going through these uh status quo, uh as far as which made sense uh and and for anyone who who isn't following we've, we've gone back and forth.

A

Like I don't know, we started as a 503 as as a way to represent when traffic is reaching the gateway and there's a route that matches the request, but that route is supposed to forward traffic to a service and that service does not exist or cannot be resolved for some reason right. So it's that that kubernetes reference is invalid for some reason.

E

There's there's an associated case that we also have, that is that the kubernetes reference is valid, but there are no valid endpoints yeah they're, two they're, two extremely similar, but at the same time different enough that that we have discussed having different codes for them, uh use cases yeah.

A

Yeah and at least for the first of those days, uh the first of those cases, uh and actually I can't remember what we've settled on for the second, but I know what what we discussed. What I discussed with tim last week was that to him the thing that made the most sense was a 502 bad gateway.

A

I would spend some time going through the rfc. uh He seems to have it memorized. At this point. I still actually have to refer back to it, but uh you know I went through. You know I think the the status codes that make the most sense at first we had well. First we had 503. Then we went to 404 because okay did not find a current representation for the target resource seemed to make sense, but then I think it was mike had some concerns about that.

A

Hopefully, I'm not getting.

F

Names wrong: it was me.

A

Okay, okay uh and now tim's guidance here is, you know, maybe bad gateway is the appropriate thing. uh Service unavailable did not sound right to him. He was also okay with the idea of something more generic, just 500 right. This is just an internal server error.

A

Don't need to be more descriptive than that uh 502s felt uh slightly more accurate, although the 404 description feels relevant it's in the wrong class, in that it's a client error instead of a server error um which all came back to 502. so without wasting too much time on. This does 502 make sense to the group of people on this call.

A

Does anyone have a different preference? That's one for me. Okay,.

D

E

D

Have to be clear that I mean the 502 is very specifically when it did not. The gateway did not get a good response from the servers, and so you know to me that's typically where um you know this request has been sent off and it's not the you know the response has not come or it's been invalid.

E

I think I mean I think that the relevant thing here is like we're going to be whatever we do here. It's going to be not 100 correct, because the because, like the the you, the spec, doesn't really like the the exact codes, don't really mean like there's, there's no exact thing. That means you know hey. This is going through a proxy and the proxy can't get to the back end like for some reason like it. I think the the more the the first case that rob talked about. It's not that you made a request.

E

It said there's no valid request to make because you've made a mistake in your in your gateway, api config, and this and you have talked to a um you- know your you've configured a backend, a back end that that that can't be found right like so though.

E

So I mean you can't route, you can't route to it and find out that it doesn't exist because because you can't even find out because it's not configured, there's a configuration error in your proxy.

D

Yeah and that's that's where I think actually a 503 is the appropriate response, because it's the service isn't available. It doesn't know how to get to a service that that is being requested.

A

I think what throws me off of 503 is the rationale that 503 is traditionally for overload or scheduled maintenance, and neither of those seem to apply here. I I would, I would agree with so I I think all of these you could make an argument for right.

A

Like I, I get what you're saying like service unavailable, the name is correct, and then you read the the rfc reason you're like well, not quite uh if I have a two back gateway, the name is correct, but as you're pointing out it's not, you know it's not like we're making an upstream request. Unless you call that upstream request to the kubernetes api server and hey, I didn't get a response from it or I didn't get the response I wanted from that. I there there is no perfect translation that I can think of here.

A

um I personally, I think that 503 is, is often used for things. That are not this, that you know that that represent that kind of maintenance window temporary overload those kinds of cases, so it doesn't feel quite right to me with that said, I recognize that our flaws with bad gateway.

A

I I've been talking a lot. Maybe uh anyone else have have ideas here.

E

I think if no one feels comfortable with it, maybe we should just take the like hack, the mean free path here and go with 500 like there's some error. There's some server-side error in the story like yeah, that's yeah wrong. You.

C

Mean xxx or 500.

E

500, the actual 500 error code.

A

Yeah, I think I would rather, since we're going to have conformance tests around us and everything else, I would rather have a specific error code, a 500 than any 50x, uh because that just adds more variability between implementations.

E

Yeah I mean so. I like, I think the most important part is that we definitely have to have something that's in the 500 class um and if the and, if we can't agree on 502 or 503, then I think we should just not waste any more time on this and call it. 500 is my proposal.

D

I can get behind that yeah. I actually think 500 is actually a pretty good choice.

A

It seems it seems pretty non-controversial it's just such a generic uh way of handling representing this um okay, uh we can uh update the does. Does anyone want to take the.

E

A

Thank you appreciate.

G

It I filmed my issue countdown dropping in slack in the chat. Sorry, um I think the one thing I was trying to disambiguate is the case that you were specifically calling out the attempted reality into a bad back end um from all the back ends exist and like are configured properly, but there's no matching route that one we might want to keep as a 404.

F

Yeah 404, for that one makes sense, even if it's not like kind of like you know there isn't a proxy 404 equivalent. Maybe that's something we should suggest to the itf, but you know whatever, um but it's like it is a commonly understood to be like this thing. Isn't here response code? um It's like it's also like 500, I usually say like something went wrong, whereas not finding like having no configuration is not something that went wrong. It's the expected result of not having any configuration for that.

E

Yeah yeah agreed, I think I think yeah the one where it's going to get a little tricky is the route's not accepted case, because the if you, if you go back to the that uh comment, rob the the one that mike mentioned there. The route's not accepted case due to an issue like supported kinds or you know, for example, uh your reference policy removed or something like that like it, it's going to just get we're actually going to have to be very specific about um about with different error classes.

E

We're going to have to say this one. This one is a 404. This one is a 503 or 50 500 whatever, like the. um I think we might have to have some guidance in the actual, like implemented guidance in somewhere in the actual um thing that yo says that actually lays out every case we can think of and what needs to happen and then have uh tests to make sure they to make sure that those things are actually what happened it could be.

E

Maybe maybe the answer here is to either write a longish issue um that documents the exact like each use case and what happens uh and um or a gap or something I don't know, but it feels like. We actually need to have a document that writes out here's all the places, all the things that we can think of, and here's the the uh the use cases and if we come up with any more use cases we'll add them to this document.

G

I've been meaning to consolidate my assorted issues into a gap or something like that. um I think like from the implementation side, it's actually more straightforward than it may appear at first, um because the not accepted case just means it doesn't get programmed. um So it's actually like the default handling of it. It's the same as like just a not match thing so yeah.

E

G

Something I can try.

E

F

E

Yeah that'd be awesome yeah because I think, and I I think that if we have, if you write out like here's, the different things that can happen and here's the error code that happens in those cases, then then it should make everything much more clear.

A

Yeah, I agree uh actually shane were you about to say something. I hope I didn't cut you off nope. No, I'm just imagining sorry all right. uh So uh with that said, uh does my conclusion. As I've outlined in this uh agenda, dock makes sense if there's no matching routes, it's a 404 if there are matching routes, but the back ends are either not found. Not they're missing can't be routed to it's a 500..

A

E

Yeah, I think that's pretty clear. I think we just need to yeah. We need to call out the uh everything all the config is valid, but there are no endpoints as as another use case, I think that's probably a 500 as well, but I think so too. That's.

B

What I mean by.

E

Writing out all of the different things that happen.

E

Will be important so yeah.

C

Yeah, do we want to be prescriptive of the response body in these cases.

A

I don't know that we can.

F

It's kind of weird to me because, like the client that receives these errors, isn't going to be able to do anything with them or really care about. You know what they happen, like the thing that actually matters for getting it fixed is like the admin who manages the http route will see like a condition on it. It says, like you, can't do this, because it conflicts with the other thing, and you take this corrective action, whereas the client's like, oh okay, it's an error message. I guess I shouldn't do that.

E

Yeah yeah, I don't. I think that personally I would say this. This to me rates is one of the things where practically there's probably tmi, to give a client that you know any any internal details about how the proxy works or anything like that feels risky to me to give to give to the client, because uh it's one of those things that feels like it might turn out to be really handy for enumerating for attackers to enumerate exactly what has been used. If you, if we put too much information in there.

D

That's exactly what I was about to say: it's uh a lot of a lot of customers. You know our users, uh it organizations, don't want to expose that kind of information. um For that very reason,.

E

Yeah the same reason that that.

D

E

Know your standard apache configs, don't include like the apache version in the error messages.

A

Okay, great discussion, um I think mike you're, going to follow up with a longer term uh description of what we should do here and nick you you've probably already commented or updated on the tracking issue for the milestone I haven't yet, but I will yeah I'll do that cool great thanks?

A

Okay, with that, uh let's move on shane. I think your next, uh your changelog changes thanks for adding this.

H

Yeah there's a couple things going on in here: um well, first, oh no! This is that one okay, so I just kind of feel like the change log process that I just went through is a little bit tedious. So um I thought that, instead of having to put your change, log entries in your issue is kind of like just a blurb, and then somebody has to go through and find them later.

H

We should actually just make this a pr checklist item update the change log when you have user facing changes and just have the change log constantly growing, with like a unreleased version that we kind of dump everything into and then, when we're ready to do a release, we can move things in or out of that, you know into another unreleased while we're releasing things that are actually in it or so forth.

H

So nobody- everybody that had got on here and like responded to it, seem to think it's okay to at least try this, but I just figured. I would bring it up in this meeting, so other people could, if they missed this. This is what I'm suggesting to do and see what other people have to say.

E

Yeah I commented on there I'm I'm fine with it um yeah. As I said, I think the we have a slightly more involved system in contour, but this is. This is a good first cut. That's definitely worth trying.

G

E

Of the similar.

G

G

We have a similar system uh to what you're, describing with the directory um for contour, x or console. The issue we found with updating a single change. Log file is that it frequently makes prs stale and conflicted um when you update the change log and then one merges and then another update said I don't know. I haven't looked this closely to see if that might be susceptible to this issue or not.

H

I mean there will probably be some conflicts right after we merge it. I would guess, but in general the way I've seen prs go, I mean one goes through every couple of days. We usually put the thing on there. I think it would be, and we don't usually split our releases up very much. We usually just have everything in there when we release. So I don't.

H

I don't necessarily predict a bunch of conflicts, but it's also one of those things where, like we can try this see if it's better and then, if we're like ah nah, it ends up being just kind of this mess at the end, then we can just kind of pull back on it and try a different way.

E

Yeah I mean, I think we have the we have the directory method in our back pocket, um like I guess, the the so, which I think it. I think it's fine uh and I think the um I think the the only the only thing is like it might be slightly irritating to change twice, but I don't think it's a big enough deal to.

E

You know to even warrant probably more discussion that we have done now. Yeah.

A

I yeah, I agree with everything that's been said. I am concerned about you know. In the past, we've had different files that ended up being points of conflict where every pr had a conflict needed rebase, and we took a lot of effort to avoid those and to re, rework our infrastructure to get rid of those points of conflict.

A

This feels like something that could introduce that again uh because of that I would prefer to hold off on it until after rc goes in uh just so that we we at least get an rco, uh but with that said, I I am interested in trying this out. I really appreciate all the effort you went to shane to run through all those pr's and get the changes out and try and make them into something sensible. uh Creating a change log is not easy, especially with our current lack of tooling.

A

So thank you, um and I think this is worth trying like like nick has said. Like others have said, we can always move back to. You know change, item profile, kind of thing in the future, so yeah cool any any additional comments.

A

All right keep on moving then uh shane. I think this one is you too.

H

Yeah so travis and I both have been pondering the because we have two implementations of gateway api now and in both of those implementations, we've kind of run into the kind of interesting conundrum conundrum. That's uh the status addresses, don't necessarily make sense, so the the way status addresses is documented on the gateway right now is basically just all addresses that ip addresses that the gateway is bound to.

H

So in theory, I mean you could put load balancer ips from services, cluster ips, pod ips. You could put whatever you want on there, but there's kind of this problem of there's no context so like as it stands. Right now we put the load balancer ips in our alpha implementation. We just put the load balancer up.

H

We make them the first one so that they're kind of the uh the nice cube cut will get displayed ip address and the only usage for these addresses for us right now tends to be um to just go talk to the gateway via the load balancer ip address, which is a context that we can only figure out because we're humans um so like the the the actual underlying api type for these addresses is just a list of strings so like there's. No none of that context can kind of be programmatically determined.

H

So if you wanted to kind of do an integration where, like some piece of code, wants to go talk to the address or to the gateway it, it would need the context of knowing like how it might want the context of knowing that it wants to do it over the cluster network, as opposed to like the public internet stuff, like that. There's no way to do that today.

H

So I just wanted to kind of throw that out there that, like it's very limited, what we can do with the addresses today, we pretty much only meaningfully put the load balancer ip address on there at the moment, but we want to do potentially more wondering if other people had a similar or dissimilar experience, um so just kind of open the floor to people's opinions. Here.

F

You actually jump in and say no, so um we do the opposite of that. We get the both the load, balancer addresses and the cluster ip address um of the service. Sorry.

H

You're right we do that, but we don't do anything with the cluster like with the.

F

H

F

We yeah, I think, well, you put it somewhere. I forget where- um and my thinking was just get rid of it because, like the kind of stated purpose of gateway is traffic going into the cluster uh from outside of it and that traffic obvious or those clients obviously cannot talk to the cluster ip.

F

It's internal, uh it's kind of the open question where, like definitely with ingress we've, also seen where, like um people will use ingress configured, you know proxies for traffic that is internal to the cluster uh and whether like there is any purpose to having like cluster ips for those types of clients or if we just kind of say, like you know, it's a bit non-standard. So if you want to do that, you just grab grab the service host name and use it instead,.

E

Yeah, I think it's in my mind. It was definitely intended to be a similar behavior to the to the ingress and service like load balancer address uh stanzas, where it's like. The point of this thing is that if you programmatically want to know how to connect to this gateway, you can look at the. You can look at the addresses and you should be able to connect.

E

You know from from the view to that address from the context that the gateway's relevant for most, which most of the time is going to be outside the cluster. You know, like you, don't know about cluster internal networking internals I mean that's the that's the sort of in my mind the definition of the of what a gateway is. Is it translates from things that don't know about cluster networking internals like cluster ip to stuff? That does um let.

H

Me, let me give you an example of where, in the context of like programmatically, the context of actually talking to it over, the local cluster might be important. So we might have other ways to solve this, but at least currently uh in one of our implementations, we have the notion of the a control plane which runs separately and being able to communicate with this data plan, with our gateway um over addresses that it finds inside of the gateway addresses.

H

Theoretically, I mean that's kind of the the suggestion of like a different context that might be worthwhile and in our case we at least potentially have a use for that. I'm not saying that's the only way that we could do it and we could definitely like iterate on this, but there's there seems to be something there that you would want to potentially have like hey. I've got control, plane traffic going to the gateway uh as it were, and that control plane traffic, probably shouldn't go over the public internet when it doesn't have to so.

A

Yeah, I think I think this is really interesting. I know it would benefit a number of implementations like for for google cloud. We have internal load balancers right and it is, although most people who are provisioning uh gateway, class ilb, basically know that. Okay, this is going to give me an ilb if you want to write something programmatically, just against gateways as a whole. Understanding that this is something like I have to reach a different way. This is this is an internal address that requires some separate way of referencing it.

A

uh Maybe a more concrete example, is when you're running conformance tests.

A

It is helpful to understand that this type of address needs some kind of custom round trip or logic like the the request has to originate in some other place or some other way or in the in the case where there are multiple addresses, like I think, shane is describing here, understanding which one you can pick like. If, if I know that I'm in the same network, I should pick the one: that's you know the same ip.

A

uh I think I I'm not sure how to represent this, but I would, I think, it's useful for a variety of different cases. um I agree that it's it's basically the same thing as load, balancer, service, type, load, balancer or ingress addresses, uh but this could use some more some more context. I think um so. I I would personally love to see a gap around this.

E

Yeah, I think I don't know, I think that yeah right writing it up with like some issues and some some sort of more detail about the exact use cases that people are thinking here would be useful. The in my mind, I think, I'm like uh the there's two things that I'm thinking of like I can. I understand it is good to have more context, but the on the other side.

E

I don't want the the sort of the the simple case that doesn't need the context, I.e the internet phasing context it'd be better to not have to have that have the context if it doesn't need it right like that. Yeah like.

B

E

Oh, I prefer to see this be like an optional thing that you add on somehow, rather than a thing that everybody has to do to manage.

D

I think to me the question is: what is the purpose of including this address, so you know, is it?

D

Is it so that um I can know all the addresses that are affected because the status is wonky or is it so that I know how to reach the particular gateway? That's having a problem, uh and you know those.

B

D

Those are two very different things, and um even with with the second one, where it's you know, basically it's like. Is it the traffic interfaces or is it the uh the management interface? You know which could be the same, but in many cases won't be. um You know, then, if, if you're, this isn't a service, but uh you know you're running this in your own kubernetes uh cluster.

D

You know you might have multiple instances of a gateway actually, so you might have actually multiple uh ip addresses. That would be a quote-unquote management ips right. So um I think we just need to be clear on what what what the purpose of the, including this address information, is.

H

Yeah and like travis corrected me because- and I think I was the one that actually did this- we did at one point just put like the service cluster ip into this, just because I think it was just kind of a misnomer, but all we've meaningfully done. The only thing that we're meaningfully doing is the load balancer ip, the public ip address.

H

I think it might as much as I'm just asking this to try to get feedback and actually appreciate all this feedback, because it's good to know that everybody's kind of roughly on the same page, you know like yeah, it's a little vague.

H

The only thing that we, I think really matters today- is the load balancer ip, the public id. So maybe it's just a like it's possible that just documenting that better and just saying. Actually, this is just for publicly internet, accessible, ips or or something like that. I mean something that kind of pushes people towards using it for the purpose that is most generally used. Right now might be okay, as opposed to like going all out and trying to add, like different networking contexts that you can add to your ip addresses.

E

Yeah, I think I think that documenting it, a little better is probably good. I think that we need to be careful, though, because it's pos, it's possible that uh and external to the cluster. Ip may not be a publicly visible ip right. Like that, you yeah you might, you might be doing like something rob talking about as like an internal load balancer, it's internal to your network but external to the cluster.

E

All right, that's the like! It's the outer edge of the cluster, so you don't know about cluster networking. That's again, I keep coming back to that's why I keep using that language to talk about what a gateway is. Is that you, you have stuff that is owned by you and probably uses like an rfc 1918 space, um but doesn't it's not like public address, but it's still external to the cluster right, and so that's the, I think. That's!

E

The key definition is that currently, the expected behavior- which I think we have 100 not well documented, but certainly the way I expected it to be- is that the any addresses that are listed in the status are the addresses that something that that is external to the gateway you can like pull a gateway and or that you can tell clients that are external, the gateway to connect to.

H

Does that make does that make.

A

H

I mean yeah, the thing is: go ahead.

A

No no go for it.

H

Like when I kind of I've kind of brought this up, I think just with rob in passing like a week or so ago, and he's like make yep sounds good and I do agree. This could be a gap, but the difficulty is like we.

H

I we don't necessarily have an impetus right now to make this a get, because we can, for instance, like reference the service that is ultimately um load balancing traffic to the gateway and like do things that way for the purposes of the one case I can think of right now, which is like the control plane traffic to the to the data plane. So I don't know if I'm actually going to go ahead and make a gap on this right now. I think maybe I'll start with based on the feedback that I've heard here.

H

Some clarification and yes I'll, be careful about what you just said. uh Nick about, like you know, just be the distinctions that are made in in what's supposed to actually be in there and then maybe something else will come of it later, because at least today I don't know for sure that there's anything else that we would be like there would be a lot of pressure for us to make any other change.

E

Yeah, I think I think the one thing that I would also include in that is that it um it feels like gateway. The gateway api is all about defining the data path, not about the control part, about the control path and about management traffic. So I'd be pretty minus one on on any of the addresses that are included in that status, um having anything to do with the control plane, um I think if we want to do something else for control plane, we should do a separate control plane thing that is very clearly separate.

E

A separate control, plane thing, the main use case. That's.

A

A good thought, yeah yeah. I agree with all those all those thoughts. I want to clarify that.

A

I guess also agreeing on something nick had said earlier that I, when I see addresses in status, I assume that they are externally accessible. uh What I think could be helpful, at least the cases that interest me about. This is some kind of optional uh indicators additions, some some kind of indication that this address may not be externally accessible but is accessible via this particular with this specific modification like this may be internal to your network internal to your cluster.

A

Some other thing, some set of well-known traits, as has kind of been alluded to this, is a very similar problem in the service and ingress apis that we have survived without any better representation of that, so it can work without any, but I I'm open to a better way to represent that. Obviously, we have uh motivation for that. I know I've seen some implementations, maybe it's istio, where they.

A

The first thing they set in gateway status addresses is a hostname for service uh and then, if that services type load balancer it switches over to the load balancer ip, when it's assigned, so you could see it. You could see like in that initial step. You may want to have some kind of indication hey. This is not broadly accessible. It's limited to the cluster you're in. um I think there's probably some other use cases like that, but, as of others have said, we've survived without this.

A

It's just. It seems like a nice to have kind of thing.

A

Yeah any any additional thoughts on this one.

A

Cool all right, I candace. Thank you uh yes june 20 is a new u.s holiday. I I meant to add this to the agenda. Thank you for adding it. um I personally have the day off, uh so I am. I will not be able to make it to the meeting. uh Are there enough people that would like to continue to have a meeting on june 20 so a week from today.

A

I will take that as no uh but uh as always, you're welcome to meet uh without me uh or anyone else, um but I will not be there, so I will plan on canceling unless I hear other otherwise, and then we can meet again june 27th.

A

Okay, I will call that cancel all right. Next, one is the o5o milestone. There's been some movement in here. Thank you to nick for adding most of these issues. This is basically just a follow-up from tim's feedback. Last week I have addressed two of them. I chatted with meha about the last one in here the status code, one she has said that she'll get it done in the next 24 hours or less.

A

uh So that's great, I think we're in a good place. I want to just run through these really quickly the actual pr's uh they're, both relatively large, but I think, they're non-controversial.

A

uh This is, I think, the only the only implementation I know for sure that wants to support both reference policy and reference grant is console, but maybe there are others as well, uh either way it's great for a better, smoother experience.

A

So, for that purpose, uh adding reference policy back into v050- I this will mean it comes with this kind of this is a really hard screenshot to see, but yeah anyways, you get a deprecation warning uh when you use this in o50 seems like a reasonably good experience and you get a warning for anyone in that upgrade path.

A

I would appreciate any reviews on this. I think hope this one is pretty straightforward and the last one is just that where we had web hook and crd validation, this just changes it to just web of sorry, crd validation, so a net removal of lines. In this case, so two relatively small changes. I would love to get these in asap.

A

I really really do want to get a release candidate out, as I'm sure many of many of you do too um so yeah, that's that's what I know of in this milestone. Are there other things we should be looking at before an rc.

E

I really hope not I'm tired of turning over rocks and finding more rocks underneath same same.

G

Yes, uh I do just want to warn all the other implementers about uh the pr that I think you merged today. Rob um that bumps the community's client libraries and introduces a couple of breaking changes due to a fork of gnostic and um a breaking change in a major version of the lager. I think so that'll be fun.

E

For implementers all right, so we need to ensure that in the changelog we call out that there are breaking up kubernetes api changes and you need to bump your kubernetes depths. Yeah.

A

Good call thanks for that reminder um with that. uh I set aside some time tomorrow for myself uh to uh work on cutting rc one out. I I'm really hopeful that tomorrow, at the same time, we'll actually be able to get rc1 out the door uh with that said, I am very welcome to anyone who wants to be part of that. If you want to just uh sanity check what I'm doing, cutting a release is still kind of a.

A

We have documentation, but we've also changed a lot since our last major release so like this is our first time with release channels, so a second set of eyes, or a third or fourth or whatever double checking what I'm doing or if someone else wants to drive I'm great with that too. But my goal is to cut a release tomorrow afternoon if it makes it and by tomorrow afternoon I know everyone's in a different time zone, but I mean the same time is now just tomorrow, just 24 hours from now.

A

um So yes, uh if, if anyone is around just uh just ping me, I I'll add you to the invite, uh if not that's fine too, but that that is my personal goal, I'm shooting for 24 hours from now having a release out, um and I would appreciate any help uh getting to that point.

E

Cool fine, I'm happy to help, but there's a good chance. I won't be yeah. I understood cool.

A

All right uh last uh on this list is travis, uh something about listener, host names.

F

Yeah, so I was looking at basically um previously. We kind of didn't really do listener conflicts correctly for a variety of reasons. Looking at finally implementing those properly, um it seems, like you know, hey if you have an existing listener, there is nothing stopping you from just changing the hostname port or protocol to something else um that could result in a conflict and overall it looks like the designing or listener conflict system. I've tried to make it. So the idea is that if you add a new listener, that's conflicted it doesn't.

F

You know it's not random, which one of those wins. If you have a previous listener, that was already not conflicted and ready, it keeps going because you don't want to like destroy existing listens, that people are probably actually using um this one sort of makes that difference.

F

It seems like the only way that you'd be able to track that like if you you know, enforce that you have to like need to say, like you know, uh if something critical in the listener changes, so it's hostname porter protocol, you essentially have to treat it as new um and remove that sort of precedence from it and that seems like you'd have to maintain state, which I think, generally, you don't want to do in a reconciler. So I'm curious.

F

Other people have had this sort of same thing with you know determining whether a listener is conflicted or not, and if they have like strategies around uh handling it.

E

I think in contour we have gone backwards and forwards with discussions on this. um I 100 agree that it's a bad experience to have someone change.

E

You know change a kubernetes object and have your config go away, but at the same at the end, at the same time, the config just change like the config is changed like I, I am kind of a bit of a purist that whatever's in kubernetes. You should do and if it's broken then yeah sure it breaks be careful yeah like um yeah.

E

That's I think the the trying to hold state in the controllers like you're talking about is so fraught with race conditions and danger, and you know all that sort of stuff that I'm you I'd kind of. You say like I. I don't think it's worth the the sort of trade-off in user experience, because the big risk you run by doing that sort of thing is that you're trying to do stuff to hold the user's hand to make sure that they don't cause himself a problem.

E

But by doing that they might cause themselves a problem and then not know about it. Because they've made a breaking change and nothing's broken.

F

Yeah they're still at least informed about it and, like you know, one of those listeners will definitely be conflicted um so, like they'll know that there's some click conflict available, even though it's not both of them um that standpoint up at least okay with it, but I can see, like you know, yeah just you know. If you make break like bad configuration, will warn you about it and like you, it will actually be broken.

A

I we we have also gone back and forth on this too, and we actually ended up, at least for now, uh taking a slightly different stance of uh going taking long leaps and bounds to uh to store last no good state outside of the controller but somewhere and and trying to use that as a reference point as a point to recover, to etc.

A

To be determined how that works out, but it is something we're exploring uh this is this is a tough problem. I I don't know I I I completely understand and get the idea of it like what nick was saying like we should do what the user wants like it in many ways, it's better to fail more quickly, so that it's when a user is changing, something and so they're aware of it they're there they can recover as opposed to changing.

A

Unexpectedly, you know days later or some other thing for some other, like your controller restarts or whatever it is.

A

This is a I don't have a good answer. I I just we're exploring something: we're exploring a different approach, I'll try and do some digging and see what we can like. There's some docs I'll see. If what we can share, but high level, it's trying to it's trying to store last known good state somewhere else.

A

I we don't know how well this will work. Yet. This is just an idea.

F

That one may work for us reasonably well um like we have, unfortunately like state, that's outside the controllers already, which causes us problems, but we're not going to get rid of it anytime soon. So we could take an approach of like you know. If you create configuration that results in a conflict like you, we just won't, even if you added like other listeners that aren't conflicted in that same set of changes, we just don't apply to until you clear all the conflicts um yep.

E

Yeah so like we've had so one of the biggest concerns I have so we've had. One of the biggest problems we've had in contour is that um okay, I'm going to delve into some implementation details for a second, because they are relevant. Give me a minute um so that when you're, when you're doing an envoy, xcs programming um envoy can send back a ack or a knack whenever you send changes right now, contour doesn't handle the knack case where you send config to envoy and it it basically says.

E

I can't accept this for some reason um and contour. Does it just drops that on the floor and so it'll just keep trying to send the bad, config and boy I'll say hey? I can't take that practically what that means is for users is that um if we generate a bad config, because we take some user user intent and turn it into onboard config and send it over to envoy and envoy responds back and says no.

E

Basically, that means that the envoy will silently accept, stop accepting all changes from contour until you figure out that there is a problem and then do something about it, um and that sort of class of problem is the reason why I tend to be you do what you you do what's in the thing and if it causes a problem, you tell them as soon as possible, you're better off breaking people immediately when you make a change like rob like rob said, then, then breaking them at some random time, because your controller restarted or you had a data loss in your external data store, or something like that.

E

I think, if you're going to do something like that, an external data source out of the controller is a necessity because of the problem that if your controller restarts for some reason, if you're holding that state and your controller restarts and you then then the change will you then it'll start up and then then the config will be dropped, probably because you'll read it in as new as net new and be like hey. I don't have a last name good conflict. This config is bad. You know I'm going to send no config right, like that.

E

It just means that one of the reasons that I'm so pro you know just do what the user says is. It makes it a very predictable thing. You change something. It immediately breaks right and it's very clear that it's broken.

F

um The other option that I, I guess kind of implied here but didn't really talk about, is that, like you know, um these aren't immutable now, there's not a whole lot of reason. We can't make them individual other than that, like there apparently isn't any existing kubernetes api construct.

F

For this sort of thing, like I know, ingress class um controller string is immutable, but it just kind of like in the api definition says this is immutable and doesn't do anything to indicate that so I'm guessing it's in an admission web book somewhere um making these immutable would be kind of annoying.

F

In that, like you know, if you want to like you know, reconfigure kind of an existing listener, you basically have to either like delete it and then recreate it or like um create a copy of it with a different name and then rename that afterwards, um but to nick's point like that is kind of a a good, but it informs the user that something is dangerous there before it could become like actually dangerous.

A

So I'm interested in in what that means or how that would work, uh you're right with ingress with ingress class. It's just uh invalidation that exists inside core code, uh api, slash, validation for networking, but the same thing we have web hook, validation for gateway, class controller name that enforces immutability.

A

I really wish there was a built-in construct that did for us. There's not, but we do have. We do have a thing in gateway api. That means immutable. uh With that said, uh for this specific concept, it feels complicated uh what what defines a listener? Is it the listener name? Is that kind of the the thing that so, if you change name at the same time as you're, changing host, name, port and protocol? Does that just? Is that fine? Because it's a net new?

A

It's it's basically like you're, adding a new listener and removing the old one if you're changing name uh or is listener defined by its index in the list like just, I am listener number zero.

A

uh I think that would be a little hard to define uh and then my other concern is, as we are nearing a beta release once we get to that point. Adding more restrictive uh validation is going to be near impossible, unfortunately, um but if, if there is a compelling thing, it may be something we can. We could look at between rc1 and but I it sounds.

B

Pretty complicated.

A

Yeah- it's probably too late, um so I I think it's something that we would have to work around and come up with some guidance for how to handle these changes.

E

Yeah we've we've specifically left the uh exactly how the listeners work to be up to the implementations. Like the you know, we've got some guidance that hey you, can it's it's acceptable to take multiple gateways that have distinct listeners and merge, merge the gateways into like an eventual config. That's 100 fine um and we did like.

E

We have given guidance on what happens when there is a list of conflict, but we haven't given guidance on like what constitutes a listener conflict right, like you, I think at the moment it's it's expected that it's, the tuple of like host name port protocol and probably name you and those things need to be distinct rather than like the index or something like that, um yeah, I think yeah, the.

E

I think it is, as you say, it's a very complex problem, but I think the the the overarching question that is very important for us to answer is how much behavior should we uh mend like how much user protecting behavior? Should we mandate in the upstream api like how much should we encourage people to do that? That's the, I think, that's the sort of the underlying question that underlies this symbol, probably underlying other other issues as well and having an answer to that question I think, is really important.

E

I mean you can tell by what I've been saying you I my my opinion is this: is software for people who should have a fair idea of what they're doing it's bet. It is, at the end of the day, a better overall user experience to like screw over the people who are making conflict changes than it is to screw over the end users and have you change copy changes happen at random translator.

B

E

Of having the api be very sort of you know, whatever's in the api is.

B

E

And you know is what the user intends and if they make a mistake. Well, you know sorry be careful like that is the sort of my position, but I think that that's a pretty extreme position- and I I would I understand that other people might not be- might not be cool with that.

E

So yeah, I think that's the that's the question that we need to answer.

F

Yeah all right sounds good. I think I learned all I intended to so.

A

This is a good, a good discussion and uh it it may be worth. Maybe this is a discussion item or an issue, one of those, uh but uh just I know, you're, not the only person running into this or, if you haven't others will soon.

A

So it would be good just to have some kind of central place where we can describe what others are doing at least come up with some kind of best practices.

E

I think a discussion on items. That's really good to me, though,.

A

Great uh yeah travis, if you want to start a discussion that would be awesome around this, and I can comment on it. I'm sure anyone else can too.

E

You know I'll be right. There, cool.

A

All right, I think, uh nathan added this one and unfortunately he had to leave. I missed that we could have moved this up in the agenda, uh but I think this is this specific pr is about conformance yeah. I have been uh failing to get caught up on conformance tests. I I do want to get some of these conformance test vrs in uh as soon as possible. I think nathan is saying that uh this is ready to go and we can leave additional assertions for later.

E

But I think that this one is a good uh is a good example of a thing that that I also want to talk about. I know we're almost out of time, but I just wanted to drop it. I can drop it and run the um that uh right now. We don't have any way for implement for the conformance test to conform, assess all or nothing. um If you install just the standard crds, then you will not pass conformance because it includes tests about reference policy and reference grant which are not in the standard crds.

E

So um I think uh we we've got two things we need to solve here, um number one. We need to move reference grant to just to be in the standard ones as soon as possible, which I think we need to do anyway, but also we need to have some mechanism that you can specify like what resources you're being conformed for or what version you're being conformed for, or something like that. We're going to need something like that as well, because the current conformance test suite is for like layer, 7 controllers, not for layer, 4 ones.

E

Once we start having tcp and udp route conformance we're going to need to have people be able to say I'm not doing http period, I'm only doing tcp route or whatever. um So we're going to need that at some point as well, um it just seems like. Maybe we.

G

Added, I think I added basic scaffolding uh like that in my initial reference policy: implementation um tbd, if it's sufficient or uh fine-grained enough, but there's at least something there. We could start.

E

G

The features thing sorry I forgot about that.

E

A

Yeah it that took me off guard too, because it's weird to move reference policy or grant to experimental, because it I mean technically we're just moving a subset of things to stay to standard and everything prior to that was always experimental, but it still feels weird because the you know prior to o50 the standard set up or the default set of crds and be careful of my wording here. The default set of crds we installed included reference policy uh and in o50 it won't uh you'll have to install experimental crds.

A

For that, uh that's gonna be weird uh and so yeah like like you're saying we need to be careful uh in how we document this and how we communicate it. um I once we cut rc one. I want to spend a lot of time working on documentation, blog posts, etc, making this approachable, because it's a big change uh and as much as it could technically be a no op for implementations of the api.

A

It's going to require some thought and understanding for anyone trying to upgrade themselves so yeah, um okay, uh yeah, that's, I think, that's all I have uh and we're at time uh any last things before we call it all right. I will see everyone in a couple weeks and uh enjoy uh your week and uh we'll talk to you soon. Hopefully, after an rc cool, see, ya.