Kubernetes Network Gateway API Special Interest Group, 31 Jan 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG Network Gateway API Bi-Meeting (APAC Friendly Time) for 20220131

Description

SIG Network Gateway API Bi-Meeting (APAC Friendly Time) for 20220131

A

All right, uh this is the january 31st meeting for gateway pi, uh depending on where you are uh anyways. uh Let's get uh straight off with uh richard's uh proposal for the uh grpc route.

A

um I don't know maybe.

B

Richard can you give a bit of an introduction for what you are thinking here sure uh so, first of all, as rob said, uh I did come to a meeting about well over a year ago, at this point uh to sort of just gauge interest. uh I've come back now with something more concrete. Having had this on the back burner for a while now and I'm more committed to getting uh this through now.

B

um So I at this point the gap that you're seeing is, I would call it a rough draft state, so I'm certainly open to any feedback that you might have, but the core of the idea here is: we want to be able to route grpc traffic in an idiomatic way for grpc users, so that does not mean using full uris, though grpc generally runs on top of http 2.

B

That's viewed as mostly in implementation, detail and folks, using grpc shouldn't have to know the way that services and methods are encoded into uris and http 2.. um So for the most part, I've tried to recreate http route as much as possible and deviate only in the ways that it makes sense for grpc.

B

As a protocol and as a framework, um so the areas in which you'll see that happening are under the matches you'll see instead of a path match. We have a method match, so you have um you're going to match against a service you're going to match against a method, and you can see the details of exact match and regular expression match lower in the section that details the go structs.

B

We also have a few filters here. I had a few more filters in mind, but at least for this initial gap, I tried to keep it limited to what http route has. I think in general, I'd like to increase the number of things that we have here.

B

One element that I have not incorporated yet into this gap, but I think maybe needs to happen- is um validation of configured features, so I think, generally, for the gateway apis folks have thought about well, a gateway, some sort of uh proxy in the middle that will handle ingress for you, uh something that I'd like to for this, to also be able to handle is um proxy list service meshes right, so clients that connect to other clients, without necessarily using an intermediary proxy. I know there is some support for this via crds using parent refs.

B

That would connect to a crd.

B

So our issue is that certain types of data planes cannot handle all of the configuration that your conventional data plane might, and so we need some way to um have people mark within their resource, in this case the grpc route and say um I'm intending to run on a proxy list data plane and it might not necessarily handle all of these features and if they configure a feature within their grpc route, that um is not supported by the data plane. There's a configuration time error.

B

Otherwise, you get run into all sorts of tricky issues where, like you know, you have um a security feature that you don't know isn't enabled until an outage happens, something like that um so other than that. I think this should be pretty uncontroversial in terms of the design. So one of the main things that I wanted to hear today is just sort of um gauge the level of interest among the community and um see if there is any just high level critiques of the the concept that I'm proposing here.

C

I think I'm pretty excited by seeing this. uh We have been wanting to like explore this area for a while, but we just put it on a back burner so having this as part of the core or extended api, I think makes a lot of sense. uh One thing that I mean one thing that I think.

C

On seeing this is that got me thinking is how about conflict resolution. I mean we have this problem in http route itself, but uh do you do you see conflict resolution as being any different, so imagine that two routes are configured for the same grpc service or somebody's or the same grpc method? If that's the granularity somebody is operating at so how? What do you think about that.

B

So my first question would be um do you. I actually don't know this detail. Do you allow multiple http routes to be configured with the same hostname.

B

Yes, we currently do you do okay, uh I didn't realize that, under like my existing mental model, you wouldn't allow two grpc routes to be configured that have the same host name. So how do you handle this for http route today.

A

A long set of rules, uh but it's uh you, know longest matching host name, longest, matching path, uh most number of matching headers. I I I forget when you go down the list, but there's a list somewhere. That's rather I don't know how broadly that is actually implemented, but at least conceptually there there's an agreement.

C

There is also a filter at the list at the gateway listener level, where gateway admins could say that okay, any any route selected by a listener, they can only have these host names or these host name suffixes. Things like that.

B

Do you merge the multiple rules that match uh according to like some precedence or is it you select one as the the rule? It's mod merge. Okay, um I think I'd need to incorporate that into my mental model, but um if you already have a set of rules that seem to be working, um I don't have a strong reason at the moment to say that we shouldn't incorporate that into this design like I don't want to reinvent the wheel here with this.

D

Yeah, I think um the the sort of match and merge behavior was like the best that we could the best. So it was like it. That was definitely an arrived after a lot of discussion uh uh position so yeah. I would definitely recommend that you have a look at it, pause it and and make sure that it makes sense for what you're talking about and um yeah and then try and incorporate it.

D

I think yeah a lot a lot of the time that sort of thing like it's one of those things where it makes writing you're running the edge case, tests and stuff like nasty, but it it's the incredible flexibility is very useful for people. As rob said, I don't know it's in the api.

D

um I don't know how well it is available, tested and and so on, in all the implementations at the moment. So you I think that it's pretty good the concepts themselves are pretty good. um You know, I feel pretty confident that we've we've dialed in the right concepts there, but you sort of made a mention before that. um You know it's it's in there and it's in you so I'm like. Well, we don't really.

D

We can't really say it's in there and it's in use like in you know and be a hunt and be 100 sure that we're really doing that. But but are we going to change it? I'd be pretty reluctant to change it at this point, like I'm pretty sure, we've got the right thing going on right.

B

um I I will I'll take a look at that. um I know at this point. This is like a rough draft of uh a gap like there definitely needs to be more detail in terms of the comments, um and I think that should certainly be one of those things. That's incorporated.

C

I have one more question: go ahead.

A

Go for it here.

C

Okay, so have you thought about grpc reflections?

C

Okay, like would those be transparently supported or like do we need to express those at the api like at the crd definition layer or not.

B

It's an interesting idea: it's it's something I've thought about. um It could represent a security issue, potentially uh we've we've thought about like, for example, in the grpc library automatically turning on uh a reflection server, unless you like have an environment variable or something like that, and inevitably we we get to the point where we say: okay, we can't do this and I think you probably have the same issue here.

B

You could maybe make it a little bit easier in terms of ux by offering, like a rule that says like enable routing for reflection, um but that has not been incorporated in this design. Yet um it is an interesting idea, though,.

D

So there's a couple questions in the chat about uh listeners and how they and how they upstream, like the parent rev stuff, will work. I would like to talk about that too, but I do want to just drill in on that a little bit um yeah. I think um we've had some requests like you know. Obviously, working contour we've had some requests for, for literally that exact same thing, um you know to do the the uh jason you know to do the http one json.

D

uh You know rest translation to gipc for you um and so um yeah. I definitely think. That's like the point that I'm trying to make there is there's. Definitely some appetite for the for the grpc reflection stuff around.

E

D

You and having you having the ability to have the gateway api represent the concept that you know. You've got an ingress controller that that's a http 1.1 gateway, but that has a grpc route behind it and have that represent in some fashion. The that translation, that translation point and have your implementation be able to say. Yes, I can do that or no, I can't seems like it would be very useful.

D

um So I guess that leads on to the other thing I just wanted to briefly mention before we go on and talk about, like the paragraph stuff um is one of the things I'd like to see you talk about a little bit and the is how the status will work in my the status for the jrpc route object. What I have found is when designing uh designing any of the route objects.

D

It's really easy to miss things until you hit the status and you start trying to be like how am I going to tell people that this is correctly configured and how am I going to tell people that, what's that something is broken and if so, what is broken and that that's? When you do this, when you do the status design is when you do that, um I think you you have got. Did you have a status thing in there? I couldn't.

B

D

So I I I it's 100 fine, like you said it's a draft. It's no problem, you know, but that is definitely one of the things I would say is spend a little bit time and think about like on a jrpc route. How would you know that everything is configured correctly? What would you want to know? That's not configured correctly, you know how would that information be surfaced? What would it look like? Look at the http how we've done the http route? That is not to say that how we've done the http route status is finished.

D

This http route status and in general route status is 100, not finished, so any contribution you have, there will be gradefully accepted because probably there's improvements we can make to the http route objects. The gateway status is pretty good. I think, but like the the http route status is not so and again the reason I say that is what that I have found that we there's been a couple of times when we go to we're like oh, this is almost done and we go to do the status and then we're like.

D

Oh no, we missed this and this and this and this and this because we didn't think about how we were going to close the loop and get information back to the person who configured the resource. Interesting.

B

If you could point me towards like any documents or prs where those discussions have happened, I'd appreciate that.

D

um I'll have a look I'll, have a look and see, if not, please feel free to hit me up. um I have thought a lot about this and yeah yeah so um and uh yeah starting to try and write some things down about it. So, but there's this is very one of those very uh deep magic kind of areas at the moment, because there's no best practice yet. Okay,.

B

I mean it's definitely something that I care a lot about. One of the things that I think most about in terms of like configuring, grpc traffic for like different kinds of service meshes, is the config validation. Because of the reasons I brought up earlier.

D

E

D

I think the um the config time, validation that you were talking about in my mind, is that's what we have the the validation web book. For um again, this is there's a.

D

I think I took the action item, rob um to do something about like defining how how important that the validating workbook is, but in essence, we haven't written it down anywhere yet, but um but basically that the the the validating workbook that the gateway api supplies is an integral part of the thing that is intended to do exactly what you're talking about that's intended to make it so that you have config time. You know some level of config time, uh validation that that says.

D

The thing that you are doing is not supported, or the thing that you're doing, I can't do or that sort of thing, so that so you know I did that it's much better! It's much better! Ux, as you say to have you know, you try to apply an object and you get uh you know, rather than uh you apply an object and then you have to go and look up the status or worse. You apply an object and then you go to check and see if it's working and it's silently not working.

C

D

Yeah so yeah, so I 100 agree with you on that, but and there's plenty there's plenty of other work. We've got to do in the rest of the in the rest of the core bits to tidy that up, but yeah. I just wanted to make sure that you knew about that. Definitely.

B

Yeah, um so I guess uh things in the chat for proxy list: would the gateway advertise an address or something in addition to the route, so the service would be dialed directly rather than through the gateway. Or am I misunderstanding that the gateway advertising address so.

D

I think in this in this case, what we need to talk about here is you mentioned briefly about.

D

You know how there's some configurability uh available for using the parent ref um right rather than so you, I guess one of the things that I would that the gep needs to talk about a little bit is like what happens when you attach this to a gateway, um and then you- and but also like as you talk about like you know, if there's a proximalist like data plane where you're talking about a service, mesh style thing like what is the upstream resource that you would talk about in that case right, like you're, a mesh resource, we put a mesh resource as part of the gateway api.

D

I don't know, but that's the sort of stuff we need to talk about.

B

Sure um so the way I'd imagine it suppose you want uh to have a grpc route that can be picked up by either sidecar proxies or in approximate service mesh. Then I imagine you define a single grpc route and you'd have one parent ref, be your conventional gateway and that's going to be picked up by your sidecar proxies and then um for proculous mesh you'd have some other resource which could be a crd.

B

It could be some new core resource called mash or proxy list match, or something like that, and the presence of this other resource in the parent refs would lead to the control plane programming. The configuration such that um a client using the host name defined in the grpc route is directed to the back ends associated with the grpc route.

B

I don't know if that makes things clearer or murkier.

F

Yeah, I I think that makes more sense, so this is like the practicalist grpc, like is the cherokee xds integration. I remember seeing.

B

Something like this for, like a year or so ago, okay,.

F

Okay, that makes more sense thanks.

B

Right so one of my one of my mo's is: um I want data plane polymorphism for grpc, like every piece of configuration I create ideally works for both um sidecar based meshes and proxy list meshes um and then mchale asks. Is there a need for a special listener to support grpc routes, and this is part of uh the gateway resource? I don't think so. I think nothing about gateways. The gateway resource should have to change, at least for the sidecar based data plan, option yeah, I'd.

A

Agree with that uh one, I I'm a little interested in grpc route from a completely different perspective here.

E

A

This is kind of a new precedent for us in that it it feels like it's a it's a protocol or a route that lives on top of another protocol route. We have with http route, it seems, like you know, from the outside. Looking at I haven't, I haven't dug into this as much as I wanted to, but it seems like a lot of the content in here could theoretically be translated to an http route.

A

I you lose a lot of context with that you lose I'm not saying that's a good or bad idea, I'm just for context and for you know future facing you know. Let's say we have another protocol a few months from now that lives on top of http. You know how do we decide as a community that this protocol on top of hp belongs as a separate route type versus something that can be converted into an hp route?

A

So I guess I guess that's a lot of questions or discussion without a clear question, but really maybe the clearest question would be. How could you is it even possible to take the content you have here and build something that would translate it to hb route, whether or not that's a good or bad idea? I don't know.

B

um I think it is technically possible. Yes, um the the data plane polymorphism stuff that I just mentioned could get tricky, because now you have to support procurement service mesh for http route when I know of no http proxy list service mesh implementations, um so in in theory, yes, it is doable, it would be not ideal for users of grpc and then because of the polymorphism thing not ideal for users of http either.

B

um I also like make the case for grpc having a special relationship with the kubernetes project in the intro here, and I think that's decently. Solid right, like grpc, is already a somewhat privileged protocol within the kubernetes ecosystem anyway, so I don't think it leads to like sort of a slippery slope uh argument here.

A

Yeah- and I I think you made really good- I I liked your introduction- I I thought those were great points. uh I think that's just something that we should touch on somewhere in this gap. Just you know, you've already shown pretty clearly that grpc has this special place in kubernetes, but just uh and maybe this isn't necessarily require requirement of this gap as a whole, but something that we need. We as a community need to figure out of what protocols deserve their own core route types, which is something we've kind of been struggling with.

A

uh It seems like there's pretty broad support for grpc here. I just need to have some kind of clear guidelines going forward of what other protocols make sense.

D

Okay, yeah. I think um I think that that that sounds a lot like an alternative's considered section to make, um but uh luanne has had our heading up for a little bit, so we should uh hear from him.

G

Yeah yeah, I I have a similar question actually mentioned about there's a proxy list or you know I had a similar question on http route itself, because today you know, when you look at http route to find out the status you have to go to its parent gateway. As you mentioned, you know you may not sign. Implementation may not want to have you know gateway, so I'm asking. Is it possible to have a static status inside the gateway object itself? You can store whatever things you need. You know um that's my question here: status.

B

In the gateway, as opposed to the route.

G

No a status inside the http route- oh so today, hd vr object itself has no status. You it basically has a pointer point to parent status.

G

Today is a gateway which you can have many http route or trs route, and you have to go through the condition list, which is the list to find out the one that you think you belong to your gateway and and take it from there. So what I'm saying is: is it possible to have a you know just for each http route, like any other object, can we have a status there.

D

I would say yes uh and we hadn't built it yet.

G

D

I say absolutely: yes: the hdb route should have a status um one of the rule. The sort of the guidelines that I keep for myself with this sort of thing is, you should be able to figure out. What's going on with the thing you care about by hopping to as few number of objects as possible right like so, if there's a problem on your http route, you should only need to look at your http route to see the problem. Is there?

D

I 100 agree with you um and the same with you know grpc route or whatever, and that's that's why I pushed the status thing before richard that you know like you should be able to find out if there's a problem with your route by looking at your route, not by having to go to the gateway to look at things like 100 agree. This is a thing that we have.

D

That has been on the back burner um that the you know like I said I care a lot about, but that I haven't had a chance to to come back to circle back around and have a look at yet. So I would encourage. I would encourage you to learn to have a think about what sort of stuff you would want to put on there. um I think we've do we have conditions on there.

D

Yet I can't remember it's been a while, since I looked at it yeah, so if you can put, if you can squash some if stuff into conditions, that would be good, if not then yeah like, if you want to be able to put some uh information about you once the reconciliation process is finished and the handshake is finished and everything is good and you want to be able to have a way to represent this route is available through these gateways or on these ip addresses, or something like that.

D

That would make a lot of sense to me. um Yeah rob.

A

Yeah we have struggled with status for a long time, and I think, uh if, if you're willing or able to make a an issue to track this, it would be really helpful. uh Some of the boundaries we've or problems we've run into uh are one uh any any status you put on. Http route needs to be nested within the gateway, because we we need to. This is true for any protocol.

A

You just need to know what controller is responsible for writing that status, and we want a route to be implemented by five different controllers in an ideal world and them not to fight over status. So maybe another way to say that is: it needs to be name space by gateway.

D

A

Yeah good good point, uh so that's why everything is already kind of namespace in that parent ref status thing. uh The other concern that I would have with propagating status from gateway down to route is, if you have a gateway, that's uh say representing a load: balancer, that's flapping between healthy and unhealthy, some, it's it's not in the the best state. You can imagine that say you have hundreds of routes attached to that gateway and every time it flaps between those states.

A

You have to propagate status, to you know, 100, plus child resources uh that adds cost and other concerns there. I'm not saying this is a good or bad idea, and I may have misunderstood exactly what you're representing, but I think at a minimum. We should have some kind of issue or discussion uh on github just to continue this.

G

Okay, so the next step is basic, raising your each uh issue on this okay cool. Thank you.

D

Yeah, I think, as I said like I'm, I'm I'm in broadly sport, but yeah the one of the one of the traps, the classic kubernetes api trap, is uh fan out problems you know, especially for status updates. Where affecting one resource then means you have to write. You know uh 20 or 30 updates and, if you're doing that a couple times a second all of a sudden.

D

Your api server is very busy just churning churning status updates, which is not ideal, and so you just need to be careful about fan out problems um in terms of.

G

That's a very good point yeah. So one other question on this. If I can ask do you ever envision, like today's service right people, service is a way of grouping all the back ends. Do you ever envision that http route will be replacing like become the first class citizen? People basically would define http route without even define I mean gateway.

G

Is there but it's more like linking to which controller manage the http route, but hp raw become like people who just use http route and even without gateway and and then the service just another like a deployment uh grouping, the back end parts together and.

A

You want to take, should I get it yeah, so I I would love. I would love to get to that point. I I think, uh there's a few things one. I think there's going to be a lot more routes, specifically hp routes, but routes of any protocol. Then there are gateways in any given cluster and I hope that gateways become not an implementation detail, but an infrastructure detail that are largely unchanged through their life cycle, where you're just attaching routes.

A

uh So so I hope that's true and- and I also this is a terrible idea that I haven't pursued very far, but it's at least technically possible to for an hp route to forward to something that isn't a service right. So you could you really could simplify your stack again, not saying it's a good idea, but like you're saying you could simplify it. So http route is really this core resource uh and you're, not thinking too much about gateway or service you're.

A

Just thinking about your deployment and your hd route, um I don't know I don't know how how long it'll take to get there if we'll get there, but I do think hp route is really going to be the most commonly used part of this api or maybe grpc route who knows but routes in general yeah. I.

C

Mean like I think, what rob just described is like. If we had to build up on ingress resource, then what would happen.

D

Yeah totally- and I mean I think it's actually one of the things. I was writing a thing today, uh the other day- and I realized that when I went to the intro page of our thing, we actually don't have like a a two cents description of what the gateway api is actually for. um In my mind, the gateway api is about establishing um a way for people to talk about um communication between services like but but also, but where that crosses a cluster boundary like uh yeah, that's what the gateway, the gateway itself.

D

That's what it is. The gateway sits at a some sort of cluster boundary. Exactly what boundary that is, is a bit fuzzy but like there's a definitely a logical boundary that the gateway sits at and that's. It translates in some way between stuff that knows about the inner workings of inside the boundary and stuff that doesn't, and so you can kind of move that boundary around and put it like that.

D

The the gateway or the thing that's upstream of your routes is like a mesh or something like that, and you in that case you're talking about things that are like talking within the mesh and things like that, but they, the idea here, is that there's some layer of like something knows how to do this lot and something outside doesn't, and that's the that's the key thing, and so in terms of service and stuff, like that in the future.

D

Obviously, the biggest thing that we are looking to uh not replace, because it will never go away. It's a v1 api, but you basically subsume is service type load. Balancer is the biggest one that we are trying to get we're trying to make it. So you don't need to use that anymore right, like with the gateway api, because you'll have the gateway. Api it'll be a better, more configurable, more explicit, less magic way to do the same thing that you do with service of type load balancer today, the same with ingress.

D

The idea here is that your hd pro plus a gateway should be, should provide you everything you can do in ingress, but make it clearer, more explicit and able to be moved between providers much more easily. So you don't end up with. You know, 400 annotations, on a page that describe all of the cool things you can do with your ingress right like that is not what anybody wants.

D

We want that to be in the upstream api so that everybody doesn't re-implement that in a slightly different way, and so that's so if I were to summarize in sort of a few things, it's like take the stuff that we've done before, like build apis that take the stuff that we've done before, but better and make it interoperable um are sort of the big goals of the go away api project in terms of sorry rob, I'm almost done. I promise um in terms of uh you know what we're going to do about like do.

D

We see that routes will replace like service, or something like that I mean service is always going to be the bucket you put your actual pods in right like that's. That is what service is really for. What services ended up being? Is it's in the it's really handy to have a bucket that you put some pods in for lots of reasons, and the service has ended up being the place where anytime, you need to make a bucket and have pods, and you need some some extra thing about that.

D

You just chuck it in service right like and that's what services end up being. It's a big grab bag of features, and the idea here is to pull some of that stuff out and make it more explicit, make it easier to understand and less magic. Sorry rob.

A

I'm done no worries, uh yeah that that's a that's a good uh description there in it I'd even I I think what richard is proposing here with grpc route in this idea that we may have proxies routes.

A

Make proxy lists, in my mind, make might be gateway-less routes, which almost turns us into an api that should have been called routing api instead of gateway, but you know we're not going to rename our api again uh but anyways. uh I I appreciate this this uh route proposal, just because it it starts to it, starts to make me think at least about uh routes that may or may not exist.

A

You know may or may not require a gateway um which is true of all of our routes, but it seems especially true here so yeah that that's all I was thinking. I don't know if there's any more, it looks like there's a bit a few more questions or discussion in the chat.

B

So a question from mikhail, uh I think a really good question, one that I've spent a lot of time. Thinking about. Are there any use cases for http routes and grpc routes to coexist for same hostname? If so, uh so that we will need to merge the rules from different route types uh yeah?

B

So uh in the past, what we used to tell people to do for um grpc, traffic and http traffic is have different host names for grpc and for http, and people don't like to hear that people like to hang things off of um like path prefixes. So, instead of having like grpc.food.com and food.com, you have food.com, grpc or, or something like that.

B

um So we decided to stop telling people to to have different host names and instead to support a feature basically like this and the thought that came to everyone's head when we asked how do we accomplish this, merge is well okay, you make the more specific match apply after the more general matches right, so you've got some route that owns slash and you've got some route that owns like slash, grpc or some better name, um and after looking at that, along with like header matches and the various other match options that we offer, we realized, I actually wrote a proof for this.

B

It's mathematically impossible to determine what the most specific match is. um So I sort of put that away- uh and I realized like we're talking about um encapsulated protocols here right. So if you just sort of assume that um the one that is lower in the hierarchy, like the more general one in this case, http is the more broad one. Then the problem just sort of goes away. You say these two routes share. Hostname one is http, one is grpc.

B

Well, okay, apply the http matching rules first and then you apply the grpc rules first and the same logic could apply if you're talking about tcp versus http right, like okay, do all the tcp traffic here, except for the stuff that ends up being matched by this and um I think the whole the whole diagram just sort of flows pretty well. If you do it that way. um Does that answer your question, mikhail.

E

Well, yeah, at least it's what I think is we will need to have define to define the rules so the so that we all understand the expected behavior. So maybe it's worth capturing it in the proposal.

B

Definitely um so my proposal would just be apply. Http matches before grpc matches in this case. Does that? Does it sound reasonable.

D

Sounds reasonable to me. I think that we, you mentioned uh the tcp http as a similar sort of isomorphic mapping, and I agree um we haven't actually talked a lot about that, um but that is definitely you a similar problem that that needs to be resolved as well. But you.

E

D

What happens when you have tcp routes and http routes that match the same things like, I think the implicit assumption we all had was that it would behave exactly as you say that tcp routes go first http or else go. Second, you know more specific trump space. Specific is the rule here. Right um and yeah. Grpc is more specific than http.

D

Definitely harry.

B

Does that answer the.

D

Question that you had about the route without a protocol like.

C

Yeah, I mean it sort of does, but I mean what sort of I mean we can do a lot of policy management because we already have a different route type, but uh what would be like?

C

I don't know? I haven't thought through this question. Right like we are introducing a protocol without we are introducing a route without a corresponding protocol. Does that violate any decisions that have made in other parts of the api or like anything like that.

D

Yeah I mean I don't it doesn't seem like it to me, because this is, I mean grpc is still a protocol right, like you mean a transport layer like or sort of a.

C

Transformation so yeah, I should clarify by protocol I mean we haven't, we have a defined list of protocols inside are inside our repo right, like yeah, so like the listener protocols, that's what I mean.

D

ah Right, yeah, okay, the list of protocols, yeah yeah, so part of this will be that we need to define. Grpc has a listener protocol.

C

I think that's where we are not all in agreement. I think rob.

D

Just said, the opposite:.

C

Thing but like.

D

That part of it would be that if, if this were to go ahead, then we would need to say we need to define what it means to be a protocol in the listener and how that works and how that implies. Rules like we talked about and that's what you're talking about right, like yeah, okay,.

E

Yeah, I agree that that.

D

Is a discussion that we need to have I'm not saying that we should do one thing of another but like that is definitely a discussion that we need to have.

C

Okay, yeah, that that's that's, I think I was just thinking about that and that's the question in the same direction. Rob you go ahead.

A

No, that that covered it well. I also am not quite sure the the implications for listener like uh right now we say I I think the way that we rely on it is that uh implementations are supposed to fill in the route types they support by default for a given protocol.

A

So you could say that if a if an implementation supports grpc route- and they see an http listener, they could say hey. I support both http route and grpc route for this protocol. You've specified- uh I maybe that's, maybe that's fine. Maybe we don't need a new protocol, I'm I'm not sure, uh but more discussion needed on that.

D

Yeah, I don't think we need to solve it in this call, but yeah.

F

D

I think the action item for you here is just have a look over the oval, how we talk about the the listener and how the listener float, how the the route to gateway binding flow works.

D

I know that a lot of what you're thinking about is not the the route, the gateway binding flow because of the proxy listing, but but like I think, if you understand how that works for the http route, then then you'll understand a little bit more about what what is saying here and and then we need to make sure that we address that as part of this kit definitely.

A

Cool well, I want to leave some time for the other things on the agenda. Were there any last comments on.

D

Jrpc route, I think mikayla, wanted to know. If would route to route delegation cover this use case, if our hdb route could delegate to a grpc route.

B

um I think it could. My question would be: is this precedented elsewhere in the api? Like is route delegation well supported.

D

No because it's not finished so so, but it's just something to keep in mind that route delegation is coming. It is definitely a thing that we are planning on supporting. That's why it says ref and not gateway ref is that ideally, parent ref will be able to point to another route yeah, and so the intent here is not to allow arbitrarily nested http routes. Either it's only going to be one layer, um one layer ever so um interesting.

E

D

Because uh so I can speak to that from my contour experience, where our http proxy does allow arbitrarily nested route inclusion, and uh if you as you would imagine, building a graph is complicated. You need to worry about loops. You need to.

E

Worry about yeah there's a bunch.

D

Of stuff there, that uh limiting the the number of uh of um delegation delegation tops means that uh you don't have to worry about. You know, building a full dag of uh making sure it is a cyclic of references.

H

The current proposal, which is incomplete is that a route could only uh delegate or include a another route of the same kind.

D

Yeah so yeah, so that would that would that would actually preclude the the uh the case that uh makai was talking about there thanks jeff I'd forgotten about that it was in it was in the before times before I was away from us, so I uh lost the context.

A

Cool all right, uh good discussion, I'm looking forward to getting back into route delegation too. uh This will add a whole new perspective on that yeah. I'm back rob yeah.

D

Welcome back welcome back.

A

Cool all right uh with that, let's move on there's just a couple things I had in uh issue triage, but definitely I I just add things that are top of mind for me, but always feel free to add things to this list to the agenda as a whole.

A

um I haven't last week was crazy for me uh busy week on other things, but uh yeah. It looks like there's. One comment left on this particular uh pr of mine, but the these are the conformance tests that I originally had in that big massive conformance test. Pr, I think it was harry wisely suggested I split out all, but one conformance test. So this is the rest of them. uh If anyone has time either fairly contained now and smaller than originally.

A

So, if anyone has time for a review, I not worth going through on this call, but just want to highlight them. I, the other one. I I feel like we've gone back and forth on this because it's another gap- and I just I I would love to reach some kind of conclusion on this one. uh I I think I don't think there's a real, strong consensus either way on this, uh but the proposal is- and I don't think this person is on the call correct me.

A

If I'm wrong, I don't see them uh the uh the proposal is really simple. On its surface, it's to add section name. Well, it's to add name to route rule uh so very similar to gateway listener name. The idea of a section name word for attachment for logging, for whatever it might be, uh the gap goes into a variety of details.

A

I it it feels like I I don't know I I'm pretty hesitant to go forward with it, but I wanted to give everyone else. A chance to you know either start a discussion here. Any any kind of hesitation you have or any support you might have for this kind of addition would be welcome.

D

You know I've got thoughts on this one rope, so uh my thought here is: I actually don't mind the idea, but if we're gonna do it, it has to be required, um which is a breaking change, which means another api version, which means yeah. We delay beta. um The reason I think it should be required is that it's not.

D

It actually creates a lot of problems. If it's optional. um How can you look at you know? How could you pass the logs if you can have an optional name or you know, as opposed to like a section number, um what happens? I think rob you raised an interesting case of like what happens. If you give a name, that's just a number. How do you distinguish that from a section number um yeah, there's there's a few other things there that I think um yeah. I think you now.

D

As I said there, I think the options are either that we don't do this or if we are going to do it, we have to break compatibility and do a v1 alpha 3 and make it required um yeah. Because yeah I mean I, I think that it it. Definitely if there's a nice ux element to it that having this sort of thing be be a named like a list type map and named essentially a map of the rules rather than rather than a list, um is okay.

D

um The the one thing that that I literally just thought of that it does do is that right now, because rules are an array they are implicitly ordered. If we move it to like a list type map, then then you kind of lose the implicit ordering, um and so that's that would then mean that we probably need to add like a priority field or something like that to re-add the ordering.

D

um So so there's a, I think, there's a bit there's a couple of big trade-offs here. I appreciate that it's uh important, for um you know for this use for the use case that the the to using this thing that you know like being able to build configs uh reliably, but I do think that- and there are some ux benefits, but I think that it's pretty it's a much more significant change than it seems on the surface.

H

As mentioned in the intro to it, um I think this is one of the things that originally was part of the route delegation that I didn't think we would need, but the more and more I've been kind of working on it. The the more I'm coming to the to the view that we, we probably do, need to have a the ability to name a section, or you know something with a with the matching rules, so that we can be clear on where we're trying to delegate or include a rule.

H

Yeah, let's delegate or include a better rule, but a a route. Excuse me.

A

That's a good point. I I've been. It really depends on which way we do route delegation. So thank you for bringing that up. uh You know if we're doing route delegation or inclusion or whatever it is, I top down. You know route references, child, the rule name doesn't seem to matter it. It's just secondary.

A

uh If we're using parent ref, it seems very, very relevant as as you're describing right. You don't want to attach to everything you want to attach to a very specific part of a route.

D

What's that may may yeah.

A

Thank you. uh So oh that's tricky. I I am really really hesitant to enter. I so first off I I'm one who actually proposed something awfully similar to this back in the days of policy attachment. If you remember the policy attachment gap, I had a section name on gateway listener and route. Rule and feedback then was well hey. Do we really need it, and- and there was a good enough use case for gateway listener that wasn't for route rule, so state on gateway listener did not on route rule.

A

uh I I've since become convinced that okay, well, everything a gateway listener, is, is very much required because a gateway is a singular representation of something with routes we've said routes. Can any route can be merged with any other route so having a route? A route with multiple rules is nothing but a convenience it. It is not required for any any use case. Where you have two route rules. You can just as easily have two routes with a single rule each so that's that's kind of been what I've been pushing back on is well.

A

You can do this today and we could technically say for route inclusion delegation, whatever it is uh to attach to a parent route that parent route has to have exactly one route rule, or maybe another way of saying it when you attach to a parent route you're attaching to everything in that parent route, so make that a small route, if you, if it needs to be, uh I don't know uh those are just my thoughts. I'm really I'm really hesitant uh to push another release like push v1 alpha 3 if we don't need to.

A

But I do agree with nick that, if we're going to do this, it should be required and if it should be required, it requires v1, alpha 3. and the only time to do that is now we're not going to do that post beta, so yeah.

H

Yeah with the route delegation and stuff that that doing it, the top down where you know the the at the gateway or at the current route, you define it. um You know so that the parent route explicitly calls the child route. um That works one way but and we don't and you're right rob.

H

We don't need uh kind of a name for a matching rule there, the opposite model, where that we want to have it so that um from a a delegation of authority or or responsibility that I just kind of opened up this match to allow you know anybody from namespace foo to attach a route to it, then that's where I need to uh probably will need a name.

H

So it's if we want to support. Excuse me if we want to support both models. I I think we're going to have to.

D

H

I'm open to you know if we just want to say we don't want to support. You know both kept models of delegation and uh flash inclusion.

D

Yeah I mean I'm inclined to so contour http proxy has an inclusion functionality. It is top-down only. uh We have got a lot of pushback from people that it's useful for for the use case. Where you have what that does. Is it forces a human to be in a loop? So it means that you can't uh add more. You can't easily programmatically or, like you can't say, allow everything in a namespace or something like that um to bind, because you can't, because it's a it's a one-to-one mapping.

D

You've got to have each specific one is done, and that was done so that you can so that the person who controls the higher layer can control.

D

um But I think that we have better mechanisms to do that in the gateway api with the, because we've solved the exact same problem with the gateway to route binding, and so I think we need to take the same mechanism we use for the gateway to route binding and use that for the for the inclusion of delegation.

D

What that means is that that's a bottom-up, I think that's using the parent ref with a um so it's the the below thing uh refers to the parent one, the parent one has the chance to uh to say no by um by having like a narrowing specification.

D

um If we do that, as you say like having the name. So what you're saying is that you that you can't that we couldn't mandate? What rob is saying and say you if you're binding to the thing you bind to all the route or none other route, um and I think that is definitely worth considering, because what that that's, like a api level, push for people to keep their routes small if we provide too much functionality around um naming, naming subroutes.

D

What we're going to end up with is people having thousands wanting to have thousands of um of route of route rule entries in a single route. You know, which I think is, is an anti-pattern, it's a smell, um and so like the you we want to. We want to encourage people in as many ways as possible to have more smaller objects, because they're easier to understand they're easy to pass the easier status for the new.

D

It's it's um yeah, it's better! It's better api design. I think, in terms of kubernetes api, to have more smaller objects rather than large objects that have a lot of things. What it does mean, though, is that, if we're going to have lots of more smaller objects, with references between them- and maybe you know- is that, then we have to have really clear rules about what happens when you have about conflict resolution. What happens when you have the same thing? Do you merge them? Is there a conflict? Do they get projected that sort of thing?

D

And so there's there's a lot of fiddly stuff that we did for. Like rob said, we've got a big list for http route of what happens when you have http routes that match on various criteria which one which one goes first, which one you know when do they get rejected? Basically we're going to have to repeat that process for each time.

D

We do this sort of thing, hopefully after we've done it a few times, we will get a sense of like this is the right way to do it and it should get easier and easier, but it feels to me like the more that we make, that the gateway to route and route to route uh relationships similar the easier the api will be to understand and use if we make the route to route relationship different to the gateway to route 1.

D

In that we are building like a big headache for ourselves, down the road um and so and all of that that higher level stuff all then flows down here that you, I guess what I'm saying is I'm kind of I am actually increasingly in favor of us. Sadly, I don't want to do it, but doing just biting the bullet. Doing a v1 alpha 3, making this making the thing named a little bit there you to some extent, but then on the other side of it is we have that then encourages that other problem?

D

So then I'm not in favor of it again right so there's. I think there's really strong arguments for and against here and that we need to be very careful with. It is what I'm saying.

A

Yeah, just I mean just from a I mean this is this is not the the best argument for anything, but when I look at our our examples right now, if I imagine that every route match every route rule we have would have to become named, that adds.

A

You know a great deal of verbosie to the api that I I can imagine, 90 percent of users 95 of users would never benefit from uh that. That's somewhat the case for gateway listener, but again I'd go back to this idea. In my mind of for every uh one gateway listener, there will be 10 to 100 routes right and within each route. Multiple rules, so we've gone from adding a relatively minimal cost of a few times per gateway.

A

uh To you know, every single time you define a route rule, um I I'm really and nick you raise a good point too, of allowing for nesting inside routes allowing for name references inside routes is, by definition, going to lead to users creating larger routes than I think we really want.

A

I I the more I the more I think about this. The more I regret having a list of rules inside route at all, right like I it you know a list within you know a list that could just as easily be accomplished with multiple resources.

A

uh Maybe we maybe we messed up there, I'm not sure, um but but doubling down on that and saying hey we're going to create we're going to add names here. So you can reference. This list item easier feels like we may be going even further in in that direction.

A

I I I don't know I I'm really, I'm really hesitant to do a v1 alpha 3, but I also know we. We should not make this decision lightly, whatever we do and we need to at least have some kind of rough picture for how this for what we want to do for route delegation, I'm not saying we should have a get merged or anything, but we should make sure that whatever decision we make here is not going to be something we regret when we're planning for route delegation so like like like.

A

I said uh this seemed like a really you know: you're adding the proposal was to add an optional name field on route rule and see. Well, this should be so straightforward and no it's not. Unfortunately,.

D

Yeah yeah, you turn over the rock and you find like three other rocks. You've got to turn it right like yeah, so I actually think one of the things we need to do rob. Maybe your eyes should do. This is um do a quick check back in upstairs upstream with uh api machinery and say you know, hey we've kind of got a decision point here where we're going to push people towards having more smaller objects or larger objects with more things inside them. Do you have any feedback about which is better?

D

I suspect the answer is more. Smaller objects is better, but I could be wrong, um and so you know the upstream api machine and we will have had experiences about this um and that that should inform our decision.

D

H

Point I'll try really hard to have the uh the full route delegation proposal um ready for next week, not to say that it may not undergo significant changes as we work through it. But uh at least it'll be a stake in the ground.

D

Like I think, I think what richard's done here today is a great way to do it like bring a draft that has enough detail for us to talk about um and and and go away and generate some action items for you. I think that's probably better than a more complete proposal that you then have to redo significant chunks of yeah yeah. This has been great.

A

And I I agree, I mean I think, what what we don't need to have all the details nailed down. I just would help a lot with this decision to have a direction for route delegation we've kind of been flip-flopping between a top-down or bottom up.

A

I tend to agree with nick that, and I think you jeff as well uh that bottom up is probably a better means and more consistent with the rest of the api, uh but again just would be good to see that, and thank you thank you very much for uh taking some time to uh continue on that.

H

Yeah yeah, when I said complete, I didn't mean like code it out and everything just the covering, uh hopefully covering all the things it needs to cover.

D

Yeah logically, complete yeah yeah yeah yeah yeah. um I also wanted to take a minute just there to say uh thanks to richard for a really awesome proposal that uh you know had just enough detail for us to really dig our teeth into and generate uh generate. Some great discussion really appreciate it. Man. Thank you all for the feedback.

A

Yeah, this is great. I've got to run to my next meeting, but thank you, everyone for a great discussion today and we'll talk to everyone next week.

F