Kubernetes Network Gateway API Special Interest Group, 19 Dec 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG Network Gateway API meeting for 20221219

Description

SIG Network Gateway API meeting for 20221219

A

Okay, thank you, let's just redo that one more time. So this is a today because of the holidays. We decided this would be kind of a town hall just open floor, nothing on the agenda to start with, but bring any agenda you'd like kind of meeting, it's still governed by the kubernetes code of conduct. So please just be nice to one another, um and with that we don't have an agenda to start with so I'll just shrink. My diet, coke and people can start talking.

B

So yeah, like honestly, this is the whole. The whole point of this is to yeah, make it so that uh Shane, Rob and I aren't monopolizing this for a change, but, like I, don't know about anyone else, but I'm perfectly capable of monopolizing the discussion. If that's what you want, uh but we figured, it would also be called to like let anyone ask questions they want. They've always wanted to ask, but we're always too busy talking. So let's give you all a few minutes to uh to uh figure out what you want to say.

B

That's what you want to do.

C

A

Have a drink and remember: there are no dumb questions. No such thing.

D

Okay, maybe I I'll start off with a simple, dumb question: I know this being talked about before I just want to get a refresh um this week. This is regarding to um this. um You know you have gateway, then you have HTTP route.

D

um There's a policy to say you know from the cluster operator who normally owned the Gateway. They issue policy who can associate to Gateway and then somehow let the HTTP route to you know. I think we talk about I'm, just curious. Can you guys reiterate this? Maybe the example, you know I think I was trying to you know. This is my question clear.

B

um So so you're talking about policy attachment and how it like, how you attach a policy to the Gateway and how that works with the H2 Bureau does that.

D

Yeah sound example, yeah exactly you know, say, for example, you don't want to like get the Persona of HTTP route owner. You don't want to you know. Sometimes they want to associate to every single Gateway. You may want to say Hey, you cannot. You can only associate to this. You know for, for example, death. What I was Gateway or you can associate to these. uh You know building Gateway or something like that. So.

B

Yup, so well, so that one that that example, that you just used um the intended mechanism there is for people to use uh allowed routes on the gateway to do that. So the idea there is that you set the allowed routes to http route or something like that or but then there's a namespace uh part there.

B

So you can say anything in these: namespaces can connect to this Gateway and then, if someone tries to connect a HTTP route from outside that gateway, then it has the the it has to fail um so like so from outside that um uh outside those set that set of namespaces. So you can say only allow billing uh things to connect to the billing Gateway from the billing namespace, for example, um and then only HTTP routes that are that exist in the billing namespace will be able to connect to the building Gateway.

B

um Sorry, um the um yeah so I think that um that that's probably one of the uh that part is sort of for the routes, but for policy attachment. um The policy attachment I think is designed to have settings that uh you may want to configure, sometimes on a like, or that makes sense on a per HTTP route basis, but you also want to be able to default it or override it um at a higher at a higher level in the um thing, so you have, uh for example, you have like you know.

B

Normally you have HTTP routes here, Gateway here Gateway class, here right, and so, if you say wanted to, uh if you just find a timeout policy, this is the one one of the things that Rob and I used as the example, because it was actually really hard.

B

We couldn't make a timeout policy, that's like sits in the actual Gateway API, but um so, for example, you want a Envoy timeout policy that um that lets you set uh say um nobody can set their client time out for their thing to Infinity, because this is a shared proxy and you want all clients to have to time out. At some point, you can't just keep their connections held open forever.

B

um So a way you could do that is, have a timeout policy object that has a default and an override. In this case you would say um you know some sort of override that says you can't set this value to Infinity um and then, if you attach that to the gateway, then the part of the uh uh contract for uh policy attachment is that when you attach something to a Gateway, the settings that will apply to all HTTP routes that attach to that Gateway.

B

So if you, if you've, got 10 HTTP routes attached to it, then none of them will be able to and you use an override. Then none of them will be able to change that. That particular setting does.

D

That make sense yeah yeah yeah. Thank you. So, regarding the allowed, Association is the name spaces. Probably the top people normally go with it. Yes,.

B

Yes, so there's there's only two pieces of information that we have in our ladder routes at the moment. That is the kind, so the group version kind uh and the namespace so, like you can say, HTTP routes from the billing, namespace or TLS routes from the billing namespace or both. But you can't say uh TLS routes named something from the village neighbor. So you can't like be really specific. You can only sort of do things by the kind and the namespace got.

D

It does a HTTP route site need to do something. I, remember hearing something like it's. A mutual you'll need like a two-way. Yes,.

B

So the HTTP route needs has a parent ref field, where the HTTP run is to say I want to connect to these gateways. It's a list, so you can have more than one Gateway. A HTTP figure out can connect to more than can attach to more than one Gateway, but each of those gateways has to allow that HTTP route.

D

To do it, okay, okay, got it! Thank you.

B

Yeah uh and in fact the the multiple gateways thing is also useful for, if you're say provisioning, a new Gateway and you want everything to be available. You can just connect all your HTTP routes to two gateways and know that the HTTP routes are accessible on both of those gateways and you can shut one down or remove it or something like that. Sorry Rob. You want me to.

E

No I was uh unrelated, so you, you did a great job, answering that uh thank you. I I was just uh going to say. Shane was looking very festive. I appreciate the the festive whatever background, uh but I.

E

Also I I noticed that there are some people, some names I, don't recognize that well on the call and I I may have I may have I know I arrived late, so sorry, but if this hasn't already been offered, if anyone is new to the community- and you know willing to give just a short- you know 20 second introduction or longer about what interests you in Gateway API. You are very very welcome to do that, no pressure, but we always like to uh learn more about who's interested in Gateway API. And what brings you here.

F

Hey sorry, I was looking for a mute button. If you guys don't mind, I'll just introduce myself right now.

E

Yeah definitely.

F

Hey so my name is Alejandro um I used to work for d2iq, um currently looking for new rules um interested in networking uh in general, so I'm kind of interested in this new project, uh the Kong has blicks I looked right, I thought it was awesome, so I'm gonna be very trying to learn more things about the API Gateway, the role-based stuff. I. Think it's interesting! So that's why I'm here.

E

A

Yeah welcome rob you might have missed this at the beginning, Alejandro and I used to work together. Oh.

E

Okay, thank you. I I heard the name and I thought. Surely there's a overlap there, but I wasn't sure, but yeah.

A

Years ago,.

E

A

Anybody else I I'm, looking I'm trying to think. If I don't recognize anyone I'm not.

G

Going to call out.

A

Anyone am I going to put them on the spot, but please do feel free to jump in and just say hi, even if it's just a high.

H

Hey uh I guess I'll go. um My name is Hamza um I'm over at Ambassador Labs uh working on my Ingress Emissary Ingress and debate product Edge stack um kind of just been hanging around here, just to see. What's going on in Gateway API land I think I briefly met Rob at kubecon like this Flynn and um doing like that break room session. So um just just uh just here to see what's going on.

E

I

I

So I work on a custom controller, um it's one of our products and we might be integrating uh the uh API Gateway, so I'm interested just to know. What's going on and see how we can integrate it into our project and it's nice to meet you all. Yeah.

A

Cool nice to meet you too yeah.

B

Nice to meet you too,.

A

Did you have any specific questions baby about like your implementation or you're, currently doing like Ingress on kubernetes.

I

Yeah I don't have any specific questions at this point. I'm just uh you know trying to get an intro into the API Gateway. Maybe I'll have more questions in the future sweet.

A

Welcome awesome.

B

I would uh I I hesitate to uh correct you on your first first time out, but uh one thing that is really important is the terminology um game API game. Api implementations are generally API gateways, but Gateway API and API Gateway can be very, very different things. So the ordering of the words is really important. I'm, sorry to be a stickler but I'm, a one of the things you've probably learned.

B

Pretty quick is that, like I am a bit of a stickler for language and so Gateway API is the name, and it's really important that we get it that way around sorry, yeah.

I

Making sure that I got the uh you know, naming convention right, yeah.

E

And and I should apologize on behalf of a greater Community for choosing an awful name. uh It is very, very confusing and I.

A

Should have named it service apis, right, Rob, oh.

E

A

That was worse man here.

E

A few a few names here and uh okay, every you know: everyone calls it API Gateway uh and it's uh you know, uh you know whatever um so yeah.

A

Inside inside joke service, apis was the previous name of the project. Yeah.

H

A

Had to rename it that's what it was named when I was at mesosphere actually and started lurking the the meetings and stuff yeah so.

B

F

A

It was more confusing than Gateway, so.

E

B

This is actually exactly the sort of thing we should be doing. Oh.

A

It's a legit question, it's just it's the the context and the history makes it funny for me, it's like yeah.

E

A

Did so much effort to like change it away? Yeah.

E

Anyway, this API started as Ingress V2. This is going to be the next generation of Ingress, but that wasn't generic enough because we wanted to be more than Ingress. So then it became the next generation of service apis.

E

But then we inexplicably called it. Okay, it was service, apis, I think is what's the name of the project, but the pluralization got really annoying to work with, and we couldn't just call it service API, because that was already the Upstream thing and it was very easy to confuse with kubernetes service the API as opposed to this new thing, uh and the Gateway was kind of the front end to our set of API uh naming is hard, I I don't know, and we clearly uh have not been great at it. So apologies, but.

A

One thing: one thing that we got was I, think service and API in any search bar anywhere was a mess.

F

A

And API is a little better. It's a.

E

Little but you do mostly end up with API Gateway products.

A

um But it's a little better modicum better.

J

H

J

C

Like it doesn't.

H

Help but there's another thing called Gateway API gamma, which is for like service meshes, which.

B

Is yeah in some ways.

H

Orthogonal to an API gateways, yeah.

B

A

F

You for bringing that up, yeah.

A

That's our project by the way, if, if it wasn't clear to people um it's a sub-project of Gateway API for doing service mesh with the apis, we already have for the most part. Gateway is not a part of that and not not that it never will be or anything. But it's just it's more focused on using the routes right now, and so that can be very confusing if you're interested. However, that is a completely different meeting. There's uh that's usually on Tuesdays, so please do check out our community page and the calendar.

A

um That's a whole different subsection.

E

Yeah this is, this has been great to uh Mikhail. Actually, you are about to I mean, go ahead.

K

Oh I would just gonna ask if it's if it's now time to ask a question.

A

That's the whole. The whole meeting today go for it.

K

Hi everyone uh so I want to bring up the validation topic it's been already just as previously, but I just want to make sure uh if it's, if there's anything new there uh so in um there is a gap 922 about API version and then in that Gap there's a recommendation there about not trusting uh webhook and crd validation and there's also an issue 926 with a suggestion to make a kind of common validation, Library.

K

Well so my question is like if we don't trust, like the crd validation, so I wonder if any implementation implemented, uh like this duplication of crd validation, logic in name control plane this, it seems like also like a heavyweight thing like you know, importing like crds in doing that. So so I wonder if anyone actually is doing that or they right there. You write your own liquidation layer.

E

I I feel like Nick and I are about to say the same thing uh but okay, okay, but thank you. Thank you for raising this and I. It's been on my mind for a while, because I I feel pretty strongly that we, if we don't communicate this clearly enough and we haven't, uh there will be some future vulnerability, something something goes wrong in some bad way. With some implementation that that doesn't, you know, read the fine print.

E

You know this is hidden away in a documentation page somewhere, like the validation we have can't be relied on or you know it can be bypassed at a minute. Can.

B

Be bypassed is a better way to say than can't be relied.

E

On yeah you're right.

B

You're right yeah, so it's it's not safe to use it for like really super secure things, because so um even for CID, validation and stuff, like that, you can bypass that by doing Cube, catalog life force right, like you know, it's not it's not a it's not like a super hard requirement. It's there to help you, um but but it's not a super hard requirement. So that's what Rob is trying to say about it can't be you can't be relied on.

B

It's not that it can't be relied on like in general, it's that it's possible and trivially easy to bypass it right. So you can't use it for like super secure things. That's the that's the thing that that is trying to push I, think and.

E

And I think what you're getting at is right. Now we have webhook validation, which you can import into your controller and run uh and- and hopefully many are doing that today and that helps a bit. But then, if, if somebody by bypasses the crd validation well, you.

K

E

Good luck! uh It's it's! It's that's harder to replicate and I've been thinking for a while and I think Nick and one of the more recent PRS we were talking about. uh You had said something like along the lines of why not make webhook validation kind of a superset of so you're. You know it covers everything crds we have in our crd, spec, Plus stuff that can't fit in the crd spec um that that feels like it would address this, uh but I don't know Nick.

E

This is probably a good segue to whatever you were gonna, say: yeah.

B

Yeah so yeah this is, this is I. Think that's a bit of a bug bit for me as well. So the um yeah I think that it's 100 fair to say that the uh validation that we have can be bypassed.

B

You don't have to run the web hook um and so, like that's one of the things, that's a bit tricky and that's what that paragraph is trying to say is you know, hey, because you don't currently have to run the webhook to be conformant like you can kind of uh you know, things could end up in a weird state, so part of that is intending you to say, hey implementers when you're writing your stuff, be aware that you, if it's an enum, it's not a it's, not a closed enum right like there might be something else in there that you don't know about don't crash.

B

I

B

Have a default case: that's not just Panic uh in go terms, but um I. Think more importantly, the other thing that I would like to do about this is I want at some point us to have effectively conformance tests for um for the validation that invalid things will uh will not be accepted in some way either that they get rejected when you try and apply them, which is the which is what a validating workbook does or they get status updates.

B

That say this is not valid, um I'm, not 100 sure which one we want to mandate there if we want to mandate one or the other, that sort of still up for discussion a little bit, but I do think that it's a really critical part of our conformance that that any implementation that ends up because the idea behind performance is that eventually we want to have a process where the same is Upstream, where you can get essentially a badge that says: Gateway, API, conformance version 060 or whatever, um so that then um you know, then people can use your implementation confidently, knowing that due diligence has been done um and part of that I think.

B

Is that your your um your implementation needs to not be sort of able to be worked around by doing a trivially, easy things like Q cattle apply Force to, like put you know, put parts in that lets. You do Parts reversal and stuff like that.

B

um You know is a good example and one that's come up recently, so yeah I think um that that's sort of my rough thoughts on on that sort of thing, I think in terms of having the we've talked before, and we have that issue about having a validation, Library I think that making a cons like making it part of conformance will push people to want to have some common Library so that we don't all have to implement the same logic over and over again.

B

uh So, like that's sort of my two steps removed way of wanting to make that happen, does that answer your question? Okay,.

K

um Oh God, it seems, like the recommendations, still stays apply, so it's not not trust and not trust the input, and it's really it's be great- to have like a common motivation code but I'm I'm, I'm sort of you know. I tried to like reproduce this, uh bypassing validation and I. Couldn't just make it like. With this the ways I understand the validate force, uh the flag is that just it turns off the client-side validation and Cube CTL, but the sort of organization cannot be turned off because it's like an ABS server. Okay,.

B

K

B

Been a little while since I tried so like I could be wrong about that.

K

um And then the EPS server, like the person who like can create crds I assume it's like a super user. You know cluster out there. So so that's why I'm thinking about from the security it's still like? Is it like? uh You know how I.

I

K

K

E

Think it depends on how your implementation is deployed to some implementations of the API are deployed in the same cluster, and so you assume that a cluster admin somebody who can create a crd basically has full control over anything. Your implementation of the API can do anyways. Other implementations may be deployed somewhere outside of your cluster and may have greater access to resources than your your cluster itself has access to, and in those cases in particular, you have to be very careful about what inputs, you trust uh and so yeah yeah. It's it's understanding.

E

The deployment model and uh risks involved. I I would agree that if the cluster admin is the one who can is ultimately the person who can change or bypass a crd validation and that's really the the top of.

D

E

Know the line for anything deployed in the cluster you're, probably okay, uh but if, if your implementation is, you know, can connect to more things. uh uh Is you know at risk of some kind of confused Deputy attack where you're confusing uh the controller to Grant you access that you shouldn't have uh then I would be especially careful.

B

Yeah I think some of that came from like both Rob and I have had experiences with Ingress controllers, where it's pretty easy to do. Yeah unless you're really careful and you really look at the fine print. It's like it's actually really is more straightforward to do some. Some really nasty confused wpe attacks with Ingress controllers.

K

Well, thanks thanks.

K

This is. This is very helpful.

B

Awesome great: that's what we wanted for today. Ask those questions so that we can give you all the background context that you know is really hard to write down.

B

And uh thanks nikaya for uh the the alternate name spreadsheet. That is a real blast from the past.

E

Yeah I I completely forgot about that. It took me a while to find the results of that uh that poll and in retrospect, I I almost wish we'd gone with the routing based names, um but I know a router is an overloaded term too. So I don't know. um Naming is hard yeah.

B

Router API I, don't know how much better that would be to get API yeah. It's like uh like I, always say, there's two hard problems in Computing right naming case invalidation and off by one errors.

A

People would end up at open, wrt documentation when they were searching for us very.

B

A

F

Like guaranteed.

E

Maybe you know I'll, anyone is free to bring anything up, but one of the things I'm curious about just open question to people here. Are there things in the API that uh are either unnecessarily confusing to you? You know like I, I would imagine at least some of the people here on. This call have started to work with this API more recently than myself anyway.

E

uh Are there? Are there parts of the API that really felt like rough edges or confusing to understand uh when you, when you first started looking at it that maybe we could improve or, or you know, describe better.

G

um Can I can I ask a question: I I, don't know if it's regarding that, but um we're trying to figure out conformance tests like how how do we fulfill that and I just started looking at the documentation on conformance tests and maybe I'm not reading it correctly and I tend to miss a lot of things when I read it, but I, don't I, don't really see uh kind of like a template or kind of like this. How you do it.

E

Yeah, no, that that's a great Point, um the I think what we've seen a lot of implementations do, especially if you're Implement implementation is written and go as we've seen implementations you know, run the conformance test as part of their regular edweed test Suite. So.

I

E

Can import the tests and run them alongside any other test that you might have uh for your controller implementation?

E

I think the only thing that we have clearly or well that we have in the documentation right now around conformance tests is how you can run them with CLI. So just yeah.

G

E

You know whatever, and that may not be that use you know. I I, don't know it's hard to know your workflow, but we definitely haven't clearly documented that a lot of people are just importing these tests and and running them in ewe, which I think has been a better experience. Is.

A

That not what you're doing.

G

um Well, I, don't know if my uh implementation is up uh to to par yet to to run the performance test yet but um yeah I, yeah, so you're.

C

Saying we can we.

G

Can import it and then run it as part of our pipeline yeah.

A

Yeah, that's what we okay like at Kong. That's what we do for all of our implementations. We just like. We have a conformance test suite and then it's literally just loading the go code, like literally loading the go tests and running it on our implementation. You can do that, but we really do not want to have that, be the only like the only way that makes sense.

A

So if you do get in a position where, like you feel like, you need to be able to run it from a non-go or just you are go, but you don't want to have to run it in that way. We should probably smooth those edges, because people should be able to come in this with any implementation that is conformant and not have to do go. Of course.

A

E

Yeah and one thing I'd say, is I- think there's a PR related to this, but I think many implementations of the API kind of built their own way to skip tests. Basically, you know because uh there are a lot of implementations of the API, especially in the earlier days where they were conformed with you know, maybe half of.

I

E

Could pass half the test, but they didn't want to run the other half because they knew they hadn't implemented this or that or whatever I'd recommend that as a starting point, uh there's been some work to try and bring that into the into the conformance test, definitions and sell. So you don't have to you know, do it yourself, uh but if you are looking for examples, I think.

E

Well, you know any I think just about every implementation out there, but I'll highlight you, know: istio Kong, Contour or Envoy Gateway many implementations are running conformance tests as part of their CI and I'm sure would be happy to direct you to how they're doing it.

G

Cool cool thanks.

G

We we had another question. Someone asked um regarding visibility of resources across implementation, so so, okay, so with the Gateway class that basically controls I, guess an implementation of the Gateway API right. So I'm thinking you can theoretically load two or more implementations on the same cluster like let's say you load a Kong implementation or you load and istio implementation on the same cluster. And then it depends on what, like, let's say, a gate. You create a Gateway resource and then you reference in Gateway class.

G

So let's say you get you in your gateway, you reference the Kong Gateway class and that and so then then, let's say and then that Gateway would obviously choose the Kong implementation would say. Oh this is for me now I'm gonna, you know provisioned the Gateway resource and all that stuff yeah or if the istio sees it. It says. Oh, this is for me I'm gonna, I'm, gonna process this, but but does that get into the problem where you have different implementations, seeing the same object that that may not be for it, though,.

A

C

Should filter them.

G

Right: okay, but that doesn't.

A

G

A

Them but I mean they can see them, of course, but they'll usually.

G

A

Them, based on the controller name right.

G

But they can still inspect them right, yeah.

A

If they're the are back permissions to do so, they can look at them. Yeah.

G

A

What's the best.

G

Way: okay, so with the best way to kind of prevent visibility or to to control with our back.

C

B

A

So the sky's, the limit otherwise yeah.

B

Yeah, so practically practically most controllers. A lot of the time get read, get read permissions on gateways everywhere, right, like it basically cluster wide to read permissions on gateways. One of the reasons for that that's actually kind of okay is that we've specifically designed a Gateway so that there's nothing secret really in the in the Gateway object.

B

So it should be reasonably safe for a controller to read or gateways and then throw away the ones that it doesn't care about um secure um and that's what the Gateway classes for Gateway classes intend to be to be a bucket that lets your controller, be like you know: I, look for all Gateway classes that have my controller name.

B

um You know I keep a list of gateway, gateway, class names and then, when I'm looking at gateways, if the gate, if the Gateway class that references isn't one of mine, then I just throw it away, I drop it on the floor, um and so when you're using controller runtime or something like that, it's pretty easy to do that. You just you when you get a thing you one of the first checks you do is to say. Is this relevant to me? Is this in scope?

B

For me, if not, you know do nothing, no of, um and that's the that's sort of the usual way to do this so and like to some extent. The same is true for HTTP, although the the process in that case is more complicated, because you've got to figure out if it applies to a Gateway that you control well, there's another step down in the hierarchy, but uh yeah it's a broadly similar. Does that make sense.

G

Yeah I mean it makes sense that you have to do that cross reference, the paragraph and then the listener allowed routes and all that checks right, exactly I mean, but the implementer can still like with HTTP route. You can still kind of see the path configuration right like the yes yeah, so you can see that a little that might be a little bit sensitive, um I I can see how the Gateway only references, the secret in the namespace, so that's kind of hidden.

G

You know with our back if that Secret in that namespace is not accessible from that uh implementation. In you know, right.

E

Yeah I would say it's entirely possible to segment your controllers in such a way that you could Grant. You know like Grant it access to a subset of namespaces and then tell users of that implementation. They can only create resources in those namespaces um I, don't know how it really depends on what you're putting in your routes and how sensitive they are.

E

I think most of the Assumption we've been coming from is is like what Nick was saying that the content of your routes and gateways, at least from our perspective, is not likely to be highly sensitive.

E

But you know every model is different. It.

J

Isn't just a fact of a route sensitive like knowing that a particular service is being exposed in a particular way, yeah.

A

There there could be if you have a bad actor within the cluster, then. Theoretically, some of these apis, especially between different implementations, could provide some telemetry yeah, which is what I.

J

Think I think that that's kind.

A

Of not specific to Gateway API, though I mean once you have Bad actors in your once. You have a bad actor in your gateway. Api implementation I think you have potentially much bigger problems than they can see. Other implementations routes, yes,.

B

So the I mean I. Think one of the problems here is that a Gateway API implementation needs. uh It needs quite a high level for for the full magic where you could have very Dynamic, Gateway provisioning, very Dynamic, HD period provisioning.

B

um You need the the implementation needs to have very high cost of wide privileges. For the for the thing to fully work, you absolutely can make it you can cut down the rbac. You can give it restricted, R back, so that only see stuff in some namespaces, but that means it only sees stuff in those namespaces and then anyone who wants to interact with it has to have access to those namespaces as well right.

B

So that's the trade-off that you're making by doing that, is that you, but like the fact that the you know the Gateway API implementation is effectively part of the system right like it's. It's part of the infrastructure of the system and you need to treat it accordingly in the same way that you would like you know, a mutating webhook or something like that, like it's part of your system, implementation and if anyone ever pops it like yeah, you like it's you're, going to get the access that it has.

B

As for a lot of people, the way they want the implementation to work is they want people to be able to create gateways reasonably arbitrarily and they want to be able to have those gateways like pick up TLS secrets from multiple namespaces or like an arbitrary namespace, in which case your implementation needs to have cluster-wide, read access to gateways and cluster-wide read access to Secrets, because you can't split secrets on secret type. Right: cluster-wide, read. Access to Secrets is often like very scary for people and that's completely Fair, because it is very scary.

B

But if you want to get the full sort of maximally dynamic experience, there is no other way than doing cluster word access to Secrets. The only other way to do it is to segment things off with our back, and then you need to have the secrets, leaving certain namespaces and only certain manage spaces.

B

um Part of what we're trying to do with reference Grant is to make it so that um users don't need to have cluster-wide access to secrets to be able to use Secrets, but your implementation will need to have Costa white access to Seekers to be able to see those secrets to be able to read them to be able to configure them in your underlying data plane. And so that's the yeah.

B

There's there's a there's a bit of a fine point there, but like yeah, I, think the fact that this is like Network Plumbing, like means that it's going to end up with a very high level of access to your system, like it's very difficult to restrict the access too much. Without significantly impacting how Dynamic your users, the experience your users will get, does.

A

That make sense.

A

Curious to hear if this is coming from more of a user or operator perspective or an implementer perspective, it's an interesting topic, but I am curious.

J

On, like the the perspective, I I mean I I, mostly work with service providers like Communication service providers, and um you know they're targets of you know. Espionage right and they've got a lot of security uh concerns, and so you know like and- and you know like, if you, if you ordinarily, you would expect that if you're implementing one set of resources like a Gateway class for one set of uh apps and a different one for a different set of apps, but they wouldn't be able to see what the other side is doing.

J

um But that's kind of not what's going on here right because and you.

A

Can like that's like you.

J

C

K

H

Implementation.

J

A

Ability to say by default the default like customized deployment, we'll just let it see everything that any other implementation could.

J

A

But that is, but some users will actually restrict that with customize like they'll have their own layer on top of that and say nope. These things can only see these namespaces like that.

J

But it also legitimate.

A

Thing, but it's not a natural, it's not a natural part of the deployment, it's something you have to choose, but.

J

It also means that, if somebody has a bug in their implementation, they could conceivably start operating on like routes that they shouldn't be right or something right. If.

A

You don't take the proactive measure of like how actually, if.

J

A

In a situation where you are the target of Espionage and you're not actively like building your deployment in such a way like forget the software for a minute, but just restricting that software in your deployment to the minimum thing it needs principle of least privilege, then, yes, you will have that that dilemma.

A

um I mean you'll still have other dilemmas, even if you do restrict things down, but that's the whole point is to get it down to the most narrow Focus possible right. So I don't know that every implementation does a good job of this. But there are some implementations out there.

A

That can let you kind of do this restriction so that if that implementation became um compromised, it would at least not have access to say other name spaces than the one for it, but there's still obviously danger there even in the name space, it's in um there's some I, don't know of any implementations that necessarily do this today, but there's some architectural things that you could theoretically do um to improve this. That I've actually actually thought about before, but not implemented.

A

Yet, where you it's still, it's still not a uh Silver Bullet kind of situation, but if you actually can make the actors which are responsible for things like Secrets, a completely different actor with a different set of permissions and stuff like that than the like main controller, basically a micro service or architecture, as opposed to a monolithic like Gateway operator or Ingress controller. There's some things that you could potentially do to reduce the the damage or the scope of an attack even more.

A

But at the end of the day what Nick said is kind of ultimately true I mean it really boils down to like what something somewhere is going to have to be able to read the secrets yeah so to.

B

Be able to configure that Donald playing something's going to read the secrets, I would say, kubernetes at large is not great at doing that sort of like really like really hard multi-tenancy inside a cluster right.

B

Like you know, it's kubernetes multi-tenancy is really pretty soft like if you've got access to one thing, it's like there's absolutely things to make it more difficult, but it's it's very difficult to make it impossible to like privilege, escalate out like around things or like things, things kind of expect that you have like reasonably broad level of read access but may, but a lot of your right access will be limited like that's. A pretty common assumption is that controllers that are running as part as part of the system, like I, said, have read.

B

Access to everything that they need in every namespace in in the cost is a pretty common. Like expectation, um you can absolutely, as Shane said, you can absolutely change that by when you run an implementation, you, the r back you granted, can be limited to a number of namespaces and then, when the API server, when it calls like list on the API server, it'll only see the stuff that you've given it access to. So you can absolutely sort of segment things that way.

B

The problem with doing things for stuff that kind of expects to be system-wide is that you're effectively building like a mini system of that sort of set of namespaces, and then everything needs to use our back that matches that set of namespaces and so you've got a sort of big admin overhead to ensure that nothing that is using anything inside that set of namespaces reaches outside that set of namespaces.

B

Otherwise you may as well just give it access to the whole cluster, because once you go out a little bit and it all doesn't match, then then you've got the way. The Escape patch, if you said I'm, saying and so like I, think a lot of people who have tried to do hard multi-tenancy for clusters, sort of end up landing on the idea of let's have smaller clusters, one for each tenant.

B

um You know like if you, if you really want to be like that's the that's the hardest multi-tenancy you can get.

E

B

I had an interesting.

E

A

Ahead, no, it's just doubling up on that. I see that quite a bit with some of my customers like they'll end up making the the actual they won't bother with namespaces it'll, be a cluster that'll, be the the the area of which they'll like do a specific application, and that's easier these days than it used to be like a lot easier. You know. What's really funny, though, is like five. Six years ago like when Ingress was really new I feel like every implementation just had cluster admin. It was like the default. Wasn't it like?

A

Everything was just cluster admin like six years ago, but so we've got. We've come a long way.

E

I had a really interesting uh comment: uh conversation with Mo from cigarth about this and and related problems, and one of the one of the areas that I think is interested interesting to both of us and I. Don't think either of us have time to lead it, but if this is something that is interesting to you, um there is room for another kind of authorizer in kubernetes.

E

So if you are familiar with how kublet works today, it relies on node authorizer to granted access to the resources that are running on that node. You could perchance have another kind of authorizer that could follow a reference tree and understand. This is a thing that this class of resources should have access to this top level thing and I'm going to Grant you access to anything that ends up being attached in the tree to this controller right.

E

uh That requires a generic enough implementation and API that you can follow the tree from the top to the bottom. But if you can do that and I think Gateway API, it could be done with you could build a custom authorizer for this very purpose.

E

um This is an area of significant concern. uh If you follow kubernetes security, uh you notice that on occasion we have cves tied to implementations of Ingress and probably eventually Gateway API uh over time and and you think about it. uh We are running applications that are Exposed on the edge which usually have access to all secrets in the cluster.

E

um You take that combination of things and again you know, be very careful as an implementer to be as cautious as you can, but also there's some room to. If, if there is, you know a problem in your application, hopefully you're only exposing the minimal set of things that you need to see, not everything and a better authorization model, a new authorizer or something like that would be a good first step towards that. There are intermediate things that we can do today. You know the cross namespace secret references actually help with that.

E

You can say all my certificates for this specific implementation are going to live in this namespace and I'm only going to get give this controller access to that namespace and nothing else. You know there are things you can do today if you're a security conscious but I. You know everything is kind of makeshift until we can build a new authorizer to really do this properly. That is a a longer term effort, but there's definitely interest from cigar.

E

um We just need people with time to do it.

J

So I mean you guys, keep talking about having access to all the secrets of it is that is the secret access purely about certificates? Yes, there are other Secrets. Yes,.

B

Yes, guest certificates, basically forgive API.

H

It's a certificates.

J

Yeah, so if the certificates are being loaded.

B

Oh I was just the same but certificates. The problem is that certificates are generally stored in Secrets inside kubernetes, and you can't get any more fine grain when they are back like in terms of type right. You can't have unless you make another thing, that is a certificate and not a secret like you can't get any more fine grain than also then all secrets in this namespace. That's the most fine grain. You can get read access to all secrets in this National Space is the finest grains you can get.

B

You can't say all secrets of type TLS certificate right like unless you make a separate, a completely separate type. That's how the type system in kubernetes works. Unfortunately, so that's that's what I mean by all secrets is, like you can't say, only significance and not like you know, environment variables that are used for pods right, because there's no distinction in the type system between that.

B

Does that make sense.

J

Yeah I I think so we've been talking about actually not getting certificates as Secrets, but loading them via grpc and storing just in memory. uh If we do that, we need significantly less access, then right.

B

Yes, yes, and so to make that work, though you'll still need like The Listener, TLS config is going to need some way to know which secret that you're loading into memory. It needs to use for that specific listener right, so you're going to need practically. Probably what you would need to do is to have a CID that says, like an in-memory search is the name of the CID or something like that and then listener.

B

Instead of having a reference to a secret, you have a reference to an in-memory cert that that tells you the name of the in-memory search right and that you are that you retrieve in some out-of-band way right, so that the API has the spot for to. Let you do that. It's just not going to work as seamlessly rather you're going to have to build some extra infrastructure around it.

J

B

And I mean that sounds like a great idea to be honest, that sort of thing for like, if you want to make sure that the the controller doesn't have access to those like the the controller, doesn't need access to all secrets in the cluster. Then the way to do it is to make it so that the TLs secrets and needs are not in the cluster right like that's effectively.

B

The the simplest way to solve that problem is to make it so that the secrets that it needs are not in the cluster and then all you need to do is to build some sort of naming scheme that lets you pick, which secret out of all of the secrets that it has access to in some other way. It picks. um For that specific.

B

um You know which soda picks right. Okay, thank you yeah, and that's what that's why in The Listener, we left that as our um it's not like secret name. It's like you. It's like a certificate name or something like that, but there's a good version. Client selector, that it's not just the secret. It defaults to secret, but you can choose any other type of resource. You want.

J

ah That I hadn't, noticed I was actually gonna, suggest that that we shouldn't be tying specifically to passing certificates as Secrets um right.

A

Okay, excellent! That's the thing we don't do.

J

A

Is that something, then uh what implementation are you working on? If you, if you can say, I'm, just curious.

J

Yeah so so I'm I'm at F5. We have a thing called the service proxy for kubernetes spk, and it's um it's. It's really. uh It's focused currently on service providers and launching five 5G, and so that's why, before I raised like the issue about egress, which is one of our big issues, um you know how we manage secrets and the sort of skeptical um nature of our customers around using secrets uh is another one of the big ones.

J

A

That's interesting, I I, don't not to put you on the spot or like put any work on you, but in.

H

A

um No seriously like seriously don't mean to so: please just take it with a grain of salt, but um I think we would be happy to host a blog post about your experience when you kind of get further along in this. Like yeah. We have. uh We have spots for like the Gateway, API, blog and stuff, like that. This would be kind of interesting, I, think for other implementers and users of Gateway API.

A

Just to hear about your experience with this, because I think up until now, there's not been a ton of concern, but there will probably be a lot more well.

J

Yeah and let me tell you that I have had um significant, like tier one, um early adopters say to me: Phil, if you think Gateway API is what we're going to use in the future for 5G I will force all the vendors to comply to it.

J

um So all right, you know yeah, so so we we need a couple of additions and again I'm super late on creating a document around egress. But um if we can cover just a few of the sort of service provider 5G, um you know gaps uh I think we can. We can have this be the way that people launch 5G awesome. That's.

A

Pretty sweet yeah.

J

A

Well, we're here to serve, let us know how we can help you.

J

A

J

Behind on the document about egress I,.

B

Got it yeah we're all behind an adult inside? It's like we are join the club, um the uh I think one thing I did: I did I just went and looked up the exact um thing uh in the in the spec uh and so um yeah. We have like a there's. A strike called secret object reference, which is where you currently store.

B

Like you just put, then you can just put the name of a secret because we default to group core kind secret um and there's also an optional namespace filter there as well so, but that that could absolutely that does not have to be. We specifically made it so that does not have to be a secret. Yes, I just wanted to make 100 sure that we've done that right, so yeah.

B

That absolutely is a way that I would recommend if people are really conscious about really strongly feel that you shouldn't have access to all the controllers like this shouldn't have access to all secrets. The best solution is to just not put it in the cluster as a secret right like give it what it needs, as as in some other way it could be a it.

B

Could be just stored directly in a CID um I'd, probably not recommend that, because Secrets have special Protections in Cube about like only showing up on the nodes that need them and a bunch of other stuff like that. um What you.

A

Can do for crds just okay.

B

A

C

Would you can add that, but yeah.

B

Yeah, you would need to jump through those Hoops for your thing, but I think the thing that you mentioned about storing the search out of band and having that record be sort of a a signal to retrieve the outer band cert from some other. um You know grpc standpoint or something like that. That sounds like a great idea to me for, for that sort of much more secure use case. 100 um Michaela asked in that chat uh that Dewey require TLS, can secrets and conformance test.

B

Yes, we do currently, so you know if we you know if this became more of a thing, then I would guess that we'd probably need to slice and dice the conformance test a little bit more to sort of be like hey we're testing the sort of common, the common cases of you, you just you store your stuff in Secrets um and then there'd be other conformance tests. That would need or there'd be some other way that you need to test this stuff.

A

Yeah there's a couple places where we have kind options where we default I think and the conformance test just assume the default where I figured in time, we'll probably have to have like an overlay like just let us know what you do instead and we'll map to it.

B

Yeah yeah, but that I think that's 100, a thing that we can do um that will again come back to exactly how we handle the. How do you submit a conformance report and what does it look like and you know what test did you Skip and are those tests okay to skip? And what overloads did you use and some other stuff like that um which is uh kind of similar problems to what Upstream kubernetes has with the kubernetes the Upstream kubernetes conformance so I'm, hoping that we can crib a lot from what they do.

B

Well, yeah thanks thanks Phil. That was an awesome. Awesome question awesome discussion, yeah.

A

Thank you, I think we can answer this last question in chat and then we are at time.

E

I can't read it hang on yeah, so Candace I think is actually the person working on this right now. So uh Candace is working on performance tests for TLS route. Looking forward to getting those in ASAP actually Candace, you may have a better status update than I. Do it's I've been meaning to review your PR.

C

uh I I just posted an update, so uh you can take a look at it again and hopefully I answered your questions and um and and it's good to go.

E

B

Thank you and thank you um notably. It is probably relevant because uh before we run out of time, um we are probably going to cut o60 in the next couple days, certainly before the end of the week. um But uh we do. One thing that's important to remember about our versioning is that we do reserve the right to add conformance tests in a patch release. So if that particular test doesn't make it into 060, we can still release a patch release, much quicker.

B

That only has you know extra conformance tests in it, and so that applies across conformance test. So improvements to conformance our patch releases only.

A

Roger that all right thanks, everybody for coming uh just a heads up, I, believe we canceled next weeks and the following weeks, uh just because we had so many people say that they wouldn't be able to show up. So we will be back um I want to say that's the second week of uh of the of the month. It should all be on the calendar if you need anything during that time, hit us up in slack and uh discussions and issues and so forth, happy holidays. Everybody.