Kubernetes Network Gateway API Special Interest Group, 12 Sep 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Network Policy API Bi-Weekly Meeting for 20220912

Description

KCSNA 2022 Weekly Planning for 20220912

A

This meeting is being recorded.

B

Awesome hello: everyone today is september 12 2022.. This is a meeting of the sig network policy, api subgroup to sig network. This is a cncf certified meeting, so please be nice to each other and yeah, let's run through what we have got going on today.

B

So first thing on the agenda was just a shout out that the for anyone watching this shout out that the new website is up and running on github pages. um This is a call for everyone to. Please take a look at it um file issues like let's add to it. Let's make this really cool. um We can add, basically anything we want because we own it and then next steps will be just to get it deployed to a kubernetes url via netify. So that's in progress.

B

um Otherwise the agenda was pretty short for today, but raul, I think, does have some stuff. He wants to talk about around fqdn coming off of the awesome presentation. He gave us last time around some of their data collection. So do you want to take it away? Rule.

C

Yeah sure so the first thing was something that I mentioned doing last time around and I um it took me a bit, but I finally got it so the goal was we pier and I presented some data that we'd gotten from our from our customers, and I was thinking that it'd be good for other folks to run the survey as well. Other cni providers, especially.

A

C

Have ftd and network policies just with the goal of understanding what customers are up to, and hopefully we can build something um you know useful for them. uh So I added a link to just a sample survey of like what we might want to circulate around. um I my goal right now was get some feedback from folks on. Is there are there any questions that are missing?

C

Are there any things that we should add tweak and then it would be good to um maybe have folks reach out to customers that they know are using this feature, but also maybe just put out a blast on sick network um and say hey just an open call in case someone wants to comment on this, we'll just try to gather a little bit more data, so we're more informed the main goal of this survey, just to like shape what my thinking was, is to figure out whether folks need the sort of wild card matcher, because that's that's a really big um feature that that has architecture ramifications like it changes what sort of implementations can work and what can't so.

C

The goal is to understand: are people using wild card matches or are they okay with just a list of fqdns um and also, I want to know how many people are also running service meshes or you know, proper l7 capabilities versus how many people are using very specifically, these still l4 policies? Essentially, um so that's the goal behind this survey, please yeah like like, send out comments or stuff. If you have thoughts on this, I actually can I present the screen. I might.

B

Just yeah I made you go host, go for it, I would have been, but I upgraded my fedora and can't share.

C

All right, let me let me try this.

B

Yeah we got it.

B

You're on mute rojo,.

C

uh How about how about now yeah there you go. Okay, I hate how it does that. That's super annoying yeah um anyways, the yeah. So here are some questions that we were playing around with this one is pretty generic just to get a feel for what people are actually doing with their fpdn policies. It's unlikely to affect the implementation, but just something interesting to note. um If you think there'd be value in more specific uh drop down, um you know narrow specifications here we can do that, but I was just curious.

C

What customers are doing um this one is trying to get to the heart of the matter is: do you use ftd and policies to match exact domain names, I.e? You know some deeply nested domain subname. Do you use ftd and policies to do a pattern match? Do you write something like star.mydomain.io and that that will really help? I identify where we want to go with this, this one's a little bit of a long-winded question and there's a lot to wrap your head around. It drops you into the meat of it.

C

um But basically the idea is is this? Is this something that requires an active proxy um in the middle of dns resolution? Or is this something we can uh you know you do asynchronously, uh like all the from what I read from dan's comments, all of the good implementations try to intercept dns traffic and you know update their allow lists with whatever they're feeding back to the pods, and so this is a we're just trying to get customers to say.

C

Yes, that seems reasonable to me, like I'm, okay, requiring a dns request before I send out traffic or if they want to write network policies using fqdns, but their traffic is just raw ips without any dns queries backing it. This was kind of a weird question. So are there any like question um clarifications here.

A

B

Presented sorry so so.

A

This sort of things works perfectly if you have two end user applications.

C

But if you would have.

A

Something that is or do some vpn like application right, then you would not do dns question. You would obviously just take whatever comes on the wire and it's the application that you vpn for where that would have to do the dns in that case. So now that is probably not the most common given at this application. That's for for sure, but uh there are many types of applications uh where you don't have the sort of actual session initiator in your own application.

C

So in that case, would you be filtering ingress traffic.

A

So I mean what's what's obvious when I talk to this, I do a lot of routers and load balances at layer, four level right and then normally you act as a middleman. So to speak that you someone might I mean if you're a little balanced, you have a whip right. Someone.

C

A

A lookup of that you do transformation, then, typically, you would not have enough kid, but uh that's the only sort of cases I can see really where, where someone they have cases where people just use addresses and those are easy to go and say that's bad practice, but then you have things where, where something will be.

A

A proxy of some type at layer, four, on the other hand, you would probably never use an fqdn in such a.

A

Firewall rule I mean it would be very hard.

C

Right, yeah, that's that's what I was hoping. I was hoping that if the application is using fqdns, then it makes sense for a security admin to write fqdns as part of the policy. But if your application is some sort of you know, router or middleman, it probably makes more sense to be writing those rules. In terms of you know the ip blocks that the router is managing or something like that.

A

B

But this question specifically is more like.

B

Is it it's more saying like? Do you want us to drop the tr like? Would you want an implementation to drop the dns query specifically or to be still mapping to an ip? Is that like what the question is it's like? Would you be right with us dropping a dns query, or do you actually want us to be firmly on layer, 3, dropping an ip.

C

um Yeah, actually maybe I misrepresented this question.

B

It is it's a good one. It's just a little confusing.

C

It is a little bit confusing. I think I might have confused myself so.

A

So I mean the the real problem comes when you have a combination of this and, like you had before right with the star in front, I mean you have all the addresses in the domain. Then you really want something to do to look up, so you can tag on on that lookup and again, then that's you have control over the dns from that pod.

A

uh Actually, that you, you have control over the resolver right, so no one goes and uses a encrypted, dns logic. What you're saying is that you want to push so that you can listen into the resolve answer and by that uh enabling uh this address in the firewall.

B

I mean, I think the wild card matching is more three like this. From what I like, when I read it again, it's it's literally like do. You want us to be dropping traffic on a layer, seven sort of my mantra or do you want this to be a true, or I say, layer, seven uh upper layer mantra. Do you want this to be a true ip fire wall like.

D

Like it's almost, this is a separate question from that. This.

B

This is not about.

D

Blocking dns as a way to block ip.

A

No, no, no yeah exactly okay,.

D

C

D

C

Probably should ask that question. um I I think that's a pretty valid question to ask. um I just assumed that the customer who we're targeting with this isn't: okay with just pure dns filtering.

A

I mean why you say fqds, of course, that many people have no clue about the ipad addresses they know what they're trying to reach, but they've got to even might not even know what the night pair this is, especially if you start having ipv6 addresses, then many people just get brain colds.

D

It seems very weird for for someone to be saying my application needs to hard code ips, but I want to write policies specifying fqdns rather than ips that that just seems like that. Not a use case like if.

B

The application is hard coding, ips.

D

Then you should write your network policy to hardcode the same ips right.

A

Yeah yeah, I agree with you yeah, like I said the only time I have that you and if you do like we said before the example I brought up. If you do a policy no way it's some sort of proxy, I mean obviously you're gonna have to base it on addresses.

A

This is sort of it would be very, it would be a very weird vpn solution, otherwise,.

D

A

Maybe for my kids, where they're allowed to go but.

C

I I included this because I I remember someone having an opinion on this in insig network, where they, I I don't remember the details now, because we brought.

A

Them I mean, I think tim had a lot of opinions. I had a lot of opinions. I think dan also had a lot of opinions. Yeah on the whole.

D

But my opinion was that we do want to be doing it. This way.

C

Yeah, I'm hoping we'll we'll be able to just back it up with some more data of customers agreeing with us here, because I think we all think this is how it should be done. Just get verification from customers.

D

C

D

Problem with the the dns idea is- and um I might have said this is like, if they're using encrypted, you know like dns over https or or something rather than dns, that we can snoop yeah.

C

That becomes tricky because then I guess you need to be. You need the dns provider to be complicit in your network policy enforcement.

A

C

But I'm hoping that that can be and implementation detail, um like you know, if maybe some cni down the line, will provide an all-in-one solution and say yeah. If you want to use encrypted dns go ahead and also deploy our special core dns pods or whatever.

A

And I guess we can also not have to think about the subset of applications sort of that set their own resolver. I mean that runs much more than just a normal stack right, so there's more than a normal pod and run their own resolve, libraries and so on that might go to a dns server encrypted or non-encrypted. That's outside the scope of the local. uh Are you listening in on traffic, or are you doping, the dns resolver right now.

D

C

The thing that I was looking at was essentially adding a a hop in the middle, um so.

A

You put a proxy.

C

In the middle and then.

A

Yeah exactly yeah yeah, yeah yeah, so the dns result would point there.

C

Yeah, so that's question. Five will hopefully get us some clarity on that. um I'm hoping customers aren't doing too much craziness here, but it's kubernetes. You can't.

C

You can't rely on that um yeah and then just asking about the service mesh. Do you use the service mesh capabilities to write? You know proper l7 rules. um Are there any other? I think one thing would be um ask if dns filtering is sufficient.

C

uh Yeah I'll work on crafting this question because I feel like it's kind of subtle for anyone who's, not a so not familiar with networking.

A

So my my biggest sort of not biggest, but one fear I have- and this is sort of not with current kubernetes but at the same time on some other place, we're starting to discuss more multi, networking and so on. We have many interfaces in a pod and then it gets a lot harder to keep control on over where the resolver is and all of these things and obviously everything that's network policies will be a lot more complicated but sort of the more special solution. That's needed.

A

The hardest sort of the more cases will be that that is not working, because someone do it from a different vrf in the pod then. So I think the I'm not saying we should do this, but I mean the the sort of the definitions on when this is possible. To use should be uh well def, well, dis, well described, and when, if I should say sort of multi-networking becomes a actuality, then obviously sort of that needs to be really really clear and overall, I think network policies will be a very interesting area.

A

If you go multi-networking.

C

Yeah, that's an interesting point. I hadn't thought of that yeah like what, if you resolve dns with one um network, attachment and then try to send out traffic on the other one that it's a little funky but not thoroughly unreal.

A

Yeah, it's like it's, not nothing, really! That's there now, but I mean it's. It happened down the road.

C

um And then the rest of the survey is just interesting. More than anything else, it's trying to get an idea of who's writing these policies.

C

um This might help inform whether we can just fold fqdn into maybe like a v v, one alpha two of admin network policy or whether we really do need a namespace version or anything like that, and then just some cluster info um it'll help with understanding the scale that we're dealing with.

C

I don't know that it'll affect the api too much, but I figure since we're asking customers we might as well get some. You know feedback.

C

But yeah this is, this is a survey that I'm thinking of blasting out um and I can make it into a google form. I'd like to ask if you guys are comfortable sharing this with your customers as well. It'd be good to get some data on what people are up to.

B

Yeah I mean, I think dan. We could probably do something like this. I don't know how that would work in openshift, but I would imagine there's some way we can send out a survey. Is that something you've done before.

D

um I feel like we must have done it, but I mean just sending an email to cignetwork seems fine.

B

Yeah, I think that's a good starting point but like if, if we're trying to get some actual customer data like I, I can definitely try to see what we would need to do for openshift like in openshift. The naming is just a little different like.

A

What, instead, if you set up a call somewhere and then you send the list on sig networks and so on and pointing people to it, because.

A

So that it can be sort of forwarded, one more layer, that's what I'm getting.

D

At I mean yeah, we made this into like you know, a google form or whatever, then people onsign network could just forward that url to their customers exactly.

C

Yeah, um I can, I can go ahead and make a google format of it. um I just wrote them up like this, since I just want to run them by folks see if there's any additions or clarifications. You think we need.

A

Obviously, if there was on the way sort of to splash it that cube con, then it might be able to get people on three.

C

Oh, that's an interesting point um when is cubecon again. Is that october.

A

Yeah was it eight yeah yeah.

B

It's october 24th I mean we.

A

B

I could even mention this: um if we get it's like a streamlined forum, I'm talking in the sig network meet and greet, and I could like throw it in a slideshow there. Probably yeah that'd be pretty awesome, um so that might be helpful and if not I'm sure we can find somewhere else to put that link so that it gets some love somewhere.

D

C

Okay, I'll I'll go ahead and make this a proper google form. So we can just link that out to folks who are interested in filling it out.

B

Yeah, no thanks for doing this. This is awesome. I think it will help maybe limit some of the implementation debate in cigna. I don't know, might help.

C

Might help um yeah and the other thing I wanted to talk about, and this is probably a longer topic, but.

C

Let me figure out how this works.

C

The other thing I wanted to talk through was just revisiting the doc that we had for fqdn policies. um I I do need to refine the user story section, but one of the things that we were still debating- and I wanted to get clarification on this- whether cni's that currently do fqdn can agree that this seems like a decent set of common behaviors.

C

That was another big stumbling block that I think tim pointed out to me was if we can't get every cni that has fqdn network policy today to agree on what this api should do, then it's pretty much dead in the water and there's not much progress. Kubernetes as a whole can make on this.

C

So I I'll probably have to reach out to folks individually, like I'll, have to ask yang what entry is doing, um but this is, I think, the core set of behaviors that I've been able to call out, um and it would be good to get you all to just take a look through this and see if what we're describing is reasonable. um I don't think it's too crazy. Like we're saying it's only egress policies, um it's this one! Might be a little bit tricky?

C

It's specifically, you know a lot list with uh a implicit deny backing it just because of the way dns works. You can't writing deny rules based on ftdns is always tricky and weird.

C

Here we call out that regex matching is allowed. um I said only prefix, no like pattern matching. I don't think this doesn't have to be a strong requirement. We can relax it. I just figured it's easier to be more restrictive and loosen it later.

C

um Yeah this one.

A

C

Just sort of says it doesn't act specially uh if you write an fqdn policy allowing traffic to go to twitter.com, it doesn't magically. Let you talk to cubedns or you know, whatever your in-cluster dns provider is. This is a it's it's good in that there's no magic behavior! It's bad! In that writing an fqdn policy while not being able to talk to to talk to the dns provider is pretty uh silly.

C

It's not super functional if you can't resolve dns names, so maybe maybe we do want to flip this. Maybe we do want to say you can talk to your cluster dns provider. If there's an ftdn policy just so the policy actually works.

A

So yeah, so I tried to remember. I see my comments on the sinusoid. I know what I reacted so strongly on with the star was that I mean I was not thinking of a solution that would sort of listen in on the uh on the lookups right when they happen from the application, but rather something that would read the rules and then do its own mappings. So I can't remember if it's clear clearly described in the beginning sort of the approach.

A

That's that's suggested with this that it actually is listening into uh the lookups, because then the star makes perfect sense. Because then you would know. Oh okay, this matches to to this rule, because we will read the text right and not just see the addresses.

A

I don't know if that's expressed really high up, I mean from the beginning the quality model that is supposed to listen to the application, as opposed to do the next lookup, and then the the system will listen, basically, look at the dns lookups and from that sort of uh be able to get the a yes or no should this be allowed.

A

Instead of sort of reading this resolving it on its own, which it's not always possible right, I mean, if you have a half a million entities under for a star. You don't want to have all that in your database. I think if that is described, I mean that then that then that can be discussed if.

B

A

It makes sense to have a uh listener or proxy, basically man in the middle or not and sort of then when the rest is described. That is done because I think last time that sort of got sort of intervened in the discussions, and then we talked around each other a bit.

A

So I think yeah that should be sort of described, not maybe not exactly but earlier sort of what is the operational model for me now. That is clear, as I don't react on the story anymore, but then I was like freaking out.

C

Yeah, um I that's a good point. I think that the thing I was concerned about- and maybe maybe we should do it anyhow- was I didn't- want to prescribe a a single architecture for this now.

A

But you could say that that's the exact expected behavior and that's why the star works. If someone figure out to do it, some other way, great.

D

A

That's what I would do.

C

That makes sense.

C

um Yeah we can call out.

A

Because I think we.

C

Had a couple of thoughts we were thinking, you can maybe do a core dns plug-in and then core dns tells your uh cni. What to do or you can have a.

A

Yeah yeah, so my reaction was that there's so many domains that don't let you do an ls on the domain to get out. What's underneath so.

C

A

That's that's a good point.

B

C

Let me just add a comment for myself.

C

Okey-Dokey, um let's see.

D

So actually, just thinking you know, another good question to be asking people is: um are you using an http based protocol and if so, is there some reason you couldn't just use an http proxy rather than a a or or you know? Is there any reason why this has to be done at l3 rather than l7 um other than just well? We don't want to have to deal with setting it up ourselves.

D

A

I guess you could say if you go in and look at it does. Kubernetes have a sort of one layer, seven network policy today or several. In reality we say we have network policies but they're, totally layer, three layer, four.

D

Right it, I just feel like a lot of people who want this feature like even like most of the people using the version of this feature in openshift, I feel like they would be happier with with a http proxy, but because they're thinking about the problem in terms of policy, they want it to look like network policy.

D

Yeah, I think but and yeah- and I don't know if there's any way we could automatically set it up as an http proxy, but yeah yeah.

C

D

Let's, let's add that, as.

C

A question to the survey: um ask them why they want l3o4 um or if they're, okay, with.

C

Having a proper http proxy or something like that and yeah, I think when uh peter and I surveyed our customers granted our sample size was like seven or eight customers, but we got about a 50 50 split between they are using a proxy and they aren't so, I suspect, there's a good number of people who don't want an http proxy, but we should verify that.

D

I mean I can totally imagine there are people who don't want to deal with setting it up um like they would like something more automatic than that right.

D

But it's like if you're going to get something, that's more automatic but not as good a feature, and you know requires us to be intercepting your dns traffic and blah blah blah. Then do you still want that? Or would you rather go with the http proxy in that case, and what, if kubernetes made it easy for you to set up an http proxy somehow.

D

I don't know how we would do that, but.

A

Well, at least, it makes it clear what's what the expected behavior is right.

A

If you would know.

C

Yeah, like the the one concern I have heard from customers, is that they aren't using http they're using some other rpc protocol, and so you know their requirements might be a little bit different, but yeah we should. We should clarify that.

C

um All right, so I made a note of that in the survey I'll I'll flesh that out, um but yeah other expected behaviors uh pods are required to make a dns before the traffic is allowed to. um I forgot to fill out this example, but I think we talked about this. I think we're all on the same page here, um yeah ip's, reported by cluster dns service are the only ips we need to allow this.

C

This comes back to the same thing if, if they have other dns resolvers we're not quite getting in that business, just yet um ftdn honors ttls uh pretty self-explanatory. I think this was a little bit open-ended. You know what happens if you've got a long-running connection.

C

Past the ttl of the dns record um and the connection's still active. It's just the ttl of dns expired.

C

I think it makes sense to keep the traffic going.

C

I don't know if you guys have any thoughts.

D

And there's that little side thread going there and what I said there like in in the case now where, when you have a connection open and the policies change, what happens? You know we're generally happy to to say: existing connections are not necessarily affected by changes.

D

So I think network policy applies to connection establishment. It does not apply to active connections.

C

Right, I think this is different, though, in that, if a ttl expires, it's not so much that someone made a policy change, it's just the natural behavior of dns. If you will like, if you just happen, to get a record with a short ttl, but you want a long connection. I think that's a valid use case that we should. We should be opinionated about right, like we should say yeah. If the ttl expires, that's okay! You can still keep your connection around.

C

As opposed to someone actually change the policy definition at which point yeah, maybe we let cni's do whatever they want.

C

C

Yeah, so that's that's one open kind of open-ended thing. We can be as specific as we want on that. uh The other thing is fkd and enforcement for cluster local services.

C

um This one's something I've heard opinions on both sides about um basically the thing that it comes down to is: if you try to resolve a cluster local service, you just get the vip, and so it behaves a little bit differently than any other fqdn, because you know, q, proxy or whatever you have going, is going to translate that fib into a pod ip oftentimes before network policy gets enforced, and so your fqdn system needs to be smart enough to handle that or you know we say it's actually not supported.

C

You should be using label selectors and regular network policy for things inside of your cluster.

C

um Yeah, for now I'm saying you should be using network policies, but if anyone has any contrasting opinions, I'm open to hearing them.

B

I mean it's exactly, I think you're right there, like I mean we're talking about egos traffic to the outside world, almost explicitly right with this proposal, so yeah.

C

Awesome yeah and then the final one is just something calling out scale and ipv4 v6.

C

um Yeah, that's that's the main things I'm I'll follow up with yang and a few other folks and just see if they have any thoughts on these. um These basic requirements I'll flush, some of this stuff out further.

B

Sweet yep, I think you, but you you might like if you can't get a hold of people either like just looking at the documentation. You should be able to express uh answer some of these, if not all of them.

C

B

It's like um that might be a good resource. I don't know if.

C

Anything uh andrea is the thing that I'm most concerned about, because it does look like they have deny dns policies, at least that's what it seemed like from the documentation, and I just want to talk to yang and see what his um experience with that has been, because her stance right now is that that's a bad idea that it won't work well, but yeah.

B

Well, it's really hard to make work. Well, I'd say right, yeah might work, but it also might create some holes, whereas that the the white light list model is definitely a little bit safer for something.

B

Yeah I just reach out to yang in the channel. Maybe I know he's still been super busy, so he hasn't been around really yeah I'll.

C

Reach out to him um that might actually pose a problem. If we tried to do this in um an admin network policy.

B

Yeah, that's that's exactly what I was thinking and it it had been so long since we first started this, but we had definitely talked about um I about it forever ago. um So, let's think about it, some more and and see because I don't think we're ever gonna be adding an ex implicit deny into avid network policy right. That would be against right core behaviors right. So it's definitely something something to think about.

B

um The only other thing that's happened since is we did add um the status deal to network policy. But again, I think going down the road of augmented network policy is the right thing, I think of anything like a network, positivity 2 or its own resource right right, but we need to sell cignet before we can go down that route right. So we don't.

D

Go down that route and.

B

D

And that's the hardest part yeah. So we'll we'll see how this goes.

B

Cool yeah and if you get a google forms, link just post it here, but I think pushing that at kubecon and seeing if we could get some info would be really great. So it's a great idea. I know it's still a month or so away, but I I'll be there. I definitely can push it manually.

C

Yeah, that would be awesome I'll, be sure to get the form out in like a week or so.

B

Cool awesome: well, thanks for keeping up the work on this, I know it's frustrating like especially early on, and are you the only one working on this or are there other folks from google helping out.

C

No, it's it's pretty much. A one-man show right now: okay, yeah! So that's why it's slow going it's whenever I get a few cycles to do it to a little progress. I I push the needle a little bit.

B

Totally been in the same boat well reach out if we can help at all, um starting to hopefully have a little more time, there's some folks coming in from red hat, who have been helping out with admin network policy, so have some extra cycles. Finally,.

D

B

Thank you. Yeah, of course, is there anything else. Folks want to talk about today.

B

Okay, on that note, we'll get some time back, 20 minutes, uh thanks for stopping by good conversation and uh we'll keep on chugging. I hope everyone has a good rest of the week. Talk to you soon take care.