Kubernetes SIG Network, 8 Jun 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Network meeting for 20230608

Description

Kubernetes SIG Network meeting for 20230608

A

A

Hello, everybody Welcome to the Thursday June 8th, kubernetes, Sig Network meeting. uh As usual, we are governed by the kubernetes code of conduct, which basically boils down to don't be a jerk, so please don't be a jerk, uh not that we have a problem with that here. um I'm gonna start with triage. We have a fairly light agenda today, which is great.

A

um If people want to add things to the agenda. Still it's uh it's open. The doc was shared with the calendar. Invite let me do some screen sharing of a window I'm going to give you an infinity, mirror.

A

You guys got my window yep. All right, I've opened them in Reverse time order. Let's take a look. Cube proxy pods continuously crash loop back off.

A

Anybody looked at this one.

B

Yeah, that's that's the oh yeah.

A

B

A

Seems like it's probably not going to be us and we'll close it I'll keep an eye on it. Next uh I opened these two. These are actually being worked on already. These are just adding warnings to uh API operations. We did some Antoine and Joe and I have been looking at um some of the uh issues around server-side apply and incorrect map keys, and we turns out we're really bad at this across the project and we have a whole bunch of places and I didn't realize that server side apply completely barfs.

A

If you have an invalid, something that is allowed through validation but not allowed through the map, keys and server side apply. Just does, unlike client-side apply, which just corrupts your data server side apply just gives up, even if you're not manipulating that field, it will give up so we're looking at how to address it.

B

A

B

Yeah, that's the we're, having kind one example of doing to install Ingress and doing a patch and the patch has this problem. So it doesn't work because, because of the duplicates of server-side apply but I I thought that you we have the the way in in the into in The Crying To indicate. If you want to do server side or try inside that Prime. You know if they removed that we.

A

No, you do um I mean on Cube cuddle. There's dash dash server side uh argument.

A

Is that what you're? Looking for yeah.

B

But I think that this is not there I, don't know there is another option to to change that path to to not hit this problem.

A

What do you mean.

B

Right so you can have the Strategic merchandisization plus and whatever yes,.

B

Easy to hit this problem with them or.

A

The so the problem is that in some of these cases the Strategic merge patch, which is client side, um will corrupt your data. It will find duplicate keys and it will just pick one and throw the rest away. Oh, is this thing too? Okay, and if you do it server side, server side apply, looks at it and says: oh, these are not unique, Keys barf and it just returns. A generic 500.

B

A

So we're looking at adding a better error code, possibly making it so that it only barfs if you're actually like, manipulating those specific Fields, um adding better warnings when you actually do these things and possibly converting some of the lists that aren't really Maps into Atomic lists. Instead of pretending that they're mapped.

A

B

It's if it's Atomic this you need to replace all of them. That's.

A

Right you can't co-own it: okay, okay, yeah, it's unfortunate, but things like environment variables is one of these. Where um the map says you can only have one value for each name, but actually they. It is not a map, it's an ordered list, because each name can reference names that came before it, and so you can't just reorder it.

B

And what happens with the bottom bracing.

A

B

Two weeks ago or something you had this problem with the bottom plane that was generating something defaulting, something for part no.

A

uh Yes, so that one's in um it is so there's different defaulting path for pod pod spec and for everything else embedded pod spec.

A

um And if you put your defaulting in the wrong place, then you will make a big mess for deployments and jobs and stateful sets and everything else. And if you put your defaulting in the right place, then it will only affect when the Pod actually is created.

B

Okay, but there is no possible problem related to.

A

I, don't think so.

B

A

A there's a separate issue open there, apparently that causes cubelet to like do the wrong thing. Sometimes, so that's a separate issue.

B

A

We should just not support patch. We should just yeah it's a mess. All right next issue uh support a list of app protocols open last week. uh I know this has been talked about. Rob, do you have any stages.

C

Yeah uh we've been talking, I mean this is very relevant to Gateway API, uh because, uh honestly, the the people that are reading and consuming app Protocols are largely the same implementations that uh uh our implementing Gateway API uh app protocols has always just been more of a hint than a requirement, um and the the problem is: if you have implementation, specific hints, there's not a way to describe that, because you can pick one, but then you can't have interoperability across. You know, implementations uh a list.

C

You know lior started this in the last cycle by basically defining that, oh, we could have domain prefixed app protocols to allow you to say. Okay, there there's this protocol.

C

That is my own unique interpretation of this, uh and this is what it means, but that is only really useful if you also have something like a list, so you can have some interrupt across multiple implementations on top of the same service: I, don't love it, but I also don't have a good alternative, so I I chatted with Leo or a bit yesterday about this, and it seems like this still feels inevitable, but it may maybe we'll just let it so can get some more ideas before trying to push this forward in this specific cycle.

C

A

I yeah I'm, going to assign you this issue and I'm gonna triage, accept it um I.

C

A

You kiss me or okay, sorry, because we have um we have app protocol. Now we have to do the whole pluralization dance, which is not fun yeah. Yes,.

C

Yeah I would love to avoid this I. Have we have discussed you know? Potentially, maybe this just becomes something like we describe in Gateway API this it that's even worse, though, because then you have to define the intersection between like if somebody sets app protocol and then they set this other thing, that's similar on Gateway API, and how do you interrupt between those I don't know, I discussion. Ideas are very welcome on this one um yeah.

A

Okay, anyone else wants a word.

A

All right: endpoints lost object, not removed when service selector is made to be empty.

A

We have we talked about this one before.

B

Yes, you just didn't do have more context with anybody. Yeah.

D

So so the issue is that if you create a service with a selector and then remove the selector, you are left with an orphaned, endpoints and endpoint slice, which is intentional. But if you then modify the endpoints object, you get a second endpoint slice for the same service, because now you have one endpoint slice that was orphaned by the endpoint slice controller and one that was created by the end point slice mirroring controller, and this seems like an edge case that we just didn't think about and probably didn't intend.

D

um We just need to figure out the exact fix for it. I think.

C

And we might be able to make Perry do the patch.

D

C

I, miss I missed that what what is the actual fix here like I, definitely understand the first bit that this is intentional I. Think that's what we're here to discuss. We don't know what the.

D

My suggestion was that when we uh orphan the endpoint and the endpoint slice that we make the endpoint slice be owned by the the mirroring controller, so that then, if you modify the endpoints object it just you know it modifies the existing endpoint slice rather than the then creating a new one.

C

Oh, that's really funny so in that one instance, the original endpoint, the main endpoint slice controller, would be pretending to be I, I, guess, I! Guess what.

D

C

Do would be to just let the primary endpoint slice controller just clean up, endpoint slices it owns and then trust the newly created endpoint slice of hearing. Well, no I, okay, yeah.

D

Implementation was, it might be easier to have the the endpoint slice get deleted when you remove the selector and then have the endpoint size. Mirroring controller immediately, create.

C

D

The the thing at that point, I was I, don't know.

C

Yeah I mean that that seems right. If it's a mess, though I mean ideally what you want here. Is there not to be much of a gap, but this is also I. Don't think this is an area we're trying to optimize for long-term. You know like.

C

When we, when we built this into endpoint slice, it felt like well we'll just mirror what endpoints is doing uh and it's a little funny. But let's not try and you know, question it.

C

As long as there's a fairly minimal delay between uh endpoint slice, getting deleted and endpoint slice mirroring controller, creating a new one, I feel like that's fine to me. Another.

D

Possibility would be that the endpoint slice mirroring controller could not create mirrors or could not create a endpoint slice. If there already seems to be an endpoint slice mirroring, although I guess, maybe it wouldn't be able to tell yeah.

C

D

Because of the labels, no, you can see that there was already an endpoint slice with the labels saying that it was part of that service.

C

Well, so right now the way the controllers work is. They are informers that only watch the endpoint slices they own so like they're, so they only know of their own little world. They don't know and changing that could actually be more complicated. At least that's how I remember the controller working we should verify, but.

A

Okay, so I've tagged both of you two on this is that fair.

C

A

Yep should I un triage it or should I leave it and let you figure it out.

C

I mean it sounds like a real bug. I I was only aware of the first bit that yeah I, just had completely missed. Oh mirroring is going to create a duplicate and that's going to be annoying, so I think it's real and should be accepted.

A

You got it pod garbage. This is the server-side apply one.

A

Among others we're looking at this.

A

Yeah, unfortunately, the only answer right now is not to use server side apply um when well, not to use server side apply with pod updates.

B

And this shouldn't be in seasonal. This is never.

A

Because the the fields in question are pod ports or the one that has tends to have duplicate ports or it was triggering the issues.

B

Yeah, but the problem is because that, although I'm scheduling added to the control and they need to fix the control.

A

I mean it's fundamentally, it's an SSA problem, um so I will tag them too. uh Okay,.

B

We need to backboard the thing it's broken. This controller is broken.

A

We need to figure out what the right fixes right right now: they're just okay.

B

Let me ask different things: there is this problem that they added this logic and because, when this logic fails because of this problem, the post that was collector starts to fail. The root cause is the the silver solid pricing, but what I'm asking is? Are we going to leave this control in this state of the week, come up with a different solution, or do we want to fix quickly this controller and yeah.

A

That's a a good question: I I, don't know the answer to that, um but I'm I'm on these threads so uh I will.

B

A

B

I have one link to the controller and to the clothing.

A

Pushing ahead, we got a flaky ipvs test.

A

Somebody has assigned themselves to it. uh That was two weeks ago.

A

Easy apology, cash is, has probably evidence, can content current map access, ooh.

B

This is Legend, This is.

C

There's already a fixed VR, I'm overdue to uh I. Think I did one very brief review, but I need to do a proper review, but I think this is on me um and yeah definitely accepted.

A

Awesome, after upgrade kuber mt's version 120 to one it's not coming up the controller manager wow. This is gonna, be a fun one to read.

A

Is there anything in here that indicates.

B

Yeah, actually, it crosses because says that the cider is occupied go down to the yeah. Oh one up, you know that one.

A

Feel to Mark Cider at index 0 as occupied for known, enter.

B

I think that we already have one of these. You know that Jay reported some time ago.

A

You want to you want to look at it. Yes,.

B

A

Antonio to the rescue whoops.

B

A

There's my contribution.

A

Network policies only take effect when pods are assigned. We looked at this one before it was assigned to somebody, but.

D

This sounds like a plug-in problem. Yes,.

A

That's what we said before uh I'll ping Casey is Casey here.

A

Is it Calico sorry, do we know it's Calico.

D

Yes, there was a comment.

A

Well, you're all are watching.

A

I'll wait for time. Oh, let's wrap up. uh Let's make this the last one then uh is a month ago can actually survive due to invalid contract. Do we know this one.

A

I remember this.

A

B

If we can't say this is symbolic, that's for this specific case. That's the best thing, but I don't think we can I.

D

Don't know are.

B

We able to do it to investigate to that.

A

Let's, um let's take this one offline and do the rest of the agenda and then, if we have time we can come back to it because we do have at least a couple of other things. um Antonio I, I jumped I put my name in ahead of you, but I. Don't know how long your topic is. You just wrote a lot. Do you want to go first? Yes,.

B

It's quick, okay, so uh I've been attending the cni meetings for because one of the recurring problems, mainly with cni plugins that can install as demon said, is that they don't handle. Well, they know the life cycle, and may you don't do a lot of trickery with paints or or everything is super Congress to get this right.

B

So one of the the way that that the network really word for kubernetes is just as the container runtime the container runtime implementation is very basically just checked that this has cni copy five, and that is ready and one possible solution is to implement a verb in the cni spec. So the container Valentines can have a better information about the CNA status and use this to provide the no node printers node network will read this to the key.

B

Another time in the meeting I don't know either want to say something.

D

So I agree that the cni status verb would help make this problem better uh for us.

D

um But overall, my feeling in this discussed in the the cni issue that you linked to is that we we need to stop pretending that or we need to stop trying to use cni as the primary interface to kubernetes network plugins, um because they're they're much bigger than cni and and so we should try and I mean this is a bigger project.

D

But you know we should try and find some way to to actually make a network plug-in specification so that cubelet can just like you know, use some proper API just to see if the network plugin is there and ready.

A

I I have thought down this path myself um for a while and, like you said, it's a big project and I haven't had the activation energy um but I I think I, agree. People I can't I'm getting tired of correcting people when they talk about Network policy as part of their cni and I. Just you know what the vernacular wins. Sometimes it's it's the kleenex of network plugins, um but uh I agree that these things are not um uncoordinated components. They need to be together and there probably should be one holistic API.

A

It should probably include host ports for pods. It should include ipam, it should include uh probably node ipam and pod ipam, maybe definitely Services Network policy and as Network policy apis evolve, probably should cover all those too so you're right it becomes a real proper, like Network driver.

A

Anybody wants to work on this.

D

I was about to say we do have some like. You know new contributors showing up looking for stuff to work on it's kind of a.

A

Big I'm not sure this is a good.

D

First issue: yeah: no, it's certainly not first issue yeah.

A

Okay, well, uh look I'm, I'm, uh broadly supportive of the initiative, I'd love to see if somebody wants to start putting together thoughts on what would all be included in such an API like starting, very vague and refining it I'd love to discuss and look at that I'm not going to have time to drive it myself in the near-term future. But I do think it's important.

A

Okay, uh Antonio! Is that all of your topic.

B

A

Mean I've seen.

B

That this at least give us a avatar the state, not the idea, but right.

D

B

It's completely broke.

D

Yeah and like I was saying at beginning, I mean the cni. People do want to push the status verb forward and and if we had it, it would make things somewhat better. So you know I I wouldn't say we should avoid this so that we can get the perfect solution, maybe someday much later.

A

um Yes, I'm definitely in favor of incremental uh In-Place improvements whenever possible. Okay, um the last thing I had on the agenda unless anybody has added something to it. Nope uh is to take a quick look at caps.

A

The cap prr freezes nominally today uh and um we are short on prr reviewers more than ever, um so anybody who's looking for a way to contribute shadowing prr is a great way to have some outsized impact, um but I wanted to run through the Caps that we've got open and make sure that I'm doing the right thing with them. So let me share screen one more time share uh uh this one.

A

Everybody see this. Yes, all right. The columns sort of reflect the code base of the features, not the code base of the Caps, so a few of them have like already moved towards GA, but, looking at anything, that's tagged, 128 I think we're still going where we did GA already. We we've merged that for pte uh iptables, restore minimization was I, think it was, it must have been checked in if I already moved it to across columns uh and DNS expansion. That's fine um I talked with uh Rob and Antonio this morning.

A

We're going to punt topology aware one more time: um node Port, static ranges, I think wants to go. Ga we're just waiting for a PR to do that. uh Iptables ownership uh Dan! Was there a PR for this yet or not? Yet that's.

D

All done for yeah it's it's beta for 128.

A

Oh it's beta for 120s. We need a I'm working on a better dashboard. It's not ready yet um okay, so it's going to Beta in 28, got it.

B

Right and another point to that.

A

Oh okay, great yes, my! Hopefully my new dashboard will be much clearer and we'll say like what the next step is and when um that's, what I'm, trying to figure out how to represent uh the service controller, stuff I think also wants to go. Ga we just need a PR oud.

A

Sorry go ahead, I'll get on the pr sorry, I've kind of lost track of these. uh It's fine! We we've got a month plus to get PR's in, um but it is planning to go ga right, yeah.

C

Yeah all right there weren't any issues reported so I think yeah well,.

A

That that was my- that was my read too. um So no no rush we've got over a month to do it. We just need the pr okay, um multiple service citers is paused. While we figure out the overlapping issues, Antonio's got some docs there that we need to revisit so not going to make the cut for this week. uh Admin Network policy is async: dual stack, node IP, stuff Dan.

A

What was going on with it.

D

So the the cap is actually approved to go to Beta already, but it turns out that it didn't actually work in 1.27 and I. Just pasted a PR that that you have to merge so it'll be staying Alpha in 128.

A

Staying Alpha, all right I will update that one second, take it off lead opted.

B

A

And change the Milestone to 29., and where did you paste? The Zoom chat great, give me one second, while I load it so that I don't lose it uh where's chat chat. Is here load that one okay pop back?

A

uh Okay, those are all the alphas. Now things that might go be going into Alpha, I didn't have time, I, don't think they've done anything there.

A

uh Field status, host, IPS I think is going to make alpha we're still reviewing the PR, but it's at PR stage. So, okay um and this one I tagged but I, don't even remember what it was. The details now or the drain terminating.

C

D

Saying if you remember that, yes,.

A

The the uh the health check yeah exactly exactly.

D

So the pr is good to go, I just updated it today and and Tony has been reviewing it. Thank you so much so it.

C

Should be pretty good to go right for Alpha.

A

Okay, great as long as they like it, doesn't need prr or anything, then we're good for this week's deadline um are there. These ones are all in some form of stasis. I think I, don't know anything about the host Network Port host Network support for Windows. um Does anybody here know what's going on with this.

B

A

All right, I'll have to follow up on that um endpoint slice into staging stuff. Is that moving at all and.

C

I know we want to see that move. I, don't know if there's anyone with enough time to get it in this cycle,.

A

um Well, I mean honestly, if it's not like substantially done with respect to well, this won't have any prr impact. Will it no.

C

Not really it's it's, it's a pretty small one, so we'll.

A

See well so uh I'll ping it, but it's got on the order of what seven days six days to get through the cap approval. So that's it um Dan I, don't presume. Nf tables is moving forward.

D

um I'm making progress on the code, but it's not moving forward. Actually yeah the cap hasn't even merged yet um I guess people. Well. When we're past deadlines, people should read through it again and see what needs to happen before we can merge the the initial cap.

A

D

I mean none of it has to happen. It's all just to make it nicer to add new, no.

B

No I'm really look it's. You cannot also unit test with your other path, so those are released, but I was wondering if the with a reason, a special reason for them, or did you just solve the call.

D

Then well, okay, so so actually the the um the incremental changes stuff in in the service chain, tracker and endpoint change tracker is, is sort of broken right now, and so we do need to fix that before NF tables can be fully working but and and the refactoring is sort of leading towards that. But no in general the refactoring is just to make everything nicer. It's it's not actually like a hard requirement for NF table support.

A

Okay, um this is the kaping issue which we need to resolve uh completely at some point: it's relatively soon, but it it's not moving forward. I think not I mean not in this cycle.

A

um These are. There was some discussion of uh command line and config config file.

A

Does anybody have any context on that? Well,.

B

I've been just in the, uh uh there is an issue with the moving the conflict to beta one and all this discussion about putting the platforms I think that's. This could be yeah.

D

I didn't realize we had a cup about that.

B

Yeah me neither so that's why I said it. We discussed it into different places about this.

A

Do you want me to reopen it? It's currently closed because of rotten.

A

D

The issue number.

A

D

Look at it: okay,.

A

uh And there's this idea of probing from within the pods net namespace Antonio I think you were championing that, were you.

B

But and and I took it with Patel and they like it idea because also it's more secure than everything, but this is that's everything. I wish I had more days. Okay,.

A

So not moving anywhere right. Now, that's fine! We are all very constrained we, the good news is. We are like getting a lot of caps into conclusion, which is awesome. um It will make room on our agenda for the big ones like Network policy like uh rebooting the network plug-in system.

A

um So that's good uh frankly, uh my my quarter. This quarter has been ridiculous and I haven't had enough time to review everything that I wanted to review so much less my own caps, um any other questions on caps did I miss any maybe no of a cap. That's not on this board. Yet.

B

Okay, what happens with the admin Network policy is? Is an independent group right.

A

It's yeah, it's just: it's not version locked to the release.

B

uh Okay, that, while we have it in the office, okay.

A

uh That's a good question: uh where is it Admiral Halsey? uh Just because I haven't been tracking it very carefully.

B

D

I mean their their own well, I, don't know how we're tracking Gateway stuff in terms of caps but like we're, not there's no Gateway on here. Okay, um yeah there's been talk about adding in a network policy equivalent of gaps except that, except that nobody can agree on what acronym to use so that that's holding it up, um but.

A

Nipeps, of course, silent end.

A

um Okay, I mean should should we have one for Gateway on here, tracking it or or several of them for Gateway Rob.

C

I we have so many things in fight it would it would get out of track pretty quickly um what we we did have a cap or Gateway API, and that cap tracked basically everything until we graduated to from, like the experimental, API Group to the standard, Kate's, API, Group uh and so like the cap was just this API I should graduate to standard Kate's, API Group and then at that point everything just was gaps: okay, um I'm.

B

Hoping to yeah.

A

Right now this this dashboard- this is the old GitHub project format and it's not very extensible. The new project format is much more extensible, so I mentioned I'm I'm working on a new dashboard for myself that that I'm going to bring back into this and hopefully convert this into that and then it'll be easier to manage, and then we can consider by then it'll probably be too late for Gateway. But we can consider um adding more caps for more fine-grained stuff, because I think life cycle will be easier to manage.

C

Yeah, just uh for anyone curious, we have our our own Gap project board uh in Gateway, uh so it it's similar to this. That is using the new style um and yeah. We've got a lot of things too, so um yeah completely agree that the new uh Project Style is it's very helpful for.

A

This kind of thing it's fractal all the way down. Every cap has sub caps and Sub sub caps, yeah.

D

A

Right, I'm gonna, stop sharing. If we're done with this and I'll follow up on a couple, uh I think: that's it for our agenda. Yes, anybody else want to bring up topics.

A

Okay, uh I guess you can all have 15 minutes of your life back. uh Thank you all for coming today. If you have issues- and you want to discuss, you can find us all on slack or uh one of a multitude of other places.

A

um If you have PRS that need to merge around caps in the next seven days, please ping me directly. Don't be shy and I'm happy to look and try to approve if we can get those done thanks. Everyone I'm gonna, stop the recording. Now.