Kubernetes SIG Network, 16 Feb 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Network meeting for 20230216 (Part 2)

Description

Kubernetes SIG Network meeting for 20230216 (Part 2)

A

At the wrong way, sorry they're wrong.

B

What do you need to do? I'm.

A

Trying to set you up so that you can share the screen. Antonio.

A

B

C

B

C

The box go ahead, Antonio.

B

What is the the button to to share the screen? You should be yeah yeah, it's in a big window, changes.

D

Yep, we see your screen, but you are muted Antonio.

B

E

Zoom does this thing sometimes where it assumes if you're projecting your screen, your screen is going to make noise and you're not so you have to like change the setting there.

A

See your screen, but you are muted.

B

E

B

I can't just hit the screen. No, it's false. Yes,.

E

B

Can't we can yes.

E

F

B

E

Three move something it's frozen, maybe.

B

Yeah he's saying that it's frozen share.

B

B

Now, you're muted, again whoops.

A

There something's moving now.

F

No, it's moving and you're unmuted yep.

A

B

We have I I, never used to have much contact with those balances. Recently, I have a lot of questions so and we don't have any way of testing those balancers locally. So you are, we are developing and- and we are practically blindly of if we have lucky and have a car provided we can test against that. But it takes a lot of time and- and it's very for me- it's very it's slow. So what I did is I created a cloud provider for kind?

B

You want to do no kind, the kubernetes in network kubernetes in docker that I work with Benjamin Elder and basically you create a kind cluster.

B

You set up the configuration to make it external I, I already created the question. Okay and you can okay.

B

There is no post here, you can run this club provider binary that they created okay, the crowd provider is going to connect to the cluster and is going to handle all the services the load balances. The instances is important because um it's going to be useful for Vanquish it he has attacked, modify the node IP. So with this you can you can I want to see with this. You can test, they know they beat him, but the important thing for us here is the loved ones. Okay.

B

So what I I created is a logic to computer log balance, and let me I have some examples that I want to show you okay, so this apply first.

B

C

B

With ATP question okay, so it is a load balancer this this here is: it basically creates a NHA process, container I'm confused to to quiz the date. So you can see the service, it has an external IP and you can connect to.

B

Okay, so you can see it's already. First name and I have, if you say, and it also this is cluster mode. So in cluster mode, you can see that it's taking a different criterity.

B

You can deployer local mode.

B

Apk local, we are going to have another service. Now is the eight seats and we can see so this is a nature proxy. So it's a growth, balancer deep mode proxy, so the source it is the sourcet field, the low balance, okay, I'm working on having a direct proxy, but but this is going to be the next person. Okay.

B

um What else can we do here? So, for example, testing the? How you see this to test the new feature that and do cycling with terminating F endpoints created. So this then, here, okay, because you can see that it's super fast to that.

B

B

Terminated terminating employees, uh this deploys a staple set with two parts.

B

We are running okay, the city service.

B

Okay, is this one: okay, five, eight five and we want to see the host name here.

B

Okay, you can see that it's going up now and now we are going to restart the the deployment and we have to have zero down time.

B

We are restarting so you can see the Apache one is terminating, it has. um The deployment has at least period and and the the application has a uh it's catching the sixth term, and it's still serving so there's no time, and you can see that now that is terminating. This is forwarding only for the Apache 0..

B

You see this should go away soon and the rose balancer is used in external traffic policy. Local is able to catch the health check and and identify the node without any distraction. Okay and now the zero has to run out- and we see that now the health check is failing. The load balancer move Apache zero, but this is still a volume out, but all the traffic is is being forwarded to the healthy part enough.

B

We have the rollout with zero disruption tested in less than five minutes. Okay, and since I mentioned it, this can be used for run E3 test. So you can write your E2 test, the ones that we have in in I created one for this demo, because languages has a nice thing to talk about that later and but you can run an E3 test and you can see that is going to pass too.

B

So uh you need to check the this, create, get the load, balancer tested or balancer check that it works, and now it's going to wait to for the service to disappear. Okay, so it can take some time and and that's it, so there are no excuses to test the loved ones. If it is now cool man and and I, don't know this. This is going to improve a lot. The this.

E

B

E

Is so overdue, like you just earned yourself, another Community award.

E

Yeah but you're wallpapering your house with these Community Awards. So um this is this is awesome. Antonio um does this now become the default CI integration, so the ede CI runs no longer take 45 minutes waiting for load, balancers.

G

Did we lose you.

E

Is he still here? Oh there he is yeah, he lost his audio.

B

No, no they're used to be speaking so soon. I, don't know who is putting the buttons, because I clicked the one that picked me up so yes, the idea is, is to have I mean we, we neglected a lot a lot of balances and the idea is to have everybody that wants to do something to have something and I plan to do to us and those to the CI and I'm happy to have I, don't know one hour or a meeting. If people need more interesting how to debug the test, how to understand, but.

E

Yeah I mean the truth is I, remember, going through all of the load, balancer ede test cases and massively adulterating the tests and bringing them all into one big test case, because it just took too damn long to actually test them in a clean in a clean way. Where the test cases did one thing, um it would be wonderful to undo that.

B

And that that's the point, that's in this point from from language I mean current state is and, and with this I- think that at least we can have you know uh uh General test for load balancers, that every every everybody should be able to support. Okay, that's my point and my I: don't have anything else so.

A

Very cool thank.

B

A

All right, Dan talking about e2b tests that depend on cloud provider, features yeah.

C

So I think this test showed up in uh Cube 126, um and so we rebased openshift and started running CI and AWS, and it turns out that this test fails because the default load balancer implementation for AWS is elvs. They don't support UDP.

C

There needs to be an annotation placed on the service to make it an NLB which does support that, um but the e2e tests obviously don't put that custom annotation on, because it's AWS specific and the cloud provider for AWS itself doesn't automatically set that annotation either, and so the guidance that um I got from some people was that the user needs to put that annotation on when they know that they want elb and it seems kind of gross to make an end-to-end test do that when I think they should be fairly generic.

C

So the question is really about, like what kind of guidance do we have here for things like this, you know: do we just skip that ede test on AWS, or do we try to push back and say AWS cloud provider should automatically figure out that the service is a UDP and therefore make it an NLB instead of an elb, um as anybody else found problems like this in the n10 tests before and what do we do to solve them?.

A

B

Ahead, Antonio yeah. Let me give you a bit more of context.

C

B

So this into it, we test in those balances were were always there. The thing is that I was cleaning up and refathering, and suddenly people started to pick them in.

C

Their automation.

B

So this is my fourth and the when I took it with Benjamin and around and other people into testing in theory day, three tests should not have anything hardcore. I mean this is AWS. This is everything this behaves this way, so everybody every test should be agnostic, but the reality is that it's not um so.

B

Basically what I think we should do is to move this test to be more agnostic and and remove all the the tests that have these. You know particular things. It's a AWS personal support is this I.

C

Don't know I mean the the simple thing. Is we just remove AWS from the platform list, because the test already has a skip? If platform is not, you know, GCE AWS, something.

E

Is is UDP not conformance.

E

Oh, that's right, of course,.

B

But that's the thing in Singapore and and when we went with the Captain G and and the thing we have this guy I mean conformance, is very wide. It's basically different everything and if, if it has an AP service, it's basically and but in SIM network we have these, things is, and languages wrote this document. What is a standard, proxy and I think we should start moving to this conformance and Network him. I have our test and and back then, and tell people well, do you want to be signature? Conformers, that's another performance!

B

You need to pass this text. I, don't know people agree on that.

B

But, for example, no has no confirm that are they are comfortable.

A

Okay, so for Dan's problem, uh what can we do? What are some of the actions we can take for this? One.

B

If the test runs in AWS and AWS doesn't support, we need to skip yes.

C

Sorry, I didn't quite catch that Antonio. You said we.

B

Need to if, if this case, yeah okay, what's the point of planning that test? That's what I would say, I.

C

Mean theoretically, it would pass if something set the right annotation to select an NLB I guess my.

C

Aws cloud provider to do that: um yeah I think there are some like cost differences between NLB and elbs, and so maybe that's why they don't do it automatically.

E

But we had I remember the discussion around multi-protocol load balancers right like if the user is asking for it, they're asking for it and uh I I used to think. Oh no, we shouldn't uh spend money, spend the user's money if they asked for something and maybe didn't realize what they're asking for, but I think it was Bridget who made arguments that convinced me like they're asking for it. It's not that they don't know what they're asking for they're, specifically asking for something: uh ede I mean the test is an interesting situation.

E

I do think it'd be worth asking the folks who own the AWS cloud provider like hey. Why don't you support this without The annotation? If the user specified UDP, they pretty clearly want UDP. You should do the thing you need to do to make UDP work.

E

Yeah! That's a good question, especially considering it's not a compatibility issue right.

C

Yeah I mean, as far as I know, like I I, don't know if it actually does work with nlbs, um because we didn't get as far as as uh hacking things up to set that annotation. But.

E

Nobody here from the AWS team just curious.

D

They do come regularly to Sig Club providers, so maybe we should get this on their agenda. The Sig Club provider agenda I mean that one's every two weeks. It just happened yesterday.

B

What we should not do is to do 14 on the test to behave differently. I.

E

Agree on all of these run if platform stuff are all like smells, we should have as few of those as possible.

C

Well, in this case, like we can run this test on the platform, it's just. There needs to be another layer of like platform grossness. If we wanted to try to solve an ed test, but I agree, I, don't think that's a place where it should be solved.

A

Okay, Dan, are you? um Are you, okay, with the idea of just like following up a cloud provider on this one.

C

Yeah yeah I can ask the uh I think there have been some conversations already with people who are involved in cloud provider and that's where, like the current AWS guidance thing came from because uh you know somebody at Red, Hat had asked about it, but I'll Circle back with those people and give them the uh our thoughts here and follow up. Sounds.

A

Good, do we think that we want to.

D

Also I don't know about an issue, but I will put an item on their agenda for their next meeting.

A

I didn't mean for them. I'm sorry I meant an issue for potentially uh trying to make these tasks like trying to pull away from having tests that are doing something. This uh specific getting into more generic tests. Is that something that we're tracking seemed like a few people mentioned an interest in that.

E

I mean now that we have an ability to do load, balancers that isn't linked to a cloud provider. It seems like a I wouldn't want to get rid of the tests, um but, knowing maybe they don't run I, don't know, I feel I feel like it would be retrograde motion to not test this I I wasn't even aware that the AWS doesn't support it out of the box like that is surprising to me.

B

Definitely we are going to have a test that is, create a load balances with a service type UDP and connect with, and it's going to fail in the AWS, because we need that theft right, you create a service load balances with an UDP port and you connect to it.

B

What I'm saying is moving forward this tester? We have right now we can. This test needs to be all of them need to refactor, but moving forward. We want to have more agnostic steps, and one of these tests is going to be I want to create a service, low balance it and expose a new decision, and they said it's going to fail in AWS right.

A

Oh I was typing live there. Wasn't it uh I'm happy to take on an action item of following up with Antonio asynchronously, and we can talk about whether or not we're tracking this desire to make the test more agnostic? How does this sound and maybe create an issue for it after talking with Antonio about it? Does it sound good.

B

No, that's that's a fact. All the low balance is tests I'm going to create them.

A

Oh, you are gonna, do it? Okay, it's.

B

Happening yeah I mean that that thing that we have now is is what we have now is. Is Anova I mean those tips are um um okay.

A

My bad I just misunderstood what you were saying.

B

B

To the an agnostic test and this agnostic test is going to fail in in Amazon, but we are not going to skip it because so what we want to do with that.

A

Gotcha, so just uh there's an open question so so was I. Maybe thinking I I was wrong, but I was right about what I was thinking originally, that we should just follow up asynchronously on this one and and kind of think come up with a plan for next time about what we're going to do, because we are kind of running short, a little short on time and somebody else wants to go.

A

Can I follow up with you, asynchronously Antonio? It's all good. Okay,.

A

Okay, uh msk Rocky.

G

uh Yeah- and this is just a follow-up on I- think from two two meetings before about the naming I did create a poll survey uh for our multi-network object. If you want to use, please just follow up, and if you can just select your your your uh preferences on the description, the topic has some links to the requirements PR of what the object has to be, what we plan to do with it. So please go ahead and and fill it up.

G

I'm gonna keep this up for the next week and then I'm gonna close it up. So I did advertise that in in slack in sync, networking slack but I think that they pick up too much, but so, let's bring it up hit here. If you'll just basically fill it up right. That's.

A

All cool thanks, Patrick.

A

All right, I guess we can go back up and get back to a couple of backlog, grooming issues, then um I had one I. Don't know that is super great.

A

um But this one was I think the main reason I picked. It was just because it seemed old. So basically, the idea is here that uh there's a potential bug at some point where we thought upgrading into a cluster with IPv6 would cause uh things to break with endpoint slices because they're not keeping track of all the references to service or the kubernetes service, API server service.

A

um It left off with Dan, saying we'll end up fixing this so I just thought. We'd check in on this one and see like is: is this actually moving or is it kind of stuck in the position that it's in? Is there anything to say about this? One.

F

So as far as I know, this does not affect any existing clusters. This is just a bug that would pop up if we added dual stack support without fixing it first um Roger that and the whole dual stack, API server thing is is kind of stalled because I keep working on other things and I actually have two pending PR's right now, reorganizing the API servers, endpoint slice reconciler, but nobody ever reviews them um so I need to like yeah poke people on API server team, to review my PRS, but.

A

Cool, so this is just a blocker and it is as it is moving in in that it's not completely Frozen.

A

Let's move on to Bridget's then.

A

D

Are issues sorry, um yes, I, clicked that one, because it was the life cycle Frozen, one I thought it might be nice since Tim brought that up for us to oh and it already went down from 35 or so to 28, which means Tim has been clicking ferociously in the background, um but yeah I figured, maybe just for the next couple of times we do the screaming, having this link handy to be like these are the ones that are sitting there saying Frozen for sure this five-year-old thing is something we want to freeze forever question mark so yeah.

D

Maybe we can grab one or two of these and go let's reevaluate, this excellent Dan. Thank you for unfrozening, one I feel like we need a soundtrack to that. Let it go, let it go.

E

I think you just gave us the soundtrack.

A

External IPS should not allow duplication.

F

And if you just jump to the end, there.

A

You go external.

F

A

Now accept inherently.

F

We sort of gave up on trying to make external IPS work, so I feel like we can just close this now. Tim I.

E

Agree, uh even if we didn't do that, changing it so that they're not duplicated would be a breaking change. Well,.

F

Like in theory, we could, you know, write some awesome controller that you know did cool stuff, but.

E

We we generally as a rule, we don't modify user input, so if they gave us a list and it has a duplicate in it, our choices are accept it as it is or reject it as it is, but not mutate. It.

F

Right I guess: yeah I meant a. We do a larger scale. Rework like openshift has this old thing that you can have it automatically assign external IPS for you, and then it makes sure that it's doing it. You know in a way that the administrator configured and is happy with so like in theory. We could do that, but we don't want to do that.

E

Yeah I, don't think we're going to spend a lot of Design Energy on external IPS, so kill it nuke it from orbit. It's the only way to be sure.

E

I want to watch it turn purple.

A

That was an aliens reference right. Yes, it's from Aliens, okay, cool. uh Do we want I? Do we want to look over any more of these things.

E

Do you want to do the honors Dan boom? Sorry.

G

F

A

E

About that close bugs yay, it is gone um jump all the way to the end. Let's look at the oldest ones: let's do it I, just unfroze, a few of them on the expectation that they would um reappear in the triage queue eventually.

E

Here's a good one.

A

A way to configure external IP addresses for a node with kubelet.

F

Okay, so and this so this gets into cubelet dash dash node IP Behavior, which is theoretically Sig node, but really we end up always dealing with it um and I actually I'm working on that right now, because we need Cloud, dual stack IPS, um basically, every now and then people propose oh hey. We should be able to do this, but also nobody knows what having multiple external IPS is even for like nothing. Actually, nothing in kubernetes actually cares about external IPS other than some of the ede tests.

F

A

Gotcha, so this one is on was not frozen anyway. It is underway.

E

Okay, should we unfreeze it yeah.

A

Should we mark it as such.

F

I mean there's still no agreement on whether we actually want to do this. I guess. Oh.

A

I'm sorry I misunderstood. It sounds like you were already actively doing this.

F

I'm I'm working on other node IP related stuff. So you know if we were going to fix this, then I can roll this in, but.

A

Gotcha, so the question remains: do we actually want to do this?

A

ah Does anybody have any opinions on the matter? We still have a few minutes left.

E

I'm still catching up paging this one back in it's, it's old.

A

Well, maybe we should just unfreeze it for the purpose of everybody or it popping back up and everybody taking a look at it. Who needs to take a look at it.

E

I support that.

A

About hey: let's uh come to some kind of uh alignment here on whether or not we're actually going to do this.

E

We've got four minutes left I I, don't suppose we have time to do one more.

D

I think we do. There was one really exciting.

E

C

D

The issue list, if you click back to the list of the life cycle, Frozen issues, it's like how big should kubernetes scale I feel like this. Almost a philosophical discussion infinite like, but it's it's an interesting question: do we plan to reevaluate setting some sort of constraints and or um uh affordances to allow us kubernetes to scale and, if so, to a different amount, Than People envisioned in July 2017 and go.

A

I'm not sure if we should, but if we do I, think that the limits should be backed up by tests. Maybe that's an obvious thing, but.

E

So the the problem with retroactively applying limits is there's certainly somebody out there who is uh one above that limit and if we start enforcing, um you know only uh like pick on something real, only a thousand endpoints in an endpoints resource, um and we don't have an alternate answer- that's already implemented like endpoint slice, then somebody somewhere is going to come and beat us up and we will deserve to be beaten, um so I, don't think we can put in place hard limits unless we think those hard limits are so big that nobody anywhere is possibly hitting them.

E

Like a million endpoints and I, don't think setting the limit at a million endpoints is particularly useful. I mean a million is sort of indistinguishable from infinite right.

E

um We can document it, and you know: Sig scalability has sort of documented it in the form of the tests that they run right like 5K node does you know XYZ stuff and that's what we certify.

D

Maybe this is not a question for Sig Network to make changes on, but it is a question for Sig scalability to say this is what currently exists and if people want to open issues to change what currently exists cool, that's a different question services.

B

Already published the numbers.

A

D

B

D

Their numbers hit services.

B

Well Services. Yes, this has a limitation I, don't know if they they have the number but Services has the limitation in Services is the is in the allocators.

B

I I created I, don't know how many thousand services and I need to be started to to go that because they allocator sites it starts to grow. That's why we have ipvc strategies with with a cap on.

B

But I I I can look good for you, but I know that they published.

E

Okay, what do we do with this one? It's a it's a good question um and for context. The person who filed it shyam was on the scalability effort back then. um So it was really Sig scalability asking Sig Network. What do we think is the right number.

A

I am okay, and especially since we're at time, I'm actually interested in this a little bit I'm, okay, with following up on this one and pulling in kind of the the parties who have already been in in here and see if, like maybe, this is just a documentation thing that we can resolve.

E

Thanks: okay, I unfrozed it for you all.

A

Right yeah sure I'll talk to Bowie about it too he's on there all right. So we are at time uh if you have anything uh that you that you wanted to bring up, make sure you put it on the next uh two-week out agenda. Thank you. All very much have a good one.

E

Thanks everyone.