Kubernetes SIG Network, 8 Aug 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG network meeting 2019-08-08

Description

Kubernetes SIG network meeting Thursday 08 2019.

A

And we're recording this is the kubernetes sig network meeting for Thursday August, 8th, 2019 I think to start.

B

A

Valerie, do you wanna have an issue triage this week, yeah.

C

I was thinking. Maybe we could do caps, the actual issue Trish, because we have pretty much cleared the backlog and look a little bit at PRS. Now, let's do I have the wrong thing up. I think I. Do please.

A

See top metrics.

C

Sorry, it's confusing having a bunch of windows here at once.

C

Alright, there we are I'll, go up to the front of the list. Ok, intersection of exclude balancer and external traffic policy needs semantic definition.

C

Well, Tim just reported this, so we can probably mark this as resolved or remove end result.

A

Probably it's also I think one of the agenda items for today.

C

Ok, to be a kid, sir 15 overflows when using string conversion and down casting the result, something from the security audit.

D

Anyone work it.

E

Says manage disk controller and does it site, although you used to call sites for this or if they just? Is this a general warning and then anybody who calls ADA I should go check. It.

C

That's a good question: I mean it's tagged that it's tagged for the sig, but this is like the only specification it has this example with pods. It might be worth just like someone opening an IDE double-checking. What the references are.

E

You can assign this one to me. I'll take a look at. It is.

C

That you Tim yeah.

E

Sorry I was a compliment late as usual.

C

All right, good proxy, failed to restart cause by deleting chain cube. Mark mask I feel, like we've heard this one before it's a new issue, though so error moving. Iptables isn't this chain that cubelet also manages yes,.

E

So I don't know why it's trying to remove it. Nobody should be trying to remove that chain.

E

Somebody for my PBS line, wanna take a look.

F

Yeah I can take it. I thought it was I thought this is an IP tables.

E

C

The IPPS proxy earth, okay,.

E

Thanks today, you're it.

C

Alright health checks failed outside of ingress, controller, AWS and I'll, be.

C

This is probably gonna, be an AWS one. That's.

E

C

Nw controller is a little bit wonky.

C

E

C

The label now Andrew, because it's not cigarette yeah.

F

I think I saw the AWS table already on it. So if you need to say just put it on cloud provider, okay, I'll just.

E

And remove signal yeah.

C

All right network service and n testing.

C

E

Is there a PR attached.

G

Looks like there's something.

C

So, are they reporting that there's a cryo specific issue, at least us right.

H

They put the localhost address on the node object and they're willing to pretend chocolate to the no Pole. They are using 127.0.0.1 they're doing they're in the park with without most network, so that wouldn't work who is putting the local host on of node I. Think that time trying to use 1 to 7, not you, the coil environment, I, don't know yeah.

E

Beside you can just stick this one well, I, don't even you touched or in you wanted to really take it. Yeah.

I

Sorry, what's decent in a mr. Han mi range region right.

E

One day, I'll figure out what that means, always.

C

Makes me think of a Star Trek character.

C

Okay, after deleting a service, your proxy cannot delete the corresponding IP tables rules.

C

This is a different one write the cube mark drop.

F

This is also I. Could yes,.

C

C

It looks like it is IPPs.

C

All right all services losing at points after a couple of hours on windows, node.

C

Anyone here familiar with Windows networking.

C

J

Sorry I'm, not a wall, remember yeah, yeah, okay,.

C

The boss, just won't, let me actually assign that but I'll tag you there. Okay,.

J

E

C

All right, Kubler start fails streaming. Service stopped unexpectedly uh looks like you had this one Tim do want to pass it off. No.

E

Can you jump down to the bottom, just refresh me where we left it yeah?

E

E

Yeah I responded to something yesterday: yeah I've still got it this way. Okay,.

C

This one looks: eight is something new on it. Alright, so sig network Deanna's failed, should provide easy hosts. I'm.

K

Actually them on this and we tried to repro it when me, let me try it on an action cluster bring up a bar, it seems to work so I'm just trying to get more information.

C

Sounds good, alright, proxy, better metrics for failed actions.

C

Yeah I would say so: yeah.

E

I guess that's right. It seems like an easy one. If anybody wants to rack up a couple of help. Log vids, adding metrics is honestly like the simplest and simultaneously most impactful thing. You can do with like a half hour.

H

I

You assign to me these people a fight. You.

G

I

What was a 5g, oh.

I

C

Sorry about that Thanks.

C

Connections to service cluster IP with no endpoints are not rejected. What we fix this a while back I feel.

E

Like I thought we did and then it came back or we missed a case. We can jump to the bottom again.

E

This was Dan. Let me this.

C

C

Looks like Dan probably got a sign right a while ago, yeah.

E

Send this one to me: I'll, look at it or close it.

C

Thank you, okay option to specify source not address for VIP traffic, feature request.

E

You see the subject. Sorry I didn't read the first one.

C

They want an option to specify the source match for VIP traffic.

C

Anyone want to look into it or have like an initial thought on how viable that is.

E

I mean it sounds like a potentially real feature request: I, don't have any thoughts on how to design it at the moment we can either you're, probably just mark it as triage or kind of feature and remove the triage label. Okay. If anybody wants to help think about how to design this like help is welcome. I do not have the bandwidth in the near future to think about.

G

This Jenny how my different interfaces can find kubernetes one of the innovators and leave the rest at all.

E

That could be this reading the first message it sounded like when you have multiple addresses on an interface. Oh.

L

E

See so I can see how I'll get confused like I, don't even know what the proper masquerade behavior is right, but I don't know what the design for this would be. I'd have to think hard about it.

E

So I think this one to triage. I, don't know if the blood where is here yeah I can talk about it. I guess are you? Are you master, yeah yeah? Can you yeah? Can you tell us just a.

B

Bit more about it, yeah we just use routes or scents for our route, so we'd have on the colonel to Potter, dresses and so masquerade will just over I didn't use an address from the interface and said, and so that's it's important for us to be able to specify the source address of a network traffic network so specify it.

B

We have a different address, it's advertised from the host, so it's it's not an address, that's actually on either of the interfaces. So this consistent isn't the same.

E

On every host, or is it different than every host? Oh, it's.

B

A forum on every host yeah so so, like my thought, would be to implement this using like a just a flag. I'm like you lit, or something in coop proxy as well, because I believe both them program that chain.

E

Okay I mean it sounds like I, don't fully understand the problem, but it could be a legitimate feature request so feel free to sketch out some design ideas. If you want or start a camp, if you want sure yeah dude does.

B

Something like this need to cap, or probably.

E

Okay, you know small, like we're trying to do more caps for even smaller things. It doesn't have to be super expensive to write a kiss. It should be relatively painless. Okay,.

C

All right, almost through this list, whoops look like I picked up one, that's not network.

C

E

Again, you know yes,.

C

Other people are using the unresolved label and that's easy, but I filter on all right. Unidirectional UDP traffic to a mid port service did not handled properly.

C

Looks like a long page, let's see.

E

I remember looking at this that.

C

People are not happy about this. It looks like.

C

So, what's the what's the context on this yeah.

C

Anyone have some bandwidth to look at it more.

K

E

You had it there P REM.

D

Need a picture mm-hmm.

C

All right, Karuna, DS endpoint, maybe ou API machinery and also not us okay. So that's the full issue backlog. So perfect timing um I want to look at the bit at the PRS because we have a lot of those. How far back? Should we probably jump.

M

M

Think your peers, oh.

C

Yeah, sorry so also.

E

You know a lot.

C

Of the old ones will probably just be slamming shut at this point, but that's.

E

A good point, I I've, been going through I, was out for the last two of the last three weeks, so I'm going through the appears that are assigned to me and I know. There's many people on this call and many of these PRS in this list that fall into that category, but yeah. Maybe we should start at the oldest ones. Okay, see if there's anything, we can just be well I.

E

Will sec I just want to say a thank you again to everybody. Who's been running the triage since we've been doing that our backlog looks much much better. We still have a lot of open issues, but they're all actually real issues now and not duplicates and help requests etc.

C

Yeah at some point we might be able to start like more poking at unassigned issues trying to get people on them, since the backlog is not piling up alright. So here we have an old one for handling annotations.

C

This one was, it looks like it doesn't have too bad of merge conflicts. It.

E

C

Like reviewing.

E

This is it really signet work as it's all in AWS should be sick cloud writer. When was the last update to it, though,.

C

Last update is May.

E

Is it assigned to anybody.

C

Is assigned to Justin Santa Barbara, it looks like it's just the AWS stuff yeah, so we can maybe just remove ourselves. Maybe.

A

Can we say why stick network was assigned? Did somebody do it with purpose? It's.

E

Probably an AWS I'm, just wild guessing that nay WS networking side of things so.

C

Not the scene where it was actually added its broadest, because it was a double yes, networking something.

E

Service right so I was probably service means networking means Sigma, yeah yeah he'll be, but we've been pretty clear that ELB is a singing or a cloud provider. Ii know us issue.

C

Yeah, this is just some small logic and the provider handling yeah.

E

The problem, frankly, like I'd love to see the PR get resolved or closed and just removing ourselves from ISM fix that but short of getting sick cloud rider to do the same process. There's not much. We can do with it. So we can just remove Signet work and reassign sit club later make it Henry's problem. Yeah.

F

Looks like Mike roots on it, but I'll follow up and make sure that it's on its radar.

C

Okay, next one add support of local traffic policy for wind kernel, proxy.

E

This assignment to st. Patrick yeah.

N

E

The bottom status of it.

F

C

Some stuff from May looks like it's just not getting updated, maybe just ping ping sure.

C

Okay, add completions index feature to parallel job, so this looks like it's a everything.

C

I once accidentally missed squashed, rebase and I put up a PR like this by accident.

I

This probably is.

E

It means been wild drive like it may have been tagged before or something no no yeah.

C

Does anyone want to look at this or you.

E

Can decide it to me yeah? Can you change the assignee to me and I'll assign to somebody and see gaps or something they've got some shrine there and also I just want to worked it. The history of this one.

E

Got more labels than code.

E

C

Right correct, been with failure past two bandwidth, CNI didn't.

E

I approve this one I thought I approved this one.

E

Sitting with some it's already been lgt and I.

C

See a comment here that it's already the changes already been made: oh okay,.

E

Go ahead and nuke, it.

E

Third call: this was simply like changing it from the KB Danby or something like that.

C

Looks like a pretty simple change: okay fix Golan issue in some package controller packages looks simple enough. A lot of comments, though so I'm guessing this just touched a couple of Signet work files, so I want to take a look, should be pretty easy.

M

M

E

So many PRS get to the point of like it almost looks good just resolve the merge conflicts and then they rot there.

C

Okay, work in progress one we can probably just skip over that check. Loopback interfaces for unicast IP and choose find a dress.

M

I'm, an idiot I'm.

C

Sorry muscle memory there.

C

Use IP address from CNI output.

A

C

Yeah, probably I need to bother anyone on this yeah.

C

Was it you who cleaned up the backlog Tim because I swear a couple months ago, I looked in the backlog was ancient for PRS I.

E

Closed some, but also fader bond has been affected.

C

All right move provider about taint from KK to cloud provider, I'm.

F

Pretty sure we can close this, it was done in another PR, but assign it to me and I'm.

L

Already assigned so.

F

C

Look at one or two more quickly and we return the value from merge. Health checks. Simplified I'll check, good one to check.

C

Just builds up, I was like firing off reminders a while back after kid con just going through stuff. So.

E

I I have a bunch of bookmarks that give me all the PRS that are assigned to me that are grouped by size. So I, like literally whenever I do my PR blocks a couple times a week. I go through them all the extra smalls, all the smalls, all the mediums, all the larges.

E

So if you send me XXL PRS, that's why they don't get to very office.

C

All right look one last one, no timeout when Cleveland calls CNI plugin is.

E

Already approved.

E

C

No much conflicts either.

E

Three days ago and there's a sign to disability.

C

Yeah looks like a gonna, be that one then, because it's still additive and almost out the door cool. Alright thanks everyone I would.

E

Urge everybody go look at all the peers that are assigned to you according to github and see, if there's any, that you can reassign or ping or closed or something, though this is a lot of PRS that are open.

M

Yes, a network signal yeah.

E

E

M

E

In sync, we've tried a bunch of different approaches to get it have notifications to go to mailing lists properly, and most of them have been object killers, let's not waste time on that for now. Okay, thanks happy to do another week or do you want to punch somebody else, I'm.

C

Fine to do another week, it doesn't take much prep.

A

E

Get out of it whenever you're a little tired in this.

A

G

A

Next up, we've got Rob Scott yeah, just.

L

A quick update on influence lice there's a couple of those XXL PRS that Tim loves, probably both assigned to you and n many and anyone who's interested because they're, massive and I'm sorry, but it's adding a new discovery, API group and then the second one is a 10-point, slice, controller and hopefully coming up next week is coup proxy integration or endpoint slice so interested in any kind of feedback on it on those PRS and hopefully another one next week could certainly use help reviewing yes,.

F

What was the consensus for service, topology and and points like? Are we doing those in parallel so that we can use the topology information from services to implement the endpoints like stuff? That's.

E

A great question: I haven't looked at the discussion about topology since I got back from vacation, so I don't know if what the discussion was. It was.

E

G's, at the same time like do, we want to add like what? What what? What does that mean? You.

D

E

Well, I think we agreed that it would be part of endpoint slice. The question I guess is: do we put it the topology information into endpoints as well, and what is the risk of like? Without that the QED proxy will have to watch all know.

E

And so that's the that's. The open question.

E

I know that the PR has reopened or not reopen but re-emerged for topologies. It.

L

Seemed like you so.

E

When I look at that as I go through my PRS this week and we can revisit whether there's something you want to do there, the simple answer would be to do it the same way and put it into the endpoints structure, I'm afraid of the blow out of that, though, like we know that we have customers that are already running up against the limit, this would just make it much much worse.

E

But maybe I, maybe maybe we can find a half-cooked solution that says topology in alpha. Only works for up to you know 100 and I, don't think about it. Okay,.

F

A

Tim you're up next all.

E

Right so there Clayton, the wonderful Clayton, has raised an issue with the node role, sanitation. Anybody who doesn't know node role, equals master, was added for deployment tools to be able to flag certain machines as control plane nodes.

E

More recently, we've been arguing about what that means and the general consensus is there's no such thing as a control, plane, node and so Clayton's on a mission to eliminate anybody who uses the node role label within core of comity. It's totally fine for users who want to use that and make assumptions about their own environment, but we shouldn't have built-in stuff making assumptions about the apartment. So one of those users is the service controller.

E

When it prepares the list of nodes to feed into the cloud provider for load balancer eligibility, it filters out any node that has the node roll label so part. One of the proposal is, but let's just remove that, and then you just say well what? How do we filter those machines out? Well, there's already an annotation that was added a while back as an alpha which was exclude balancer, and so maybe we just formalize that move it to GA and tell people like if you're Cupid minun.

E

Do you want to set your master control play nodes, do not participate, load balancing you should set the exclude balancer, annotation and I argued sorry for the long history I or you that this is super cloud provider specific, because all the load balancing stuff is really very cloud specific.

E

But at this point this the behavior is so consistent across the cloud providers. That I think we can just give a generic name to this annotation and call it okay.

E

The question at hand is what happens if I have a service with only Laura external traffic policy equals local and it lands on a node that has exclude balancer. We have an intersection of two features and I am asking that this group consider what we think the semantics should be in that quadrant.

N

Two more notes on the problem: one is the not only the service controller consumer of the no no master, and also there are a lot of ingress controller out there. My thinking they used the same library to determine what knows to be included behind the no balancer or not and I think they share that know. Can you respond to that issue and sure yeah?

N

Yes, that does one thing and then the second thing is that lets say for regular, open source communities deployment the usually there is no queue proxy running on master right, usually, there's no queue proxy and, let's say we sort of removed, no role, the role able and the service controller sort of okay treat the master. No and other notes the same. That includes master no into robots, pretend there's no queue proxy on the master running, so that means either. We need to standardize that to include queue proxy running on master or exclude I.

E

Think the plan the Clayton's written and you should go read the cattle.

H

Since you have the context.

E

Is what at least one release where we have the old annotations supported another release where we have it enable almost basically covered by a gate and go in the other direction for future gates? The feature here is the removal of a feature, and so we would start with it enabled move to it disabled, but enable and then remove it over the course of several releases.

E

That's that's the proposal, so we could get all the tooling and everybody else up to the the correct, exclude balancer, annotation and there's a few other places internally that use the node role so that he's working through those two from the networking side, though I don't have confidence in what the right semantics carries. So I laid out in an issue that I linked off the agenda of sort of an analysis of like should, should X work should Y work, should it be required or allowed, or it's really not clear to me.

E

I can probably argue all the positions and I would like to here to think about it.

O

To see exclude balance set also mean no service proxy, no proxy.

E

On a note, it does not necessarily mean no service proxy. On the note that is a totally different decision. This is that the intention of this annotation was a hint to cloud providers that saying, if you're doing to hop load-balancing, don't use this known as a general gateway for the load balancing infrastructure.

E

The question is with only local. If you land a pot on that node, it is stranded, it will not be able to get any traffic and it's unclear whether that's okay, intentional or whether cloud providers actually have the freedom to do better if they can like I, don't like inconsistency so somehow.

O

It seems to be tied to scheduling not intentionally, perhaps, but what.

E

O

No I mean if you schedule a node on the master and it is stranded, like you said, what's the purpose of scheduling.

D

E

Scheduler has no such information right and in fact it can change dynamically right. Don't don't conflate this with just masters? There are other the reason that the entities was added in the first place was to be able to decorate other nodes, non master nodes and say don't make this part of the load balancing infrastructure, not just for masteries. These are really they're, really orthogonal questions like should you schedule a pod behind a load balancer on your master when your master doesn't have load balancer actus, you probably shouldn't do that, but what about the non masters?

E

What about your? Your I would almost that's.

D

Up to the users they're setting that for somebody, and so if they're studying that they know why they're starting that, you probably know.

E

Yeah, maybe so I posted it so that we could have the the issue on the record, and people could argue about it. There I didn't intend for it to be here. Please go take a look at the issue and weigh in and if we can get to semantic agreement and I can tell Clayton what to name the annotation and how to document it.

A

Cool thanks, Tim.

A

So that was there's a couple more items but they're from lucky and Cal, who aren't here so I think the main thing is just that the services endpoints dual-stack phase, two PR- is ready for review those who are interested in reviewing that. Please take a look at it and then there's an FYI status document. You can look at if you're interested in the overall status of that effort.

E

Three weeks ago, trees.

C

Yeah on that subject, I have not had the time they're anticipated and the proxy stuff is falling behind so review, especially on the API, would be super helpful. Some of the like iptables proxy stuff is still up in the air, but the API is a bit of a fuss right now and that's usually.

G

C

For review sorry, it's just the KU proxy configuration API because it has to be passed multiple by notices and so on, but there's some kind of nasty bug in there right now, I'm trying to chase it. Okay,.

E

It's in my it's in my queue and I've got more than a few pings from lucky, so yeah.

C

Also, if anyone wants to chip in on that I'm concerns that I'm gonna be blocking this cuz I. Just don't have bandwidth right now, I'm not doing this full-time I.

E

Appreciate all the effort that you and every you're putting into this and I think it's also worth saying that while we wanted to make the release- and we will all try hard to make your lease- if it doesn't make the release- it's not the end of the world and nobody should be killing themselves over making images like there's another train, it's in a couple of months, but it's coming I.

C

Guess I'm just concerned about holding up something that I technically don't have as much of a stake in as other folks understood.

E

Your your, we are all volunteers mostly and I. Do I frown on heroics, so please don't you or anybody Royal you. Nobody should be killing themselves to get things done because their deadlines. Do you miss the boat there's another one.

A

Alright, that's the end of our agenda for today,.

A

Unless there's any other last-minute topics, I guess we get I have.

E

One we have a cube con prep coming up. We have until I think next week to decide whether maybe the end of this week to decide whether we want to do a maintainer track session. Again, this is cubic on San Diego. This is generally the most well attended of all the queue comes. The sick, Network, intro and deep dive in Barcelona was very well attended.

E

People want to do that again or these proposed to do it again. I think it's a good idea. We had like a crammed room for that session. Yes, do you want to lead it again? I, don't want to sign you up or it or do you want to? Let somebody else take the stage I'd.

C

Be happy but I don't want to step on anyone's too, as if anyone else wants to take over that.

I

C

Me down is like a fallback, alright,.

E

I'll write it sort of generically and say that we're looking to get new faces in the rotation so, but we have coverage like in a pinch. Alright I'll write that proposal this week. Then okay,.

G

O

A

A

All right, I'll give you all 15 minutes back and see any couples.