Kubernetes SIG Network, 3 Mar 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Network Meeting for 20220303

Description

Kubernetes SIG Network Meeting for 20220303

A

This is the sig network meeting for march 3rd 2022, and I forgot to ask: do we have anyone lined up for a round of issue triage.

A

I have this hand.

B

Yeah, you should be able to share now. I had forgotten to toggle the give me a second share window. Okay,.

C

Right so I appreciate everybody who ran through today and pinged bugs and closed out old issues. We have exactly one issue for triage and actually it's barely here for triage rob. Are you here.

C

Okay, well, this issue was a user who's, doing something interesting, trying to put their own annotations and finalizers on an endpoint slice. um I think everybody on the thread has disavowed him that that's a good idea.

C

The question that is open is: do we actually cover that in tests? What happens if, if somebody puts a finalizer on an endpoint slice and the controller fails to delete it or thinks it's deleting, it does cube proxy keep. Those endpoints seems like a really good corner case to make sure we have covered in tests so I'll leave this open.

C

I ping rob as the endpoint slice person, but if he wants to open a new issue or something that's fine, we'll leave it open to decide that anybody want to discuss this one seems pretty straightforward to me.

D

That is that too, that the final I said it's overrated.

C

So that I'm surprised, if that's true, I'm surprised- I mean somebody had to write code to like clear the finalizer explicitly, which I don't think would happen. So my guess is, if you put the finalists, I haven't gone and tried it yet. But if you put a finalizer on an endpoint slice and then you delete the service that the slice will stick around until the finalizer is removed, which is what I would expect to happen, but I'll bet that it might confuse cube proxy. Maybe the controller too.

D

C

If there is no service, there is no problem sure, but if I have a service with two slices and I delete one of them, it might very well keep those end points around, because I bet that cube proxy doesn't bother to look at the deletion timestamp.

C

Like it doesn't consider in process of being deleted, I don't know I I can't help but feel like we're missing um some hooks or something in the watch. Client. That makes it really clear. Not just it's been updated, but it has. The deletion process has been started and that's different from an update that sets the deletion timestamp the command.

D

I know we don't I'm wondering if we should. The common pattern on the update hooks is is to check if it has a timestamp to so it means that it's been deleted and people try usually.

C

But it seems like the sort of thing that everybody will forget, because it's not an explicit hook like it seems like there might be an interesting opportunity to provide a delete hook or you know, delete started hook so that the client could capture that as a distinct like just so that in documentation. When you look at it, that's a distinct thing from a plain old update like it's an important difference, I don't know we don't need to discuss it here.

C

It's just something that crossed my mind of like how many controllers out there will sort of do the wrong thing. If somebody puts a finalizer on a resource that isn't expected.

E

There is, uh there is another another common uh common problem with the way the way client works. So I don't think it's the client, the problem, but it's in all controllers. Everybody on on update, they start doing their dance right. The problem with is on update, doesn't mean a field that you are operating on has been updated.

E

So a common example is the updating, annotation updating labels triggers resynchronization all over the place. Even the default controllers that are part of kubernetes does not really do that than most places. So that's that's also something I've witnessed of all the places. There's no way you can ask the controller. Tell me if this field was updated or those fields are updated and nobody does the check sanity check off. Oh, I have old cached version. Is it different or not?.

C

So yeah yeah I mean.

F

C

These are maybe these are things that we should build into the example controllers more completely right, like first here's a filter process is. Does this update apply to me right yeah? If the this update is delete, call a different hook? If it's, you know not applicable just toss it in the cache and leave it alone,.

G

I'm I missed the start of this conversation, but this is this related to the bug that is: uh okay, the annotation and uh finalizer, and an endpoint slice. That's managed by endpoint slice controller. That's right! I I'm very confused by this use case.

C

So you you missed the the intro, which was, I think, we've disavowed this person of the idea that this is a reasonable way to use it. um But I don't know if this is covered in test cases. I don't know if cube proxy and or the various endpoint slice related controllers will actually kind of go crazy. If you put a finalizer on a slice, I wouldn't be surprised if they do- and I tagged you on the bug to say, hey rob. We should maybe put this into the tests.

G

So finalizer on the slice would prevent it from getting deleted, so would cube proxy keep it around.

C

Yeah is that the right thing to do.

G

G

Yeah yeah, it feels like someone trying to do something they shouldn't do and getting and their things breaking when, but yes, we should have a test. I don't know what we. What can we do? Can we re.

C

I mean I wouldn't I would cube proxy would ignore it. I think when we receive that update and we see the deletion time stamp set, we should just ignore that slice from then on.

D

Fair, but the problem, it's a slice, doesn't have a service. You don't have a problem.

C

But if you have a service with multiple slices and you delete a slice, then you will keep those old end points around right and and okay like there's a like the endpoint slice controller. Has this rebalancing logic at some point when you hit a threshold it'll pull all day and points out of one slice and put them into the other slices in the empty spots right, something like that right, rob and then it'll delete one of those slices.

C

The question is: if it doesn't clear the slice if it doesn't like empty the slice, but it just deletes it, it seems like it should work, but if somebody put a finalizer on it will those ips, like I just don't know what'll happen. Yeah.

G

Yeah and if, if it's pending deletion, I think it would still show up as an existing endpoint slice, which means our controller would try to potentially add endpoints. You know, I don't know if we have a check on deletion timestamp in the controller itself, so I can see you know it's an unexpected input to the system, but we should exactly in fact, if they're, adding.

A

This it's because they so.

H

Man, what you're describing.

A

Active during the deletion process right.

C

Right, in fact, maybe the api should just reject any updates once you've got the deletion time.

I

B

Do we reject the.

J

C

Updates: okay, that's good.

J

Tim and what do you describe right? You would have the same object occurring in two slices in worst case right, because it was not properly removed.

C

Sure, but I bet you could like if this were to trip up the controller, I bet you could synthesize a situation where you could crash the api server or something I.

J

Know yeah the semantics needs to be tightened up. That's what you're saying.

K

Yeah this this seems like the sort of thing where the controller ought to be able to say nobody is allowed to put a finalizer on this object, because it's my object and I don't want them to.

K

I

K

With this anyway, but.

E

Right you're asking for a lot of pain. That's that's a lot of pain right.

C

It feels like a like a escalation of conflict here right, but I mean how many other.

K

Controllers have exactly the same bug right now. I bet more than one.

E

True, I do have, I do have a essay about what you just said them about finalizer.

E

I don't think we should react until the thing is deleted, because you don't never know that the finalizer is there to do some pre-clean up right. Somebody needs to do some. Somebody said: do not remove this object until I do some prior cleanup. We are not in a position to say exactly what is this, so I acting on the aside from what then just said, which I somehow see where it's coming from I actually been thinking about that like there is, has to be a way where we say this or mine.

E

You cannot touch them, but commenting on your earlier comment about which do anything when we see lead time stamp. We don't know if this, if somebody depend on the fact that this object exists or in in a way we don't know if somebody depends on the fact that there are end points system, routes to, and everything works until this person decide that it's okay, now clean up.

K

The semantics of finalizers are, you can prevent the resource from disappearing from fcd, but but you're not allowed to assume that it continues working. Just because you have a.

C

Finalizer, are you no it it like. The semantics of finalizer is exactly what you said. It's dead that object is dead. The minute you delete it, just you're waiting for everybody else to figure it out.

C

So I I'm gonna disagree with cal, but we can take it to the bug. um I think first step is rob, go see what happens or see if it explodes yeah we'll do, um and this might be a fun one to bring back to, like, I don't know, sig arch or api machinery or something like hey. This is a problem that probably exists for other people. uh Is there a pattern that we could use to solve this right? It would be what would.

G

You start writing a book yeah. It would be very nice if you had some kind of guarantee that I I own this resource and no one else can touch it like others have referred to because.

C

I mean you could delete it. You could delete it in the controller and just clear the finalizers list unilaterally. That sounds like a terrible idea, but you could do it.

G

I mean, maybe not so terrible if it's managed by yeah.

C

Yes, you're juggling very sharp knives right now, yeah. Well, that's it for triage.

A

Yeah wild turn of events. One issue has filled up the full 15 minutes of triage, uh I'm going to shuffle up the agenda a little bit because I know that dan williams needs to leave halfway through, and this is last. So, um let's pull that to the top and quickly talk about the sct trick.

L

Yeah, I know I was about to say: let's you know if we don't get to it, that's fine, but no. The issue came up the other day because we were looking into some bugs that we had and uh it looks like cube proxy treats sctp contract the same way as it treats udp contract, but that seemed a little wrong because sctp is closer to tcp, and so I guess the question is around no there's been a lot of change in this area.

L

I think in cubeproxy and antonio you've been looking at some of that over the past year. Is that my read of this correct and is that the right behavior do we think.

D

So the the the code that is now I was good to cargo in because it was, there was a method that say contract a contract needed or something and sctp was there and I'm an ignorant on http. So I I carry over with that. But then there was another bag and I was checking with one person that perhaps in the net filter call and the contract filter code checks the session. So the kernel should be able to track the station and and clear the contract entry.

D

So he fixed the back and- and my understanding is that we shouldn't clean the entries here.

J

Was the first one to say, I think it was pear who said no yeah I mean so. The problem with sctp is really that it has this multi multiple endpoint thing going on right, but you you can set two addresses, but it's supposed to go to the same state and the many applications require that you have that set as the tip is a very I mean lars is here as well.

J

uh He has a lot of opinion on satp when both satp and a multi-part tcp is very tricky to implement through a contract because read you need to go and read what happens inside the protocol. You, you cannot get all the information just from what comes in the intcp. I mean in the sin that you get or any of the people that matters that you have the five tuple uh with both multiples disappear and sntp. You have additional information that is carried in pdus when the first recession is set up, they're, very tough, to.

D

Treat properly yeah but our services we only publish one, I think I mean I I I I'm a naive on http, so I'm just saying what I learned during the past, but in this case I agree.

M

uh I don't think multi-homing is, is an issue to be considered because it's not supported in kubernetes, but one important fact is that very many http clients binds to to a certain port uh tcp clients never do that. They they use the fm aerial port, but satp clients quite often binds to the same port and it reduces it for reconnects all over again and I believe it would be the same problem with tcp. If people did that, but they don't.

J

I'll disappear, you have the time weight and well specified and close weight right. So so the loca, the local stack, will take care of it to make sure that you don't even try to set it up, it will block it there, but yeah I mean you see. We have a couple of udp test examples right when we uh that that blind binds the client port.

M

Yeah, that's common in udp and it's unfortunately common in sdp also.

C

Does that answer the question at hand then, do we need to keep sctp contract logic or fix it.

K

It sounds like we should add tcp tests that explicitly bind to the source port, because we know that the bug report involves people binding the source port. So maybe we should just add tcp tests that do that and see if it works for the existing code and if it doesn't then figure out what we need to do to.

J

Make those work without.

K

Breaking anything else and then.

J

K

Should work with sctp.

J

The local client just should stop you from doing it. The second time right, uh you should not even get to the service.

K

No, the issue is something like.

L

It was a service deletion right dan.

K

Yeah, it's like the client is is is connected to the service ip. The service back end changes.

K

L

B

Forget the details they deleted.

L

The service and then recreated it and expected it to either I forget which one it was not work or work, and it was doing the opposite of what they expected um and we were like. We don't we're, not sure what is expected upstream, like what should happen in that case,.

K

Maybe we should make sure that the sctp experts are cc'd on this.

L

Issue, probably I mean in general, though, what is our expectation if um around draining a service? So if you have a connection from a client through a service to a back-end pod, and then you delete the service itself, what should happen?

L

Do we expect that connection to exist until the actual pod itself goes away, or do we expect that, when the service is deleted, that all connections to any pods backing that service are dead in tcp, you mean I'm saying in everything really and yeah? The answer is different.

C

L

C

J

No, not necessarily no no.

D

K

J

Type of firewall rule that you remove the service, then you should tear everything down right, that you should continue the service. That's ongoing.

D

This is based on on the concept of sessions, so you don't break the session that this establishes and you in udp you don't know djs and we are no. It's.

E

The same both if you think about it, if you think about attack as a control, plane, you're saying I am no more interested in trouting traffic that way, existing traffic should.

F

Immediately because the fact that it takes seconds to do that, I agree with you in.

C

Hypothetical, but in practice with a tcp session, I can't like break the link and then stick something new in and start intercepting traffic with the udp session. I can't session I can so I think the correct answer is different, depending on whether it's udp or tcp.

L

The other question I had was: how do we gracefully drain a service? Is that where you just you, keep the service around, you kill all the pods to that service in the back end, and then you delete the service or like how do you stop accepting service traffic but then allow existing connections to gracefully age out? You.

K

You kill all the pods, but they have termination grace whatever, okay and because they're terminating the end points go away and because the endpoints go away, the service stops accepting new connections, but we don't delete the contract entries. So the existing connections stay there until the pods finish them and exit for their grace period. Okay, but.

L

Currently, for sctp, we do delete the endpoints or the contract entries when the endpoints go away. Yes,.

D

But that's that's! That's there because we were carrying over with the udp. I mean this. I don't know how this starts.

J

The the reason is simple right, because if you have something so with udp- and there is stp is a little bit different, but the udp there is no guarantee that you have traffic coming back that way, so you have unidirected udp streams, which means that you can have something sending and it will never know that the server service is gone. So there's nothing that can recreate the stream, because the contract you're already have an entry in there. So you need to delete that entry.

J

But going back to the first office, I mean, if you say that the service should for the xx, inflow should continue, and then, if you want to have them removed right that they will do with that, you would do with a network policy entry. Then right, you will remove the ability for these two objects to talk to each other, and then you would have to clean up the contractor.

J

K

But I'm not sure we have a well-specified enough question here.

L

Yeah um I mean I guess my question is: is the current behavior we have for clearing sctp contract on endpoint deletion, correct, that's the basic question and then I guess I would. The other question I had about service draining is more general.

J

The answer the answer there is: does the contractor handle satp or tcp right? You will get the reset sent back. If the end point is gone, you will get the reset sent back or you will give up because you don't get the axe back, so the stuff will clean itself up after well.

J

It can be minutes or hours depending on what's there right, but if you have a unidirectional unidirectional scheme like you can get with the udp, the sender can keep on sending there's no receiver and there's no way that the the infrastructure will tell him that there's no one receiving this anymore, there's no knack or anything going right or sntp. There is state control, so there needs to be something that goes back and sort of say the equivalence of a reset right.

J

This is not there. That's the contractor, react on that and in that case, clean.

C

Up the entry, does this get more complicated, I mean uh with user space sctp. We we took this sort of middle ground on the sctp support that we weren't going to load the sctp kernel, module right and- and so we we sort of tried to be a little bit of half half of one and a half of the other mode.

C

Like does that.

K

This so so the that mostly doesn't matter because that's about what pods are doing in their own name space, whereas this is about what we do with the sctp packets that are passing through the the host network name, space.

C

My concern was whether there would be a side effect of like doing anything, contract sctp related that might break user spaces. The only thing we can't do is open a socket.

K

We can do anything else with sctp.

E

I think we're completing multiple things together, what life cycle service life cycle connection life cycle? Absolutely so the answer is different depends even tcp the artist there are no guarantees, gonna get rst or friend or whatever, and that's why keeper.

N

Lives are configured and and.

E

J

J

So there is traffic going both ways. Yes,.

E

And and udb, if you decide to use udp, you are consciously saying that I am not. I have an upper level app protocol where I expect a knack or an act or whatever so yeah, so even with tcp, by the way, like the fact that this thing works is astonishing, just to sit there like, I said it. Yes, the fact that tcp works most of the times is actually astonishing. If you read the specs and the way connection, tear down and rst and fanon holds this entire dance and all those things expected to work yeah.

E

J

Lars had a pretty good answer. You set up a completely new five tuple right. Then you have a fairly large face on that and the old one will die, uh sometimes a nice death and sometimes a painful death right yeah. I think it's six hours. It can stay in the contractor. Today.

C

So is the answer, then sctp should taste more like tcp. In this regard. It does have a protocol level termination.

J

Yeah, I'm just scared. What happens? I need to look into it better. I think I mean large might even know the combination of encryption and and connection state. Is it as in tcp, where you actually can see it from the outside or or is it not.

I

J

And can I see the state changes in an encrypted connection, or can I just see that his pack is going back and forth.

M

I don't call that here, you ask me.

J

I don't know it either. I will, I think, we'll have to to look.

C

Okay, dan was there a number that you want to send everybody to.

L

uh Not quite yet, but um somebody from red, hat's, gonna file, an issue or a pr, and we can follow up the discussion there.

O

Yeah I'll do it.

L

Yeah, there's syria good. Okay thanks, syria.

O

And stdp does have states, at least on contract. You can see the established uh shutdown shutdown act, which is same as the finn finnac in tcp. The only thing that's confusing for me is what um fair was saying about reset being sent or when in time.

J

If you have the axe, then it's really do you still see them in a.

J

I guess you will you know again that that case we should be good, then you should be able to treat them as tcp and we should check.

C

It out, I like the idea of adding a test for tcp with a client, bind port and see what happens.

M

Yeah there was an issue about that when a client only had like four ports to use- and it hammered like crazy on those portions that the invalid entry state, of course, I can't find it now, but I will try. I remember that one vaguely.

M

I don't know if it was sold or if it was just abandoned.

J

I mean so to bind the port works. Fine. The question is just to guarantee the uniqueness of the connection, so you don't get the local stack to go nuts on you.

A

Well, it sounds like uh we're reaching and end on this one. um I think we should move on to uh syria and winship.

K

Yeah so um surya was was doing uh internal traffic policy and over kubernetes. She asked me to deal with this topic because it's 11 00 pm there and she's half asleep, um but so the the big question came down to whether node port connections are internal or external and in particular, if a pi connects to a node port on the same node. Is that internal or external? And if a pod connects to a node port on a different node?

K

Is that internal or external and the answers seem to be um awful and because, with with external ip and load balancer, no matter where you connect from? If a pod connects to the ip, it gets redirected and it's treated as an internal connection, not an external connection. So you would sort of expect node to work the same way and it can work that way when you connect to a node port on the same node.

K

But when you connect on a different node like everything falls apart, it gets treated as external, which seems extremely inconsistent, but at the same time it's not clear how you would make it work any other way.

K

So we just sort of wanted opinions, like is the right answer that node ports behave inconsistently or should we just say, internal traffic policy actually only applies to pod to cluster ip traffic and and not anything else, um since that's sort of the the canonical use case, although that still leaves open the question of what external traffic policy apply to these connections if they weren't internal, so that might not really be an answer.

C

So my gut feeling, I read your questions before the meeting and I chewed on them for a little bit. um My my gut feeling is: they are external like. If we were refactoring the service api from scratch, we would likely have a different resource, or at least a different stanza of resource for cluster ip and for node port and for load balancers, and each of those would have different policy statements, and so I think internal traffic policy would only apply to cluster ip traffic.

C

Now I haven't thought through the implication, the implementation implication, um but that that's where I would default go, and I see boys said the same thing on chat.

C

Is there a reason that that would be wrong if you've looked at the implementation? So okay wait.

K

So pod to node port on the same node you're saying is external. Yes, okay,.

P

Yeah, my reasoning is like from a use case perspective is that the person opening up the node port probably is not thinking about internal traffic.

K

Right, okay, so, but currently, if you do pod and connect to a load, balancer ip or an external ip, that's treated as internal, not external.

C

That seems broken like I know. I know the code path that you're talking about there, but that seems broken. Can you repeat that sorry, the last.

K

So, okay, so the difference between internal and external traffic policy is where the client is it's not about how they connect it's about who the client is. So when a pod connects to anything except that a remote node port, it's treated as internal traffic.

E

That includes going through an external load balancing back.

K

If you have the gcp azure kind of load balancers, where we know the id and short circuit, yes,.

Q

Wow, why why is this terminal notepad.

C

I think it's about intention. It's a like when you are making a decision to open a node port, it's to bring traffic in from outside, and so I think, from an intent point of view. It would be different than internal traffic policy. Yeah.

D

But uh when you talk about topology you you talk about locality, we're not.

K

Talking about topology, don't talk about topology, it's confusing enough already. Sorry.

D

Then, okay, I went to the next okay, that's the intersection between topology and polish. Okay. I can.

K

So so so tim I get what you're saying, but that is that's the opposite of how we defined internal versus external. So far like right now it's about who the client is not how they accessed the service, so we could possibly change that, especially for internal, although I mean external traffic policy is like has been ga forever right, so I don't know if we can change that.

C

Don't we have a weird uh corner case? Is this the same corner case with external traffic policy and no like node port's the thing that falls in the crack all the time right? um Does that trip this up? Also, I'm not sure what you mean like if I, uh if I set external traffic policy local and I go to another node's node port- is that external traffic. At that point,.

K

Yes, that that's that's where this this problem started, your node's node ports are, if you connect to it, it's treated as internal, but somebody else's. Node ports are it's treated as external because it doesn't recognize the source anymore. At that point,.

E

But if it goes.

K

I

The load balancer.

E

It's still internal, that's that's broken.

C

I I agree that I think that that's broken. I think that, like as as we're staring at how to slice up the um endpoint assignments and the policy decisions within that code path, we should be clearer about these ones are internal ones.

K

If we want to change internal versus external based on the destination rather than based on the source, that would actually simplify the code, a ton, but it means then that you lose the case of the node, can connect to the load balancer ip and not have to worry about whether there are local, endpoints or not.

K

I guess that's a special case.

J

K

J

That there is no address translation right and stuff like that. I I I wouldn't be surprised at some point. If you see things for v4, would use map t to map it to a v6 address and then map it back and all these things to to use the sources as to classify unless you're on the node, where that source is.

K

Right, that's the thing: if you, if you're not on the node with the source, then it's external either way right.

J

K

J

I mean, then you would you could solve it by. I mean having some way to add metadata right in the packet when it goes over, but I mean you cannot require. We cannot require that sort of.

C

So there's one there's the one classic corner case for same node, node port, which is the local host thing that doesn't work in ipvs, but does work in iptables unless you disable it and that's like I want to access a local host registry or something right. That's the one use case that I've seen for this in the past.

C

Would that break if we changed same node, node port to be external traffic policy instead of internal.

D

Yeah, the problem is that people is using node ports to at least local services on the class.

K

I mean, presumably somebody who is using connecting to node ports over a local host would not set external traffic policy on their.

D

Service in general I mean you can see manifest there, that they create that charm or whatever and they install and they connect to the node part in the same class. So we in kind we have a lot of this, but in the same cluster.

C

But what I'm saying is today, if I call out if I get my pod status host ip right, node node ip yeah.

P

C

And I connected that on a node port today that is considered internal traffic, not external traffic.

D

Yeah- and why did you consider that traffic external I.

C

I don't, I think we just didn't think about it. I think I think this is a corner case that we didn't recognize as different. I would have expected it to be external because it would have come into a node port. Therefore, the intention would be I'm publishing it outside of my node.

D

I think that I mean I don't agree with that, but because you you that is, is assuming that no part is only accessible externally.

C

It's not that it's only, but that isn't why it exists.

D

Yeah, but we are dealing with the problem of what the people did, the.

E

Problem the problem was according to how things are described. It's not that we misnamed a thing and documentation. Clarity will fix it or some big note somewhere, it's the intent is broken, so either we treat all internal as internal or all external as external, but right now the idea that, if I am a part setting on this note talking to a notepad on another note, I'm considered external. But if I go through the load balancer, I'm back um internal as a behavioral there's a broken behavior. It's like I don't.

O

E

We if we have the wrong behavior, that's fine like if we have a behavior, that's called internal, but we call it external, that's fine. We can explain that, but.

C

So here's here's here's! What I'm thinking, though, um the fact that it's different, whether you talk to a local node port, same node, node, port or a different node, node port seems like the big red flag, like it shouldn't, be different.

C

The load balancer that just feels broken, and I think we justify that that is in fact broken, um and uh I mean especially because it's different on different implementations right on a proxy-ish load. Balancer, you don't get that um and so then the question is: if we're going to fix the big red flag here, which way do we fix it? We can't say that node port traffic to a different node is local, because you can't preserve client ip.

C

So therefore it has to be considered external right ergo. I think we should say same node. Node port is also external and we bring consistency to node port. We uh fix the glitch on the load, balancer ip and tell people it's about your intent, there's a long standing to do also in there somewhere. That says, should we apply the load. Balancer source ranges to node ports right so like there are policy things that apply to load balancers, but not node, ports that we have historically, you know not come back and fixed either.

K

Although I feel like at that point, I mean at this point that would count as an api change right, like oh.

C

Yeah, I don't think we can fix it now. I think we should just remove the to-do.

D

So you're saying it's internal based on the source, ip.

K

D

C

K

C

Yeah because we have a it's the: um what's it the api, the interfaces.

D

Local traffic detector.

C

Local traffic detector. Thank you. Yes,.

D

What happens if my pod reaches the node port in another node with the source, ip.

C

It's external because the local traffic detector says this was not my mission. This is not this machine. Therefore, it's external.

K

It's undefined because some of the local traffic detectors, if it made it there unmasqueraded, would still call it local, but.

F

C

Not supposed to.

C

The point of I remember going through this process last year or the year before the local traffic detector was sort of changed so that or was introduced so that we could move away from the idea of a cluster cider into. I only really need to consider this machine.

C

N

I think the answer.

C

Is node? Ports are external load, balancer ips are external and that's how our policy statement should go. The only real caveat here is: can we think of a case where that would actually break something?

C

That is a reasonable thing to do. It.

E

Will break I just remember that connection weight thing, notepad, opener, stuff, that we were talking about it's actually triggered by somebody had like pair node registry cache and they connect to it via pods and stuff like that, and I can assure you it's to them- it's localhost the way they think about. It is localhost. I don't know if they have policy on them or not, but I'm quite sure if the policy.

K

So one of the solutions we were thinking about is saying that the whole idea of a pod connecting to a node port is kind of ridiculous and therefore we don't mind having the ridiculous answer that if you can match your own node port, it's look it's internal, but if you connect to someone else's node port, it's external.

C

I don't want to write that doc. Man.

K

All right, I will I'll think about this more and nobody planned.

N

Email eventually, I guess.

K

What we can, what we can get away with doing that won't break existing use cases and stuff.

C

I have a pretty strong lean on this, like I I'm hard-pressed to come up with a case even a synthetic case where you would be connecting to a local host node port, and the current policy is important to you.

K

That's a good point.

C

Like I, just most, people would just not be using policy for that.

E

I wonder if anybody who's using a multi-tenancy using hyper, hypervisors container and so on, what do they do about? What do they expect?

E

Because if they are using somebody's using like hyper hypervisor container uh firecracker whatever, and they are running two containers using two different tenants on the same note and you can loop back traffic, I wonder if they know.

E

C

Let's start something last last comment: when we move on, I don't know if we have any more agenda, but um internal traffic policy is very new anyway right, so before that there there was, it was either I'm getting. When I talked to a same node, node port, I was going to get global behavior cluster scope behavior anyway, because there wasn't a way to specify otherwise right.

C

K

I mean if external traffic policy got applied to the connection, then you would get the external traffic.

C

Okay, well, let's see if we can find a use case where this matters, I'm not seeing it right now.

H

Yeah, I don't think so either and I know at least in oven k. It made external traffic policy a nightmare to have to account for pod to node port like it was an absolute pain, and this would make it a lot easier. So.

L

Quick clarification, oh go serious. Sorry.

O

No, I was about to say that's because we didn't ask dan venture before we implemented that, because danmanship solution, where um no to a pod to other node is external, it would have simplified things.

H

uh I think we got told not to do that, but yeah.

C

So the short answer here is: whenever you have a question, just ask danmanship.

H

That's what we usually do at red hat, yeah.

C

Does it involve some esoteric corner case of cube proxy dan winship's fault.

L

Quick clarification for dan winship. um I had written that second issue there or second item there, and then you crossed that out and said, but it can't be for compat, but I thought that is exactly what tim said earlier. Is it just that we can't do that? I.

K

I I think he was saying that we should have done it that way, but, okay, if at this point like if we were to change it like people, are depending on the current behavior, presumably.

C

Oh actually, I was wondering if pod lb being considered external is something we could change, but I hadn't thought about the compat issue there. What I was saying was completely broken was we couldn't change?

C

Load, balancer source ranges to apply to node port, like that, would break people for sure.

K

I guess the load balance or short circuit thing. The big thing that it wants to to preserve is that if a potter node connects to the load balancer ip, when it's on a node that doesn't have an endpoint, it shouldn't be rejected, and it currently does that by treating the connection as local. But I guess it could also.

K

We want to transparently treat it as though the packet actually went out to the external load balancer rather than being redirected. So we could do that without treating it.

C

As local, I guess right I mean the the reason this logic exists in the first place is because on those vipish systems, that's actually a local address anyway. So if we don't handle it, it will go out to the host's network interface and then try to connect to something that isn't running on the host's ip right and it will just fail completely.

A

So we do have one more agenda topic. um We got 10 minutes left so uh andrew. I think those are all yours.

H

I don't need the 10 minutes. It's just. um I posted a pr implementing or starting to implement the amp api and associated crds. So if folks want to take a look, feel free, there's two main unresolved, you can still find in the cap that we want to fix in this api pr. So keep that in mind when you're reviewing uh that's all I have, though, thanks.

A

That's great I'll definitely take a look. Probably gonna start prototyping implementation of it. Soon too,.

D

Awesome. Thank you. I still want to discuss more about the internalist democracy.

D

Because, for me, internal is the traffic that is internal to the cluster, the service ips, the load, the load balances like you may know, but the cluster ap. They know they p. Those are internal ips.

K

So the the definition, the way that internal versus autonomic policy was defined in past, is that it depends on who sends it not where you send it to so like we're stuck with that pretty much like.

K

Unless we want to deprecate external traffic policy and redo it no.

D

No, no it's just because we were all I mean I I was surprised. Everybody here has a different idea. I mean so for me, that's the internal definition is is like when you have a an a.s or something everything that is inside your network is, is internal.

D

So that's because I.

O

Don't think you're the only one who feels that way. I just read the cap and changed my way of thinking. To be honest,.

Q

I mean I come. The other points are bodies too.

C

I mean we have what we have historically defined or left undefined um or implicitly defined via code, but as long as we can convince ourselves that we're not going to break users, we can define it how we think it should be defined as long as we have reasonable confidence that we're not going to break users and an escape plan if it turns out we're wrong and we broke them anyway,.

D

The the origin of this is the the internal external traffic policy change that dan quinchi is going to run no or what andrew.

K

I'm just fixing an edge case, which I mean you should go review that pr again by the way.

C

Yeah, okay, I have a lot of pr's to review, I'm coming out the other side of a long process um they.

C

So if, if I talk to a node port on my same machine, then I can apply, I could apply. Let me think for a second before I speak.

C

That's currently we consider that internal, and so we don't apply external traffic policy right yeah. Well, are you a pod or are you an outside source? You just said: if I talk to a node, I'm a pod and I talk to an import on my node, um then I can preserve client ip, because that will work just fine, regardless of whether the external traffic policy or the regardless of whether internal traffic policy is cluster or local right, because if it was local, it's fine.

C

If it's cluster, it's also fine, because the source ip will be like all. The semantics are correct right. If I talk to a node port on a different node and it forwards me off to somewhere else, it's going to, at least in the cube proxy implementation, it's going to masquerade!

C

So I'm going to break internal cluster semantics, so that cannot be internal right.

C

So this is how we end up in the gray space.

H

Right same, it's also a use.

F

H

Right like why why, if you're a pod trying to access this service you're, you should be using a cluster ip anyway you're already in the overlay. Why well.

P

I mean people might have a consistency, they might be migrating right, they might be going from vm to pod, based applications.

C

Sure and we've seen things with people using external service discovery stuff where they just registered the node ports, yeah. Okay,.

D

C

Are reasons people would end up here, they're not great reasons, but there are reasons: okay, fair enough.

C

I'm just gonna have to think about the use cases. I can't come up with a good one. You.

J

Come back to is it better than he has to describe the existing semantics and describe it with his warts and live with it and fix it in a new generation or try to fix it.

C

What let me ask a separate question this. We know the cube proxy implementation. Well, what about other implementations? Do they retain the same behavior or did they internalize sorry implemented internal differently.

I

So so I can speak on the ipvs one a little bit like. It takes a more naive approach where it considers like ex, like the low bouncer, ip and external eye piece to be always external, even if you hit it from this from internal um and it always considers like cluster ip internal.

I

So it actually just does it based on the use of the ip and not the source, but that also has led to like a lot of people complaining about um like if you, if you create a load bouncer and you set external traffic policy to local.

I

But then you want to use that same ip from inside the cluster. Then your packets get dropped. If you land on a node that doesn't have the endpoint, which complicates.

K

I mean that that last part should be treated as separate from the question of internal versus external, like if you connect to the a load balancer ip from a node. That has no end points like it should we we should just forward it to yeah.

I

The problem is that people assume that if I connect to this little bouncer ip from a mic from inside my cluster, it should it should route out to the load balancer and back in, but the ipvs rule binds on the network stack and it intercepts it before it even leaves, and then right, that's the same as the the.

C

Vip load, balancer, behavior, right and the assumption with external traffic policy. Local, is that your load, balancing stack, will not send you to a node that doesn't have end points so as to avoid the black hole.

C

So in this case, if you are a pod on one of those nodes, you are subject to that black hole, and so maybe that's the reason that same node is internal.

K

So it seems like if we just.

K

If you send to a a bad node, a node with no end points, if it just treated the service as though it was cluster, that would have no externally visible effect. Oh no because there's the problem that that the load balancer might be slow to catch up with endpoint changes and might send okay, no nevermind.

C

And andrew, is there ever a resolution to those issues or do we just tell users they're holding it wrong.

I

Yeah, I don't think we've ever fixed um the like, if you, if you connect to a low bouncer ip on the pod and there's no pods on that same node, like that traffic is still dropped and there aren't a lot of like we've discussed workarounds of like uh defaulting to ip tables for that kind of case, but like they all, seemed really hacky like it's really hard to work around the fact that you have the ip binded on the ips stack.

B

I

So yeah that hasn't been resolved.

C

I mean you could maybe do crazy crap with policy routing to send it out a different table.

I

C

Let me erase erase that from the recording. Please.

K

At that point, it seems like it would be easier to do like you know, split horizon dns so that your local pods get the cluster ip. Instead of that ip.

M

For ipv, yes, there is an open issue on this, or it's a pull request actually with the implementation of a very simple solution.

C

um Maybe you can link it to the mailing list or something or dan. Do you have an issue where you're discussing this there we go.

O

All right also just one quick thing: uh can we have a decision node where someone is going to start documentation where we discuss more because the way ovnk does? Is we ask what does q proxy do and it would be good to get some of these things explicitly defined in a documentation for q proxy so that we can just point users saying upstream: does it this way? That's why we do it this way.

C

Yeah, we should get interesting.

K

Yeah I'll submit an update to the the big dock in the caping repo of how cute proxy works as well.

O

Thank you so much don.

C

All right I'll go read the issue that lars just linked. Thank you. um Fun topic.

B

All right, thanks for coming everybody thanks all see you in two weeks.

B