Kubernetes SIG Network, 14 Jun 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Multi-Network community sync for 20230614

Description

Multi-Network community sync for 20230614

A

um Okay, welcome everyone. uh The multi-networking community sick meeting um today is June 14th and we are continuing our discussion on a multi-network. um Please add your agenda items. If you have other topics to to discuss. uh Could it be in front of the discussion because it's going to take like the most of the rest of the meeting? Probably uh let's I think Pete. Did you added that one or I'm not sure who added the first one.

B

Yeah I I added the first one, it's just really to flag that I put a section into the dock, but dra how this fits together, um I'm happy to discuss things in any order.

B

The thing that I would like to do, though, is is point that out to people and say, there's a section there please feedback's, very welcome uh I, don't think it's completely coherent or that it hangs together yet but I'm happy to talk through it and if anyone's got wants to comment on it and one of the things I've got wrong, it's just things: I could do better, that'd be great yeah.

A

I saw it as well: I I started reading it through, but I apologize, I I haven't had a chance to kind of find that finish it yet so uh yeah I will try to get through it and and and read through it and give you some feedback on that. um Yep.

B

um I'm happy to give you a few minutes. If you want or I can just leave it, but I think the key thing is, it does feel like Dr Ray should work pretty well and I. Think if we write the right kind of controller, we could save ourselves writing a lot of core kubernetes code that maybe we don't need, and that would be great if.

A

You want yeah, we can do that if you want to. If you want me to share or do you or do you want to.

B

Share the picture um and then I can talk through the and then I can kind of talk through it from the dock and then I can kind of talk through that.

A

Can I give you just like 10 to 15 minutes? Is that enough? Yeah.

B

Absolutely I: okay, I! Don't want to turn this into lots of bike shading, because there's lots of technical details here that aren't 100 and wait it's better to discuss them offline, but okay, okay! So let me start explaining I think the key thing um sorry I'm fighting Zoom a little which has decided to make itself really small, which doesn't help. Give me a moment.

B

There we go right so a couple of things here: firstly, um there's a bit of a link here to all the different pieces here here that there exist, so we've got pods, so we've got pods and pod networks. Pods are some kind of link to pod Network, that's fairly uncontroversial blood networks have got some custom resource and I've got in my mind, a cni implementation, so the customer resource would have some kind of network attachment definition type stuff in it.

B

In that case, um the other uh case that exists now is that ports kind of resource claims which say oh I, want to have a resource like this, which could be a GPU device driver. In this case, the resource claim would correspond to I want to be on this network, and then there are various links from that resource claim which can point to custom resources and provide arbitrary parameters. So all of those things are provided by dra and the flow that I am thinking we could make work.

B

Is you instantiate a pod at the point where you create a pod, there's already some flaws. That say, if you've got resource, claim, templates, go and add resource claims to the Pod. We could extend that so that for pod, for if you're attached to networks, we add a resource claim for each Network um with potentially customer resources you might want to have to the ability to put your own resource claim in there rather than having a resource claim template from the custom resource. uh That, then, gives you a couple of things.

B

It gives you scheduling, because that resource claim can affect where the Pod is scheduled and because this is relatively arbitrary, it can do all kinds of arbitrary scheduling complexity, and it also gives you some interaction with the kublic, because the Kubler does access to these fields. At the moment, it's just passing them across into a plugin.

B

The minimal way of making this work would be that we write a driver, I'm going to say cni again, because that's the way, I'm thinking about things that would um the data that it would set up and the scheduling it would do would be very minimal. It would read the relevant resources of custom resources. It would then put them into the data blob that is going to be past the plugin on the kublet when it gets the Kubler that data is passed across. So no core changes, yet we've just written some drivers.

B

The core changes to this would be well. We need to have the Pod Network feels to actually set this stuff up and then some questions about how the reserve claims go in there and at the far end we actually have to set the networks up on the Node. But we've got the information into the node and I've done the scheduling stuff again with what's already in Dre.

B

um So it feels like there is a path here where there's a number of things a little tricky, but there it feels like there's a path that could make all this much easier and do a bunch of the work for us.

B

We wouldn't have to write any scheduling code, for example, we'd be able to just reuse the scheduling logic from um dra, and we wouldn't have to write a bunch of logic to say or we'd have to write much less logic, something much less core kubernetes logic to get parameters onto the node, because dra allows you to has got a path for doing that.

B

um There are a couple of open questions, I'm not going to go into the detail of the areas where I said. Okay, this bit I, don't really know about, but something like this could work.

B

um One of the more interesting points is dr8 is problematic if your, if your cluster needs to be functional before dra is up and running and because dra requires you, installing controllers there's a kind of chicken and egg problem here and the right answer for that is probably to say. Well, you can have a simplified Network where the default Network would always be a simplified Network that just doesn't use any of this. The way that that works is if you've got a simplified Network. It doesn't use Dre at all.

B

The only thing that gets down for Kubler is you're on this network you're off this network and it's down to the Kubler to have some understanding of that which I think for the default network is fine.

B

um It does mean that the default Network would be very much different to other networks and I. Think that's it for a summary. If it's so promising lots of details still to work out and I'm very interested in feedback on what I've written here.

A

Just quick question in in this picture: you provide both objects, you specify in the Pod spec both objects, I.

B

Think the Pod spec will have a link to pod Network. The resource claim um I think we're going to have to be able to support it manually for complex use cases, but I think it is also better if it can be set up automatically for non-complex use cases. So.

A

The kind of thing whether your pod Network becomes the resource, because that's what I have in my mind all the time.

B

Sorry, it becomes which resource you mean. The resource claim.

A

But Network becomes the resource claim yeah because the whole, but that's what kind of I would think in the long, so the the the special resource handling, let's say, qos, which is not on the on anywhere on the roadmap. Yet, but let's bear with me what I would imagine right, because the scheduler already I I see a path without any of this. What I'm saying is in scheduler, if I see in a podsback pod, Network I ask this special controller that we write for this purpose.

A

Basically, and then it does what you're saying about the cluster creation having the chicken egg issue. That goes away, because we create that controller that the default, the reference controller for the default, Network or very basic it shouldn't be for default. It would do the basic thing that we do today, which would be just okay.

A

This network is available or not, and there is a some way in this controller that can tell you uh this node has it or this this one this node doesn't have so basically that is all there and we leverage the whole code that the array created pod and dra code understands. Oh, if it's support, network I will go and ask this driver, which is hard coded because it's standard right. So there is that one and then the same way in cubelet. Oh, it's a pod, Network I'll just use the dra path to pass.

A

This all arguments to you.

B

If you have a controller that sets up the network, the control controller can't work until the default network is working.

A

That part, is that part, yes, that that party will be there because you always have to have KCM uh before you do anything KCM has to work in that static pod? Usually so, and this is where the controller would run, because this will be a default controller inside uh core kubernetes, it will not be some or maybe it would be, but then you would have to maybe start that guy some some other way so something to discuss. I'm, not saying you might be right.

A

There is something something around that one we can either include it in KCM and then that's problem solved, and then we flag it whether you enable it or not- and you can- you can start up your own, but then you have to deal with the chicken issue and but but still what I'm saying is I would not want to create a separate uh apis, because this is very awkward what you're presenting you.

A

You have a pod reference pod network, but if you want to have additional resources and management for resources for for that, Network Define, the resource claim. This is very awkward I. This is very clanky, I I. Would not want that.

B

If you have pod Network I mean all I'm. Thinking is that if you have a pod, Network you're going to have to have something that says what resources it needs now, you can have a default that says if, but if you just have a pod, Network and knows um uh and no resource claim attached to pod, then you have a controller that puts that resource claim in and says you're on this network I'm going to build a resource claim based on this.

A

Or or I think yeah exactly, we could do something like that, and this is what I wanted to talk about, because uh because I do want to talk about the the like continue. Our discussion on the cap, which was from from on from the API point of view, was about the attachment to the Pod right so yeah.

A

At one point we can extend it so that either what you're saying- maybe maybe that should be the case right, pod Network, and maybe this will create a resource claim so that we don't have to do some special code in the inside under the hood. We just we just already have the resource claim right uh for it, and and um we create one pair- maybe interface, per attachment to a pod, something along those lines. Maybe right, uh uh I would not want to uh this I. Don't like this. This Arrow here this guy.

A

If this Arrow were coming from pod Network, that's fine, but if it goes.

B

From so that is that's a pod specific thing, though right.

A

um It, of course, if you have a resource, claim, that's completely independent from from networking yeah go ahead. This is like you, you want to use some I, don't know fpga and it's not a network and, and you do or Nvidia stuff, like Graphics stuff, you want to share it and divide it in some ways.

C

One question yeah: if I, if I may, if I may make some comments, so the first thing is here that this Arrow doesn't have to be something that the user needs to deal with themselves.

C

Part in in the document uh from Peter is is a solution where we just have the whole API as you define it, and then generating resource claim is done under the hood, so the user doesn't interact with it. It doesn't yeah and that I think that is indeed the preferred solution.

C

So this by name and resource claims field is kind of outdated, already, I think the the actual proposal now is to generate that dynamically. There is some ideas: how to store that link in the Pod status. So if users care they can look it up, but they don't need to set this up themselves.

C

Address is one of your concerns, the other. Do you have a comment that I would like to make? Is that if you do only one resource claim per pod Network, you basically allocate resources for the entire network, and then you may get some limitations like this network is available on on these nodes. But then your ability to influence pot scheduling is fairly limited.

C

You can basically say any number of ports can run on this node using this network, but simulating things like a certain number of ports per node becomes tricky and an allocating additional Hardware like in like what I all uh thingy that that becomes harder. That's why a per Port resource claim does make sense for I guess at least some drivers or.

A

Some networks, one that we treated as a additional condition for the scheduler, so I have a pod Network, which says: okay, which network. Let's say um it's disconnected in some ways it can be: maybe it can be replaced, but basically the basic kind of uh schedule.

A

The controller for the scheduler will be saying uh just based on some Flags, very basic flags on saying, okay, this network is available on this node, but then I'm gonna have another resource claim for that Network, because I I want to do something like that and that resource claim will have additional.

A

um This will be another like another round of scheduler, because this guy is going to have some additional kind of controller or not. Or are you all saying that this pod, Networks uh dra stuff will go through this Auto created resource claim.

C

A

That's the idea, yeah I think that's the idea. Okay, I hear you so yeah. Let me let me read through that because um yeah that might be as well then, but then uh what will be the option for for a user you're right, because then you can only have one for that Network right, uh so so then kind of you screw it. If you want to have I, don't know based on bandwidth, qos and some other pieces right. Do you have to like for each parameter you have to have separate resource claim.

A

Is that what it is Patrick.

C

Now, for each for each Network, you would have one resource claim and.

A

That's, but what I'm saying is, can one resource claim uh contain multiple parameters like yes,.

C

Of course, yes, yes, parameters.

A

What I mean is gauging parameters so.

B

A

Let's say my single: can my single resource claims have things like contain and and look into things like, let's say a bandwidth and um and qos.

C

So the resource claim uses the same concept that you have in your in your in your proposal. There is a reference two parameters and those parameters are specific to that drive of attendance resource claim. So if, if there is one part Network type implemented by someone, they Define which parameters they accept and those parameters, they need to get passed through to the resource claim and then through the resource claim to the driver and those parameters can be an arbitrary cod, with whatever Fields make sense for that pod Network.

A

Okay, but is it basically I think it can be true because it's all up to your controller, so basically what I'm getting at is like today, I, uh let's let me maybe do a better analogy: I can I can request a CPU or a memory right. uh Can a single resource claims uh gauge it all based on those two parameters at once: yeah, and because it's custom, your own controller, so basically uh yeah you're right, because then it can just see who has what resources and then uh list those return.

A

Those nodes that um that kind of fits both conditions right. So what I'm getting at is that it will boil down what sort of controller for Dra we will create what will be the default one right that will come out of the box, because I think with this one we will have to have one right and then it has to be. There has to be ability for you to either um Point your pod Network to a different control controller for the array.

A

They, let's call it I'm, calling it dra control machine related, that's true or not, but let's, let's say that is pod. Network can point to a different control, a dra controller right. That will say: okay now use this guy for this spot Network, because this guy is my own and it includes stuff like us, bandwidth and an additional stuff that will kind of identify which network is available.

A

C

I think for the entry solution. We probably want to focus on the default Network and, as Peter said, this is the one that needs to be usable right away, and it also needs to be very efficient because you don't want to slow down parts that only use the default pod Network.

C

So for that, for that case, we probably don't want a resource claim to be created, because that that implies that the scheduling becomes more complex, because the scheduler needs to ask for controller, and even if that part is very simple, it still implies this extra object that also might confuse users.

C

So I think it would be useful to have the fast path for the default pod Network somehow and then have this extension mechanism, where we support also Port networks that depend on a driver provided by some vendor, and it may and probably is useful to think about how to have an example driver for such a network implemented either in kubernetes or in some repo to validate this entire mechanisms.

C

Even if it doesn't do much beyond what the default network does, it would just be for testing purposes, and then that is when the solution that we pass on to vendors and say: Here's: here's how you can do it. Here's an example now Implement your own custom, Hardware logic or whatever else, parameters what you need uh using using this new new apis. So.

A

So, for that reason, I yeah, I I, would maybe those yeah that's something that we need to think of whether we automatically create this resource claims or a pod. Network can reference one and that will indicate which path I want to use the more complicated one or not all right, I think I'm, gonna cut it here. uh I don't want to kind of get into this for let's read through this doc. Everyone- and let's start commenting on this one and let's kind of adjust what we are discussing.

A

B

One, the one thing from that conversation I think be useful, which I think is what we're going on to next is what the interface looks like to specify in the Pod. What what you want from your pod, Network I. Think, that's, probably what we're about to talk.

C

About now, so, if we can yeah exactly if we get a.

B

Picture on that, that'll inform them yeah.

A

So so, basically, just to- uh and let's let's move to that, okay, um so to kind of resume to what we discussed last time, the first basic things right: the Pod um uh pod spec changes. We are thinking of adding the Pod Network attachment uh and that's that's a list of structures which will be the reference to a pod network, name uh optional, uh name of the interface that you want to add to your uh to your um attachment and whether the specific network is a primary or not so.

A

Basically, primary means uh whether um this guy is the default Network or not inside the Pod. So anyone had a idea of Uh something's missing for them. I know that maltus has much more like you can specify Mac address IPS static IPS. um There are some bandwidth, Flags I think there is much more but keep.

D

A

Is something I'd.

D

Like to hop uh the some holiday additional informations areas, such as The annotation, was some some stuff. It should be the implementation specific parameters in the pop Network attachment side.

D

So so maybe they use the won't to add. So, let's, let's imagine yeah the as the you mentions. The amounts uh have a way to adding the static IPO Mac address of this stuff. So maybe this should be not in the Airport network attachment object, but I suppose this should be put in the uh additional information inside the port Network attachment right there. We have the extremely string map all right, so they are using the string map. We could add India some parameters, so so I'm I'm just proposing that here to adding the additional stuff.

A

A a string of strings map is I, think I'm, not sure anyone would allow us. This is usually easy reserved just for for labels, I'm, not sure whether someone would allow us to just arbitrary parameters like that. That's kind of um yeah it is convenient because it handles everyone's case, but on the other hand, that's I'm, not sure we that will be okay and I'm, not sure whether we will be able to push that through in the API.

A

Just maybe before we go there what's kind of what information, what I was thinking and I'm going to go to that? Let me just maybe close that one, the.

C

A

I want to kind of just point on this one, just from from the basic point of view, I hear you tomorrow, let me just maybe just from the basic so right now what we are saying: we're gonna have the interface name Network and what is the primary just.

D

For the sake by the way, I also have the question about the primary. So this this design is expecting. They are one pod. Network should care about the IPv6 as well as ipp4, but sometimes some some user wants to ipv4 uses the eso. Zero I mean okay ipv for the default network is used, but to that for IPv6 case they want to use the data plan at that time.

D

How you express how you explain the uh this stuff at that time. The both data playing on the defaulting network has the.

A

Primer, so you separate what you're saying Tomo you can separate the default gateway for V4 and V6. That's new and I. Never is that true. Is that what you're saying.

D

Yeah so so the ipd4 and IPv6 have.

E

D

Different uh Gateway I mean that I mean that happens.

F

D

A

A

Is that what we're looking for.

A

Sounds like foreign.

A

This course there is no need that to have other struct right, so basically I can otherwise I can do just this right say that so that covers all the cases you. The thing here is in the list.

A

uh This part, this flag can be only in one element. It cannot be in two elements at once right so that that validation will have to be there, but it can be only in one element of this list right. This is a list, so only in one element.

F

The default gateway is, plural, are gonna come from Road advertisements in V6. You can do this, all you want, but it's not gonna on a you know, maybe on.

A

Michael now you're going into that's up to your implementation on how you do that I agree with you. So.

F

If I have seen 80s whatever the hell, it is no.

A

It's up to implementation because if they use the ATP or slack then I agree with you. But if not, then it's up to the implementation and and again, if you just if your network uses like I agree with.

F

You you don't have nothing to do with slack. Slack is how you get an IP address. If routes come in IPv6 from Route advertisements, if you are an industry standard, compliant IPv6 implementation period,.

A

Sure Michael Let's uh I'm pushing back on that because this.

D

Is not an issue, this is.

D

Yeah today, I'm understanding what the mic is saying so maybe the, uh but also that I'm also understanding that you are thinking the uh you are thinking the different type B6 right. So so maybe the India cap, you should Implement. You should explain the uh this in this cap. Ipv6 is the uh a little bit different from the huge ipp6 such as we do not care about the router advertisement. We do not get the ring work ourselves start to specify.

A

A

It's it's not that normal, it's up to the implementation and uh Michael. Let's sort that out in this should be a separate cap. If that's an issue in kubernetes I, don't want multi-networking fixing that, because that's gonna be this is a completely separate cat. Where she was saying is an issue today in default Network today and let's resolve it there, not here, please because I think that's that's a separate cat. That has to be there to say the default gateway comes from the route advertisement, but kubernetes does it differently?

A

Let's resolve it as a separate cap in the default network. Not here, please, because that's I I see what you're saying I know what you mean, but I don't think multi-network should support a suit, cannot resolve that because that's something that needs to be resolved on the core component, then we can inherit that. So maybe what we can do is yes should.

F

We build on the house of cards, then.

A

I agree with you, but what can I tell you I I, don't think we can resolve that because, as I said, that's completely different story resolving how the V6 should be done. It's not a multi-networking issue, it's V6 issue and how it's supported, I agree with you we're building. Then then, let's say we don't support V6. Is that what we should say then I don't think so that that won't fly as well. Neither so.

B

That's not I think this is wrong.

A

Wrong Forum, on on on on that V6 and how that should be held, I, don't want to handle V6 and how it's handled in kubernetes, because we want to just create multiple interfaces. I agree with you: we are building how uh what you're saying uh how House of Cards can be right, but.

D

The one questions matter not not regulated the IDP or B6, so why we need a primary field.

A

D

A

F

A

Know which, which network is supposed to be the primary and the default gateway without.

F

Conflict has it.

A

Yeah, that's your implementation. Michael I, let's put aside the implementation. I know that can be one of the implementation that can have it, uh but I, don't use. Let's say cni config. How do I tell which network to use as a default gateway.

F

Doesn't use cni today.

A

It does, but not all implementations rely fully on cni Michael, please. This is I'm trying to push back on on cni. That's the only go-to. This is not it's not.

B

Aren't we saying that at the moment the default gateway is a parameter, that's specified in the networks, as opposed to in the pods, and therefore that could continue is that is that no.

A

That's not HP, because I can have multiple, so it will not cover the case where I want to have um uh multiple networks per namespace. Each then separate pod network will be default, will be primary for each of those namespaces. So I will have to configure every pod Network as a primary, because I want all of them to be Primary in their respective namespace. So what you're saying I cannot have one network saying that your primary and the other one? No, not because then I can attach those two into my pod and then what.

B

A

That's that's the case. We need to cover as well yeah.

B

I'm a bit concerned that we've got we're worrying about this particular field, but there's presumably other fields that need to be passed on a parapod basis. What's the what.

A

B

In mind um so I don't know, uh but are we saying that the only things you would ever be able to pass to a network is the network interface name and whether it's a primary I'm.

A

No uh but I think this is the most generic across uh the board. What we are trying to support and I'm gonna, throw at you srov for dpdk, you don't have the name, you don't even have the primary at that point, the primary might be because your implementation might might use the interface, but then it's up to your application, the PK application onto how you're gonna make sure that this this interface is the default.

A

So, and we want to support such case right that this network is all let's say, dpdk type Network, which you just pass around uh pointers to a PCI device right, and so this is an extreme case. An interface name already has no meaning here, uh but that's- and this is just an example- that's why I'm I'm saying if you will want to say Okay I, want to specify Mac address here, someone's going to say I'm using completely I'm, not using ethernet I'm using something else. So, let's add now uh infinity.

E

I, don't know that these are like are mutually exclusive right between like dpdk not having any of these and then there's the other side of the equation, and I can tell you how often this happens because of how often I do see people do this, which is where okay, so you have your uh you have your definition right, your pot, your pod Network object and then you're going to attach to it, but you happen to have some like non-sequitur type of PODS that don't fit exactly that pod Network and you want to tweak something about them.

E

I think it is a fairly common use case, and it's just for the reality of how people's networks are. Is there there are exceptions and just because uh some things like dpdk says? Oh all, this stuff is like kind of optional right, like you nailed it like interface name, is meaningless right, Mac addresses potentially um meaningless or whatever, because it's all handled on a different IP stack right.

E

However, if somebody does need to set something at run time, you're creating a like, potentially just a like paperwork problem for them- that you've got to have like copies and copies and copies of these pod networks in order to reference all these different variants of them.

E

When you could potentially have something to say, like hey, I need a static Mac address for this thing, because it exists on a Network that has Port security or whatever and I need to like have one that matches what's already in that external equipment or something so I do think that we need something to express uh a like uh that. You may need some particular detail here and yes, I realize cnai is a implementation detail, but this is something that, in the cni world we handle with cni args right.

E

So these are the like other things that you can pass around to handle that that's my two cents.

A

D

A

D

The so I mean okay, so yeah yeah, so they are not only the uh cni, so the implementation specific. You mentions that those are the Implement. All almost implementation would introduce the additional feed for PowerPoint basis, so I think so that not not the specific data structure, but we should provides some string map or some stuff, and that idea and then the implementation can consume uh this field whatever they want. Maybe there's some implementation, May, adding the Json industry, it's okay, some implementation May consume just a swing. It's okay.

D

A

Not sure, that's that's, okay, to provide a just simple string Json into the Pod spec. That's nobody's gonna like that Tomo I'm,.

D

Afraid so yeah, so this is the implementation specific I mean so the from the apps. From the kubernetes point of the we should. We should provide the empty string field for our implementations, I'm just talking and then the each implementation can consume the string field as they wanted.

D

We should not enforce into implementations to to use the. Do you stop having that so user can using the just kubernetes, just provide a string field string back.

A

Nathan yourself.

G

uh I, just I was just wanted.

A

G

Just wanted to second that I think there is a use case, in my opinion, for some implementation, specific metadata. uh If um passing, if we're only needed to passing the string, maybe an alternative view Json would be, would be a crd reference of some sort, yeah.

A

I was so so what? If, um okay before we go there, because I think we are moving to another topic, I think we didn't finish. The primary discussion and I want to finalize that before uh so before we go there because I want to talk about the network interface that I am proposing, not that proposing that I was thinking initially to have, and maybe this answers all our questions and all the kind of concerns used. But before we go there, let can we finalize on the primary.

D

The different questions about the Airport network attachments. So so maybe if you want to conclude the primaries.

A

Yeah, let's, let's conclude on the primary, so is that something that we so I understand today we have only one network, so there is no other way to kind of this split, V4 and V6. But that's a fair point right. We can have that and putting aside Michael the case of that, it's completely Incorrect and the default gateway is coming from Route advertisements.

A

um Should we, then we not even worry about the primary six? Should we just because that has no meaning, because it's coming from.

F

Right, Allah, you probably 4, has no meaning what's Primary in the V4 case, what's primary mean what there's no default? What's the next top Gateway? Is it even Ethernet or emulating ethernet I mean unless you put something to up just saying its primary is not going to help me.

A

Why not? Where.

F

Do you, if it's if, if you're, using a bridge interface for example, what do you want this into I need a next stop.

A

Okay, so your uh your network, attachment definition, will have uh the uh definition for that wanted.

F

Why don't I just put all of it in the network attachment definition.

A

Then that's that's your choice as well. Yes, so.

F

Is this primary optional.

A

It has to be defined for one right now. Let me see.

F

I'll put it all in my cni conf list.

A

It is optional, yes, but then you control, but keep in mind that you lose control of what's the default gateway, uh you lose control for it in this form and then you have to make sure you attach the correct pod name to identify what is the default gateway? That's what you you sign up for then.

F

Yeah, do it all the time with pod man darker.

A

That's: okay! That's fine yeah! So basically your name defines whichever okay I'm using this name. This network, this network is the primary. So it's going to be the default gateway. That's what you're signing up for that's what I want to avoid. I want to the network to be more flexible, and it says okay here on this pod you're gonna be default, but on the other pod you're, not because you're you're, just a secondary.

A

That's what I want to achieve to be Network like in my implementation,.

F

A

To have achieved that kind of flexibility.

F

I just have a problem with. There is no such thing, an ipv4 or V6 of a primary Network. You have a route lookup for every prefix and you will egress out of eat X, Y or Z, depending on that well lookup, some of which ecmp.

A

That's correct, but then you can Define what what 0 0.2 and then the next hope is up to your implementation and imagine you have an implementation where the next hub, I'm gonna, bring up Google and our gcp. Our whole infrastructure under the hood doesn't rely on any any of what you're saying, because we're using end to end and end end point to endpoint. So basically, my next help. Okay, just just make sure that the packet goes out of the note. The rest.

F

And that's implementation. Specific I'll use the same.

A

Exactly and yeah that's what.

F

I'm getting I don't need.

A

So so that's what I'm saying and implementation specific where, where at some point that the Gateway in your case you need the Gateway. So in your case, if you need the.

F

Gateway, why not not if it's point to point I, don't need a Gateway if I'm using slash, 31s or slash 127s I, don't need any Gateway.

A

That's what I'm getting at that's what I'm saying: that's why this flag can be useful, but then, if I need to, if I have slash 24, let's say then my I would imagine my parameters. Ref will contain such Gateway.

A

That's what did I would achieve whether.

F

I put it your one icmp away from that Gateway changing underneath you.

A

I'm confused why icmp.

F

A

This is before, are we talking about this.

F

And then, if you, if you're running with routing protocols, it even gets more, you know no.

A

I, don't oh okay, so why would I run that? Why would I run a new routing protocols, then well I. Don't want to go that far. I want to keep it simple and, let's say: I have simple: L2 L2 configuration yeah.

F

Being simple is how we got into this predicament in the first place, with only one interface in a pod.

A

Then you want to push every every single clustering to be more complicated. That's where no.

F

I want to call it I want to put it in the conflicts like people. Were you doing today with things like Docker import man among others, and then you pick the one that suits you.

A

Sure but then you're tight, the single one is tied to be the the only the single one or I need to create future version for the same network, because I want to for, depending on what I'm doing, I want one network to be the primary, the other one, not on the other case. That would don't support my cases or I have to create double versions of every Network one that has the is, is the default? Has the default set and the other one, which is? Where is not that's.

E

What you're telling.

A

E

I I: that's not what that's not what I heard, but uh that much.

A

E

uh Which is yeah I'd like to reduce the amount that people have to change that base object for a detail about? It is essentially what I'm saying you know it's back going back to the conversation. What's your point.

F

There's also the Linux aspect to what you can consent, whatever you want here. If Linux already has a quote, unquote default route in the container, it's going to ignore, whatever you do anyway, forget how it already has, when I'm not going to bore you with the details. Yet.

A

No, but then, if you I, if your implementation handles and manages the pod namespace, how anything else can mess it up like unless you use DHCP and you rely on uh somehow some of those and you pass everything. And and basically you rely on external things, for your pod management and.

C

That's I agree.

A

With you then I agree with you. You mess up with things, but if I'm not mistaken, like uh Kaliko doesn't do that, celium doesn't do that. uh Those are the two I know the bigger ones that do not do that. Don't rely on external things to configure your pods namespace because of what you're saying there is no control over it then, but all the all those.

F

Ironic, because the telcos that use the ACP just for an example want complete control or you don't get on their Network period.

A

And and that that that's it that's a choice, right I, understand that, and there you can. There can be various ways on how to implement that Michael, but I'm talking about like what is 90 of kubernetes networking today, that's what what they do and and I I understand your case and I understand your networking uh kind of approach. But this is like probably five or ten percent of kubernetes usage and it's not doesn't cover the 90 that we kubernetes uses today.

D

At that time that they do not be using the multiple Network period.

A

D

A

D

A

That's okay, yeah I said that now and now I have to swallow that, but you you're kind of right, but then uh what what I'm getting at I am trying to uh make this fit into today's standard of kubernetes I, don't want to completely break the standard that kubernetes has today, because that's what you're asking for and I don't want to go there. Let's do that, then, in kubernetes, in the standard network, uh because I.

F

Want you to know hold on hold on today, kubernetes fines, the very first cni conflict, escabelically or whatever term is in NCD in slash Etsy cni net D. So we're not breaking anybody.

A

That's fine, but that combination doesn't use us, doesn't assume that whatever you do in in the pod, what I want to say? No, it's and then it's up to that cni on how you manage your poding site, and now you want to push on the the spec on the API, more of the kind of concepts of what uh that you have to kind of address, all the dynamic see as well and do all the over there in the cnis and basically boil it down to um to the cni conf how it's being done.

A

I know, that's one of the approaches and that's that's! That's fair and I agree with you. That's that's one of the approaches. Why cannot be both ways? Why cannot be flexible, I'm confused on that.

A

Michael you're muted, if you're saying.

F

Something yeah I'm just.

D

A

Because now I agree with you and and that's fair that uh that's.

F

Why you specify this at all, you do what we do today, so everything works. If you want to specify pod network name is the name of the conflict, and we, you know you can do the same thing specifying it.

F

Why can't we have it both ways? I just gave you both ways.

A

No I I've, no I I missed that, then. What do you mean.

F

If somebody wants to do what they do today, they don't have this object at all. They'll get the first cough list in like.

A

F

You want to specify it explicitly. The name of that conflict is POD network name in the object. It's okay.

B

F

And if you want to specify a second one that isn't the same name that would have been found by default, today's default, you specify a different one, just like I do with Docker and podman.

A

Okay, that's fine, and now how do I choose, which one is the primary? It's.

F

In the conflict and.

A

And and now okay, that's in.

F

Your life like it is today no.

A

I I understand you that's in your case, but in my case it's it's not in the conflict. So what do it? How do I handle that.

F

Today, if you this object, doesn't exist, you find the first Comp List in Etsy, slash Etsy, cni.

D

F

D is it or is it not in in that cni plugins configuration in mine.

A

In mine, it's not.

G

I I would second that and also in addition, we are so in my case. I have used cases where uh you have put a one thing, one being the primary and not B one thing, the other one should be the library so either way.

A

G

A

I I hear your case and, yes, it can be supported. Let's assume the primary doesn't have to be specified and then uh the which which network is uh I, agree on that use case and that's fair point which I might have missed.

A

The primary doesn't have to be specified and the definition of what is the default is inside the podnet Pod Network. That's fine! Let's have that. Let's have that case, and that covers your use case, but I want my use case to be covered as well.

A

My cni account doesn't specify that the network is default, I want to explicitly say it through the Pod Network and then through my parameters, because my parameters when I refer to parameters I'm referring to uh where was it to this parameters right those parameters specify how and how my default gateway is going to be handled, whether it's for V6, if possible, whether it's for V4 and that's up to my implementation in my case and I, don't use conflicts?

A

Okay. So how do you? How do I support that I? Want you to support me as well, then? Okay? So that's why I would want to have this primary field and I agree with you. That's one feedback for what you're saying this field. I think I I I mentioned.

F

That primarily these things that, in this document, don't exist so how's primary set today, if it's not in the cni conflict, if it's not part of like the IPM or whatever.

A

Michael I'm trying to.

F

Please I'm just trying to understand so I make I mean.

A

And and but basically because if you don't buy uh because my implementation doesn't leverage cni at all, cni is just a stop to make sure that cubelet calls it and then follows up and calls it to a agent.

A

F

What puts the default gateway in the Pod? If it's not CNA today, I I, don't do it so because I use cni so I'm looking to educate me I, don't question Enlightenment! Okay,.

A

So that's what I was referring last I think that was last meeting is I. Think maltus is working towards the same thing where the cni is just a stop is just a a a proxy because you have to have of how the API is constructed. You create a binary in that path that you said, and it does stop what it does.

A

It will call into a socket, or there are some other ways socket local on the Node, and that binary knows it's hard coded in that binary and basically it calls that socket and that socket is part of an agent that runs as a pod in that node, as a demon set across all the nodes. Basically, so that's why it's available everywhere and that demon set has a full knowledge about kubernetes API and basically this is like another controller and a fully fledged pod.

A

A kubernetes pod that has the full knowledge about everything has a privileged access and what it needs only is the signal from that stab with all the with basically just in the Pod name and the namespace, and maybe the the ID of the pods namespace that cubelet created so.

F

So this sounds like it's a cni implementation, specific solution.

A

Yes, that's that's.

F

What I'm saying yes, you can do it right.

F

Cni specific solution saying this one gets the default Network.

A

uh It's not mine, you, you call it it's mine, because I would say that the industry just does that all the way uh I think multo's moving there I think Calico and celium. Does that so I think it's not like my specific, you call it my specific, but keep in mind. That's that's the.

E

Standard! That's what okay, let's give a little bit of context for everyone else, for what.

A

E

Explaining mache, because now I'm trying to figure.

A

E

What's the disconnect here so in the in the today's Maltese and cni World and Other cni, there's a number of cni plugins that work this day, we have an architecture that we call a thick plug-in, which is essentially where your typical cni plugin would be a thin plug-in, which runs as a one shot. It's a binary on the host. It runs in one shot and Away you go um now. A thick plug-in replaces that one shot with a shim that passes the information to a resident Damon.

E

That goes and does the rest of the work for you. So you can uh have like a little bit of a richer life cycle. There you've got something always resident in memory. It can have a little bit more state right.

E

um That's not um to me at least exclusive from what Mike is saying, because that the way that operates today is by the same conflict. That Mike is is talking about right. So at least the way that I understand it is Mike is saying: here's what we have today, how what's our equivalent of that for the future Mike did I totally butcher that or help me.

F

No, no, no, and that might account for why I'm confused about the need.

A

Yeah and and Michael at that case, this uh TX cni that that Doc is mentioning, has a full knowledge about kubernetes. So imagine that that agent, it just need the name of the Pod and it can get all the data it needs from the API server again outside of whatever cubelet provided it to him. What it needs only is the ID of the United States, because that's a hash that cubelet and basically pass to cni- and this is something that is not anywhere publicly available. You can grab from API.

A

So that's the information. You definitely need from the cni right. I need to get the the namespace or the hash of the namespace, and then the agent uh looks at that namespace. It has privileged access to enter that namespace and do whatever it wants with it right, and it can do it based on the additionally on top of that, based on the kubernetes API, so I grabbed my pod. Oh, the Pod has those pod networks. I will grab the Pod networks. What's in the Pod networks? Oh it has parameters.

A

I will grab the parameters it it gets. I, don't have to pass all that information through the cni config I, just pull it from the API myself. I just need the references and then based on that my implementation knows what to do and it can do whatever it wants to set the default gateway set. All the routing uh put the names on the interfaces sky's the limit. Basically at that point right. So that's what I'm trying to kind of support as well right.

A

Your case is valid as well and I agree with you, where I'm limited just the thin cni, where the whole configuration and the whole logic is contained to the binary and has no access to the kubernetes API and everything what it needs is inside the conflict. That's valid use case, I, agree with you. The primary in this case is completely unusable because it's over there, but then there are other cases where the thick cni, where uh I don't care. What's in the conflict complex, it's just a stab. It's just a shame layer.

A

Everything is the whole logic is uh in the agent and that agent has even full access to the kubernetes API and grabs everything what it needs. So that's the two cases, a Tomo I catch. You can go ahead.

D

As you mentioned, I've been India. How do I say the uh this cap is forced to network implemented us to choose the thick problem, not the same problem yeah. Definitely so this means the current funnel is not working.

A

D

A

D

A

The current flannel I I hear what you're saying uh so. What we would do right, that's something that we can think of.

D

Right, whether the current funnels, like yes, the Primitive the uh difference Sienna, which the cni CNF cni Community, is providing.

D

It's not not fair, I think I mean.

A

That no I agree with you, so maybe at that point uh for those use cases it we use what Michael said. It's just the name of the conflict they are named and basically what Michael said the Pod network has too much to the cni. If you don't specify anything and you're uh you that's what it may can default to. Maybe that's. We should look into that right.

A

How to support what you're saying I agree with you: how we do that uh we point to the cni, and maybe that should be the default and- and maybe this is where the vendor with we would have to support from the get-go the default vendor provider. Basically, it will be the the cni vendor right. Basically, the default cni vendor, which.

D

We should maybe then this is network provider right, so this.

A

D

As you talked this design, the uh the port Network design is not the implementation specific.

D

We are more thinking that they're, not only the cni, also the other. Something can be utilizing so at that time the opportunity should have the uh uh how they say not only the way to access the data and they also reached the uh the cap should provide the way to passing the additional implementation. Specific parameters for each area, I think.

A

So so, okay for the attachment itself, yeah.

D

By the way, the time is now yeah.

A

The time is up: let's continue on this one next week. I think this is I, hope, Michael, that we managed to kind of at least found a common ground. Now.

F

No I'm just I'm trying to understand sure.

A

So I think this is a fair point from you Tomo on how we would support that. Maybe that's something that we should look into and have a ways a means to support the default like the the current uh conflict, because today is just the first on the list. Maybe this can change that, and this is what what Michael was saying. This name identifies the cni conf list and then the only thing that will distinct it will be.

A

We Define a hard code, a specific provider that will just say: okay, just use a simple cnis and the names and those are the names for those. Maybe that can be it and then there's no parameters, no, nothing! You can expand them if you want to, but something to think about all right. Let's, let's cut it here. Let's count, I will add this as a continuation point for the next week and and if you have a chance, please read what Pete wrote about the dra uh still, so we can continue next week as well.

A

Thanks everyone hear from you next week.

C

Yes, thanks thanks.