Kubernetes SIG Network, 26 Apr 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Multi-Network community sync for 20230426

Description

Multi-Network community sync for 20230426

A

All right welcome everyone at the multi-networking community. Sync um today is 26 April after a week, uh Hiatus figures we are back in the game. um Let me share that and.

B

A

So today, I.

C

Want to kind of Deep dive. This is something that last week, kind of uh I assume something uh about the KC Micron controller and I want to kind of Deep dive.

A

Into that, what it is, what.

C

Is the whole icon story behind it? Then I'd had added Michael Michael your comment, something that stuck to me last time we talked you mentioned something about uh what networks a separate, City and cni config I want to kind of touch on that and then after this, let's continue our discussion in the cap.

C

um If there are any other topics, please add them up before the continuation of the cap, because.

B

This will just take the rest of the.

A

Meeting so make sure that you add it before before that.

B

C

um So the first bullet, uh the KCM iPhone controller and maybe just so I'm, not sure how many folks are aware of that or not but like to kind of to level set everyone and everyone being on the same page. What the KC Micron controller is.

B

Basically, uh this is a small piece of a code that.

C

Runs inside the cube, Cube control manager, Cube control manager. If someone is not aware, this is the kind of one of the core components that comes inside the control plane of kubernetes, so we have the API server and then we have the KCM with it inside the KCM. There is a bunch of various controllers, it's not a mistaken! Probably all the controllers that you can think of like that are related to operations of the kubernetes. Are there so one of the pieces is iPhone controller.

C

Iphone controller is basically responsible.

B

For populating the no the dot, spec dot pot, cider field- oh sorry, let me just do it. So there is a field in the node object: core node objects, which holds a pair node cider.

C

This cider is mandatory for a node to be ready. That's basically that's what it is. That's the kind of condition for this field. This field, of course, can be said by this iPhone controller, or we can disable it and the user can set it up by themselves all right. So, however, they bring up the cluster in the uh in their platform.

C

So basically that's the mechanics behind it. Now, whenever I was talking and referencing ipam inside pod Network in the subject that we have discussed, I was always talking about this field being populated, and only about this and the reason behind it is because today the whole ipam per pod is always done by the cni, and we I doubt any one of us would want to as defining a one way to do the iPhone and and decide for everyone else how they have to kind of work around an iPhone.

C

So we cannot Define it on this level because it's done by the cnis themselves. There are so many implementations for it and they can handle it. So I never had in mind and I apologize. If that was the the thought, but I never had in mind that the thought network will take over something like the static ipam or the host local ipam that we have available in the cni package. That's not the case because those this job for the cni.

C

What this field basically is is a I would say a Helper, because it's not mandatory to use it. It's mandatory to set it for the node to be ready, but it's a it's not mandatory to use it and that's what I talked with the with um with some folks about this and with Team on the history of this, and that was initially introduced to kind of help with these ciders that needs to be used.

C

But from my understanding it it's not the best thing and the reason behind it is because it's not enforced, so it can be said, but the the platform doesn't actually have to use it. So that's what the story behind this is, and this is what basically, in our level of approach for the Pod Network object.

C

This is the only thing that we can control, because that's the only thing that is exposed today in terms of like ipam 4 pods, and we don't want to get into the story of taking over how the cnis do their icon, because that's that's too much and I I wouldn't want to go there.

C

Is that clear? Are there any questions to this? What I just said, or are we all on the same page and that now that's kind of understandable.

D

uh So I have the uh question about about the the the API side, but before that the uh the my Cambria May left at the uh maybe the 15 minute later so there may be how about to proceed. The uh the micro cambia, Sport, Network or separated cni conf topic. First.

C

Yeah: okay- okay, let's let's: let's do that so now, Michael I and correct me if I Wrong. uh uh Last time we talked um you mentioned something like like this, like the Pod network, uh should point to a separate cni config or something along the lines. If I misspelled it and then apologies uh is, is that what you had in mind? Well,.

E

Yeah, what works today is: what's in Etsy Network the Etsy cni Network deal whatever the cni environment. Variable points to for configs today, the very first or the only config found is what's used right exactly and that's true and uh with things like podman or darker, even though it doesn't use cni the point I can make.

E

Is you just name which one of those files instead of taking the first one, that's I, think the terms used asciabatically instead of finding the the only one of the first one, the file system returns, just name it, and then we're done in a Backward, Compatible manner. Mana excuse me. That's.

C

uh That's that that's all correct Michael. What I want to point out is this is one of the ways of implementing this. This is not the only one to do this, so when you describe all this, it's uh it's all valid, but it's not the only use case that we want to handle.

C

That's what I want to point out and and what you're saying is it's perfectly valid way of one way of implementing this I fully agree, but I want to point out: it's not the only way, and there is various folks here that wants to do it completely differently. Don't want to kind of deal with with the cni uh in full. What you you, what you would think where I would have separate cni or some networks in my CRI that then I just point to through the Pod Network.

C

That's valid approach, but that's not the only approach. That's what I'm trying to say here, but.

E

I think it's a scope creep. It's in addition to what we were originally talking about. I.

C

Don't think that that's the thing because uh scope creep- it's not because that's left how you implement this, it's always left to the implementer and that's what we always talk through all the dogs on how we implement this and how we do we deal with this. What you're saying is you always have to have cni per pod Network, and you have to do it this way and I say no, because I don't want to do it through the cni I will have one controller, a demon set per note.

C

That will look at this object and do stuff for me, because that's how I'm doing the cni is just a app for me. It just estabs for the cubelet to call and send a signal to my controller like most like uh I saw something that multis is doing today is a something they call Tick cni, uh that's one of that. That's what I have in mind right. You don't have to deal with the cni config at all. It's just a signal to your controller.

C

To then take over and and do all the imagination it has to do without any awareness of the files on the file. The the basic uh concept of what you just said live about the uh The Ordering of the files and the path that still is the same, nobody's changing that. But what I'm saying is you have the various options and I'm not saying that the cni in the future- and we have uh Michael here, Michael Zappa- that that's cni eventually can't fully support this and can be a valid approach to do what you're saying?

C

Let's change that, but that's I'm, considering uh changes to the cni as a orthogonal thing and an adult then in the future. When we have this, then cni can adjust to be what you just said or or something else, and of course it will have to be Backward Compatible, because that's what we need. That's what we're trying to build across towards, but I I say we don't have to Define how the implementation has to be done and unless there might be some places. We have to talk like our back.

C

uh Where we have to how we enforce things, then we have to discuss on how kind of kind of that's that's going to work. That's I agree on that part that this might be a bit tricky around that one, but I'm keep kind of underlying the the the the fact that see the way. You're thinking it's one of the ways to implement this, and- and this is left by the way- to the implementers not to the uh not to ask to decide that, because there is so many there is like I'm.

E

Not sure what, if I have to add a whole bunch of crap to the API.

C

That's where, where we are, that's why we are here uh to identify uh what is the uh most common element that we can add to the API, so that everyone is happy right. That's why we are here. Everyone yeah.

E

That's a trick, including my my data center people, who, rather just tell you, get on this network, so.

C

Okay, so maybe I agree with you uh 80 or 90 of the use cases. What about that? Because that's a I I agree with you, everyone it might.

E

Be I have a way my suggestions, 100 of the use cases.

C

Yeah um I I yeah, that's something that that's why in my dog, I think you even commented on that or someone was cat commented most of the use cases right. I did use that word over there. Someone in my in my current doc as well so um yeah uh Michael you've had a hand.

F

Yeah, like you might have missed the discussion around parameters, ref that might have might be insightful for you. I can send you the link to that recording uh just so you can give it a look through yeah.

C

Did Michael did I uh Michael cambia did I kind of explain some and and kind of clarify some of the things yeah.

E

No I was for personal reasons: I was away for a bit and I'm sure I'm still catching up on stuff. Thanks. That's fine! No.

C

Problem: okay, um okay, uh Jamie Network versus attachment. Could you elaborate?

C

What is the question here.

C

But we cannot hear you if you're speaking.

C

Yeah so before we go there, there was a question so what score kubernetes do for uh for each case, yeah I will I will dive into that. Give me a second I'm gonna start sharing the cap and we're gonna go back to that, and I will explain that uh to that answer to that question um uh Jamie. Did you get your mic result.

A

D

Hi me, can you hear me? Yes, oh sorry, I I'm, just just that the uh so behind me about.

D

All right so so yeah yeah that the Jaime is called the high measle. Oh.

C

Jaime I apologize uh yeah, my mistake: I apologize hi, uh so just maybe to what um still you fix and you have I I ask more questions to kind of what is the difference between Network versus attachment?

C

Is this about um so maybe I will I will, because that uh are we talking about pod Network object versus the multuses Net nads Network attachment definition, or, if that's the word it does. The ask uh basically nods are multiples way to specify uh multi-networking and and pod Network can reference Network attachments right, uh but uh From perspective of.

C

Because really wants to call the kind of operation of connecting to a network and attachment as well. There is that aspect I'm, not sure whether you have done some something in my around that one or or the objects so.

G

Him he says he's having audio problems, yeah yeah, so maybe we should just have the same question next week.

C

G

That one one piece of context is there's a bunch of discussion about terminology of network and attachment when when maltes was devised and I suspect it's related to that. But let's let him speak for himself next week.

C

G

C

Will add your I will uh I will add you to the next of the next next topic to the next meeting. All right, let me just switch to this guy ah sure to stop okay, so now getting back to our, let me just zoom this in a bit, can you see it? Okay,.

H

C

Oh now, yeah we can hear him.

H

Oh, this is a good time to ask the question or yeah.

C

Go ahead, yeah yeah, definitely yeah.

H

So uh what I wanted to ask here is, it might have been discussed before, but I went through the requirements cap on the port, Network draft and the thing that has been happening.

H

This is referring to motives, as you have mentioned before, so the thing that has been happening with uh multis and the networking Plumbing group specification is that sometimes so you have attributes that apply to the network, and then you have attributes that apply on how you attach to that Network and and having a single entity to try to describe both things leads to a situation where you have multiple networks.

H

If we call it uh here by that name or multiple Network attachment definitions, if we call it by the airplane specification, you end up duplicating all the network information just to change an attribute of what just to change something about how you attach to that Network. So because you only have one thing that is describing the full things describing both how the network is and how you attach to it right. uh I, don't know if I'm happy to I'm happy to get a short answer. If this was discussed and I missed, it.

C

No, it wasn't so Jaime we we. What we are going to talk about is right now we're talking default network, but eventually I'm, just attaching pod Network to pod I think this is your topic and basically, what can we specify there right?

C

So uh I have some proposals here, look at those, but we have to create something that is um I think this is what you're getting what sort of parameters can I specify in my uh pod versus what can I specify in my network and let's discuss that when we get there, we haven't yet so I think I know what you say. Yes and.

F

Basically, yeah.

C

And basically, what you're, saying and I I am aware of the kind of Rich approach for uh for maltus what it can do today through the annotations, uh but keep in mind. This is maltus and that won't apply everywhere, but still it won't mean that you won't be able to use the multusse annotations to do some of the additional configuration that can still be there if we want to take it into into the core approach. Okay,.

H

C

Let's discuss this uh after when we get there. Okay,.

H

Okay: okay, uh there's a second side to my question, though. uh Okay I've seen I've seen that there's going to be or I saw in the cap that there's some kind of uh Downs downward appy being proposed to check on the ports environment, some details about the mirror or whatever, but I haven't seen anything referencing. If any of this information is going to or how it's going to transpire into the cni invocation itself and and I, ask this because with motors as well, I am always referring to multus, because that's what we have.

A

H

Time uh we have had difficulties when the CI is involved. We have had this difficulties identifying to which attachment of the multiples attachment a port can have. Is that invocation being referred to? uh One way of correlating is the interface name. So if you specify the interface name in the pod, then of course you get. You get the same interface name in the CNN invocation and you know which attachment you're you're talking about, but the interface name is here being defined optional, as as optional the same as in multiples with this optional.

H

So, basically, some difficulties that we have been having in the past is actually in the CI CI invocation being able to identify which of the attachment is that invocation being referred to in.

C

An easy way, I mean right, so so uh Jaime, it's the same uh same answer. We didn't talk about this yet so when we get there and I, as you said, as you read through the requirements, this is one of the requirements to have that and I think it's in Phase One, if I'm not mistaken, so we're definitely gonna cover this. In this time frame for this camp.

H

Okay; okay, all right! That's! Okay! Sorry! For the time.

C

No, no! No! That's! That's fine thanks thanks for asking the question uh before we dig this into the I forgot about one more time: one more thing: uh next week uh in two weeks, I am going for a two weeks vacation, so I needed a volunteer uh to lead this meeting, uh so I will throw that in.

C

uh Please reach out to me uh if you want to do that, if there will not be anyone that would like to drive this meeting, uh unless you want to judges to me, then you you can definitely can, but um I will need someone to drive this meeting in for for two weeks that I'm going to take vacation for so reach out to me, if you want to do that in my place and continue the discussion, otherwise we can. We can always cancel the meetings and then we meet after I'm back so um yep.

C

Just throwing that in uh please.

C

Let me know so that we can kind of uh warn everyone accordingly, um all right, let's going back to the ipam story, so there was a question: how, then those flags uh um control um the the DDD the whole thing, so there is a bit more functionality that we are I'm trying to kind of push here to what we have available in KCM with this iPhone controller in the KCM iPhone controller, what it says is uh I can enable this and then I have pod cider that will be defined in the node and that's it.

C

What I want to be more restrictive here is to say, if I have any of those set to kubernetes and external I then expect that IP to be present all right. So.

D

That's the one so so does this means the uh this equipment. This cap is enhancing the uh the original the.

B

D

For each Network excited currently the node spec proto-sider is having the only one field, one string, but today the you based on your comment. There was once the iPhone 5, the uh node spec for The Cider could be at least or some other stuff, and then it for each Network which specifying the iPhone type equal kubernetes. Then their KCM will provide the approxide for each Network. That's.

C

That's that's the ultimate goal. Yes to have and I want to talk about this in this cap. I don't have it spelled out, it is, but it is slightly spelled out in the initial draft.

C

If you look at that one, uh because I'm copying things from there to here uh some of the ideas and that I'm discussing and we we adjusting, but basically what you just said and and I want to use uh KC the ipam controller to and expand the node object to now not only specify the Pod cider for the default network, but as well for all the other networks, including and then whether you use it.

C

This will still be the same approach if you're, not whether your network is going to use that side there on from the node is optional, because I don't want to kind of enfor kind of change that part from from someone who's doing.

D

It all right, so how about adding these I mean that so so, currently that this field I mean that yeah I pump for iPhone 6 and then also the the commander disk external num variable is for this stuff, I suppose so I mean that I mean that yeah, so so yeah So So. Currently, as you mentioned, they are this so that this field I pump six and the iPhone 4 is used to kubernetes the KCM to uh kcmi from controller to uh uh provides the uh the some certain cider from the cluster cider right.

D

Yes, so this is only the purpose. Is that.

C

Correct yeah: that's the only purpose of this. Yes,.

D

Because and and then this this purpose is not in this cap- is that correct I mean that the in this cap you do not enhancing the node spec portal cider.

C

I will I I do want to do that.

D

In this cap, yes,.

C

Okay I do want to do that, because uh pot cider is, is the the core piece that we can: control from the core side uh and I want to modify that right to adjust for this right to have KCM and modify KC and ipam to understand multi-networking and having clusters either to have a field or something you need to specify. Oh I'm, going I'm creating now this object for this network, not for the default right.

D

So so, currently the this cap dance and dimensions, the.

C

It doesn't yeah but step by step, I I'm doing it phases, but but that's my that's my goal. Yes, uh that would be my ultimate goal and what I want to kind of point out is use specifying uh the internal ipam we'll just enable the KCM to populate those fields in the nodes. It doesn't mean you have your your cni, your your staff that you do for multi-networking has to use it it can.

C

uh It can because you're going to have a Fields with with paranoid eye pumps if you want to use it, but you don't have to that's it's still optional, so that part will be there. But what it does mean, though, is that if you specify the kubernetes in one of the V4 or V6 means you have to provide that IP on that pod, otherwise, what I would like to say is otherwise uh the the cubelet will dim your pod unready. If you don't provide those IPS when you return, that's what that's the additional functionality.

C

That I would like to kind of add to this, that the download API would expect from you and IPS for families that you specified here. This is the new thing that this is how I would like to expand this this whole the kind of the ipam story behind it, because then I can say: okay, my network has to have V4 and only that right, because my my my uh six was was set as known, so V4 is set to uh to kubernetes or external. That means. Cubelet will expect.

C

Give me the V4 for this for this interface in your pod, because this network has a has, has that value set and then, if you don't specify, don't provide it to me back I will deem your pod not ready.

C

That's that's the new functionality that I would like to expand on this. It.

A

Will and basically.

D

So that this is the just the comments, so the So, currently there this how they say this, the your the your mentions, the feature is not described in this cap, so the I'm I'm a little bit confusing that why we need it. So maybe that the the how they say the this field should be the visited uh when the uh you you describing the uh the uh the this the functionality I mean that they're changing the.

D

Node spec portal cider changing the ideas API to support the multi-network, and then they also there. You need to uh also changing the cluster side of field as well. So.

C

Let's talk about it now, then. What do you mean.

D

I I mean that they're not talking I mean that so currently, the you do not describing in the description.

I

D

The changes of the anode uh spec portal, cider or some sort of stuff.

C

So that this we will get there I will I will add that section, but.

D

Right now, I don't want I didn't want to I understood I, understood that you are going to add, but also currently the uh so I mean that today the I'm I'm looking at this stuff I mean before this meeting. But at that time I don't I. I cannot find that.

C

I see what you say: yeah yeah, so maybe yeah.

D

Because maybe the the other and attendees is also the same understanding, I think the first. We need to uh the look into the com, the uh entire features, and then we should discuss about that.

C

Give me give me a second here uh me talking about whether this is how it is going to be implemented. Doesn't change the fact. Do we really need these fields? That's what I'm! That's what I'm coming from I'm, introducing those fields and defining how difficult gonna function and me describing how the kubernetes way is going to work.

C

This is just one of the three options that it has, so we don't have to know the details to decide just for this one to decide whether we need this field at all and whether do we need the V4 and V6.

C

That's why this is where I'm coming from, and that's why I decided to Let's push it further on, because it's not important yes right now, maybe some of the backgrounds why what it will do is needed, but the details, because I I kind of assumed and I apologize for that I assume that everyone's aware of the KCM and the iPhone controller in it, but I think right now we are on the same page on that and how those details work is not important to decide whether we need those two fields.

C

I really agree on that.

I

Yes, I think I mean you're saying so it's an abstraction and we don't have to fight look at the exact implementation of that abstraction right now, exactly.

C

And let's right now, I would what's the high level of this parameters. Are we agreeing on that part and I will add it to the text that um if I specify a other thing, then, uh if I specify external or kubernetes on one of the fields, cubelet requires you to provide those IPS on a downward, API or, however, we're going to identify how we're going to return back the IPS, but they have to be provided otherwise, I will throw you an error or not show you another I will otherwise.

C

I will just not makes you ready, because this interface for this pod requires, let's say two networks and you provided just one sorry I will I will deem you not ready. Is that concept fine with everyone I'm, not sure whether that's possible to achieve in terms of implementation? uh That's I'm, putting that as a secondary, but if we have this kind of high level um goal. Is that something that that's that's agreeable with everyone.

J

Was it two interfaces mean uh I4 and iPhone 6 separate or two interfaces? Two pod network, two four Network support.

C

No, that means a single um whatever. In your case, it is right. We talked about because pod Network usually means a separate, Linux interface right, so let's V8 a V8 into a pair right. So basically you have a one end in the Pod. This is what I would. This is the the most common use case.

C

L2 use case where you have a each pod network, is a separate interface inside the Pod, and for that one you expect either one for 1v4 or one V6 or if you specify both I, expect both for that single instance of the interface for the Pod and basically because keep in mind uh how it is represented inside the pod.

C

It's up to your implementation. It can be just a ciders and what we talked I think. Last time we mentioned that someone was saying: can I? Can this be a completely L3 I?

C

Don't care about additional interfaces like Linux interfaces inside the Pod I just want to have access to those Siders inside the Pod, and if you want to represent the Pod Network as such, we just need to expect that, for whatever you connect because- and this is where we're gonna get to probably next step is after the default Network pod Network, we we will talk about the attachment. So basically, let's use that word per each attachment. You expect that specific eyepiece, that's what it is.

C

So that's, maybe that's the perfect word for this: the attachment for the Pod right so pair each attachment that you define inside the pod we require either. uh This will specify what is required right by this commands that did I, make it clear.

J

Yeah yeah uh the the one thing which you're missing I think you covered that as part of the attachment is how what interface this broad network will be attached to I. Think that's what you're talking about, because I don't see any of the required field that this pod Network will be attached to what interface.

C

So that's something that's that.

J

Is if, if there are two Nicks coming on the VM, I want to attach it to this this it of that yeah yeah.

C

So, basically, that is up to your implementation on how you then link it. How you link your attachments to the your host namespace in your in your node. It is up to your implementation.

C

This is something that we do not specify and will not specify, because in maltose and then I will just keep and and drive on multiples here all the time, if you, for example, specify a network attachment definition that uses Mac VLAN in magdalan's definition, you have to specify the I think it's called master flag master of parent I'm, not sure, but there is a conflict called Master which identifies which now interface on the note I have to use and that's how it's done in that implementation.

C

Your implementation might pick it up based on completely different things or some expectations, I, don't even care about which host interface, because they're using v-switch and it's all virtual and it's all inside that V switch. So because of that purpose, this spec will never specify that part. It's up to the implementation.

C

Okay, okay, thank you.

G

So the thing that I'm finding a little tricky about this I I get the idea and I'm quite happy with it. I think the bit that I to me, ipam has connotations of this- is about how the addresses are actually managed about the interface you're using and how that is done as opposed to how those addresses are being reported back and so I can't think of a better terminology. So maybe this is a helpful comment, but I find the terminology a little confusing. So.

C

I agree with you on that one, and maybe there should be the name should be slightly different, not ipam, but um IP type IP, something or a pod side there for person or pod cider, maybe uh I.

G

Agree maybe, and maybe just having a couple of sentences in the in the comments yeah.

J

And I clarifies.

G

This actually would just make it much easier because yeah yeah.

J

I agree: I sound confused, confuses.

C

Yeah I'm open for com uh for throw me some ideas. Then it's almost like.

G

I'm not alone, it's almost like a returns address field or something.

C

Like that, but that's a.

G

Terrible name, so that's the right concept, but it's in it that name's even worse and.

C

I agree, the name. The name uh is unfortunate and I agree. It should be. It should not be called ipam. I agree on that. One um please, please think about the name and- and let me know uh comment on the doc. If you have something better.

J

I mean I, don't disagree completely. It is kind of an ipam because internally you are calling that specific IPM to set up the network for this, which is either external or kubernetes so um yeah it. But it is.

C

um Yeah so, but still uh just having this discussion that we had till now uh kind of makes it kind of everyone was thinking that ipam, okay, we are going to do purple daipan, which is uh which is not not ideal, so it should not be that, and- and here I had a a time to explain all of you on how what that means. When someone's gonna read through this cap, that might not be the case and then they're gonna ask the same questions over there.

C

So, let's maybe think about some better name for this to I will think about internalism I I ask my team. Maybe someone there will have some idea as well, um but I think on the requirements for this any other questions and, and what this mean.

C

If not, then we come back to my idea. This is where we from one from two weeks ago, where we, uh where we talked so since I pump, the internal one is based off of a KCM. The control for the iPhone control inside KCM is done through those two Fields. There are Arguments for this KCM so allocate nodesiders, and this is an argument to the cube control manager and there is another field called cluster cider all right and keep in mind. This is in the context of default Network, so this is for the default Network.

C

So what happens if I update my cluster from 127 to 130 or something where this guy is in and we require a default Network to be present, how we now create that default? Network, because that's what it boils down to, we have to have it right. If we are making uh this thing as a core component, we need to make sure that we how we are migrating from one to the other and the way to do this is through those fields.

C

Since we are controlling KCM on there and as you can see here, this is all description on how how we do this, because we need to set when I I want to automate the uh the or maybe I want to give an ability to to automate the creation of the default Network pod Network, which is which is required, because we, if we want to depend everything on the default for all the other pods. We have to have it right.

C

So what I want to do is uh give an automated way to do this, but of course, give a way to manually, create it as well- and this can be done then, through if I upgrade my cluster. If someone will want to finally support this during upgrade of my cluster I can set this flag to say that no I will create the default Network for the cluster. That now requires this object to be there right. So then I can use this flag to kind of set it up myself.

C

So the first automated way what I'm thinking is since the only fields we have to set in this object is basically those two Fields, the iPhone, 4 and V6. All the other fields are optional and they don't have to be specified uh and we don't have any knowledge about the other ones. So, basically, that's not something that we would uh really care at this point, but we do care about those things: the iPhone, 4 and 6 fields.

C

What we can we can deduct those values based on what are the arguments provided to KCM? Basically, first thing is this allocate defines whether we even run the ipam controller, so basically that can give us a notification on uh okay, all the. Where was this?

C

uh What I'm saying uh all the stuff is not defined, so basically, what I'm going to do is if they allocate notes either is false. We don't really know how this whole thing is done, because maybe someone is using um the Pod ciders, but the whole thing is done uh by some external things, and here I have my in mind: Cloud gke as well as well. We don't use an outsider in gke.

C

We There is a flag called cloud cloud provider cloud provider or something like that, which then sees the whole even modification of the an addition of the Pod cider to some external different controller, not the KCM one. So basically, this lock is a disabled, and in this, in this case, I would I would like to just not specify this field and the reason for that is because I don't really know which eye pumps are you going to use. Is it before?

C

Is it V6 or are you dual stack, because it's you don't specify that at all, so I cannot kind of infer what you really have to set and since I want to enforce what IPS you have to return by this. This is very important of which I pumps, four or six I do set. So that's why I leave those fields empty, and this is what I was referring and to remind you all.

C

Even if the object is immutable, I would want to use the same mechanism that the port cider has on the Node and to remind you, the port side there on the Node. Initially, when the notes are created, the Portside field is empty, it's not set, and while it has a special functionality, is that you can. You cannot mutate a value to a different value, but you can mutate from empty to value. That's what it has.

C

The Pod cider has that kind of capability that from empty you can set any value, but after it's set, you cannot change. It. I want to do the same here with this with the default object. Where, if you don't set it it's fine, but then you can set one of the values that you wish to do right. So you can sell it once, but then you cannot change it to kind of to keep in in mind of what what can be done.

C

So, basically, that's how I would kind of deal with the migration in case where nobody's using the um the the iPhone controller.

C

In other cases, as I said, uh the uh arguments here are the all the options, basically that's possible. So basically, if I had allocated residers set to True means the iPhone controller is taking uh is taking part in the whole thing, then user has to specify cluster sliders and basically this cluster cider has ability to specify either a single IP, a single family or a dual stack with the comma separated list, and you can look at I think I.

C

uh This is a link to the to this um to this argument, so you can read more on this, but basically you have to take my word for it. This is a way to specify both types to this argument, and when we do this, then I will just say: okay, you want to do dual stack, so both are set to kubernetes, because you have true, you have specified both I will. I will set this to like this.

C

Otherwise, if you specify only single family I will change the other value. To, None, meaning I, don't expect any value and I don't want to change it in the future right. So basically, I will set it to only the one family being set and the other one being set to none any questions to this. Does this make sense?

C

This is for the automatic creation.

I

Just a clarification, uh if you set it to Nom during the automation, can you later set it to no? No, no.

C

That's the thing right, because that means that it's already set so in uh in case where it's not set. Then then, because this is a value, that's the point of it. I want to explicitly say a value to State, I I, don't want to use V6 right, I'm, not sure you can change. Can you change existing cluster to to dual stack?

C

Probably you could or you cannot. So that's that's a tricky piece right, I'm, not sure whether that should be allowed right, changing uh from non or not setting this. Maybe that should be the case. Then right we could, uh when it's singles family. We do not set it so that in the future, if you want to change to dual stack, then you can just add that yourself.

I

And that's why my concern is that that you will never be able to enable it to.

C

F

C

And then the last thing is manual creation.

C

Which will basically disable the whole uh this all algorithm.

A

C

How to create the default Network and it would there will be a separate flag into KCM, because I'm, seeing this this default, Network being created by the KCM itself and based on those arguments. But then you can specify this disable default Network creation because, as the cluster comes up being upgraded, what's not I am going to myself introduce that that object, and this is important because I would like to make cubelet's Readiness and then nodes Readiness based on existence of this default pod Network right.

C

So if you this will be explicit flag, you would have to specify, but that will mean that you apply this object to the cluster on your own through your automation, your your your installer of the kubernetes cluster right. So I want to give that kind of capability to uh I. Don't want to I. Don't want you to do any of your automation. I will create default because, for example, I want to have a I fully implement this whole thing. I want to Define as well parameters.

C

Ref I want to so my default network has to point to a deep to a parameter draft which then just uh my cni uniformly uses for additional networks and for the default as well. So then, I don't have to you know, do something else for the default Network versus the normal Network I do have it you I can create I can implement it so uniformly, so that every network is treated in in par with everything all right. So that's kind of the idea behind the as well manual creation.

C

Any questions to this I think this. This, hopefully makes sense. That's why.

I

A

I

One more question that I had is roll back. Let's say something went wrong and you go back. That would be problematic. I guess right with this.

C

I think if you roll back, if you roll back your um see your uh your KCM, the controller that blocks this object, or maybe even an awareness of this object, will go back right will get deleted. I'm, not sure. How does that work exactly with KCM in crds? You have the crd definition and then you can delete that and all the object goes away. But if you roll back, uh you will roll back the as well the guards on deletion of this object, because the KCM will have all the information about.

C

Can I delete this object, or maybe it will be an API I'm, not sure exactly where, but basically the let's call it a web hook that will guard the this. uh This network default, Network object to be deleted will be gone, so I can easily delete it so and basically I'm not saying that we are disabling those fields. We are not, they are still there, so basically you're all front or back it should work.

C

I, don't see this as being an issue right, it's only up to your implementation to properly support. You know both ways and it has to right when you roll in front or back right, uh I'm, not sure who was first.

J

Thanks Mastery um I find um making this as a immutable Fields one set I find it um hard to, especially in case of scenarios. Let's say my cluster is created with one cni now I have implemented uh a version of like a better version of different cni which will handle the default Network and I'm doing the upgrade from that from the from the from the previous one to the external one.

J

um Now, because this property is set on a default on a cluster, my upgrade story is broken. I can't move my app uh cnis yeah.

C

I will I was about to go there and I will uh because I talked with the Sig Network uh and we have to support this guy. That's uh one of the requirements is that we do have to support a ways what you just described Neha. uh How do we upgrade change the cni completely? We have to support that case and uh yeah. Let's talk about.

J

It or if we do it um like asean, is evolving in Microsoft as well we're moving from the older ones, which has limitations to the new one, which is more advanced and scalable.

C

Right so yeah and and I I I heard that that's a feedback uh I got from the Sig networking as well, where we have to support such use case. What you're describing uh that I can change from one cni to the other for the default Network, because for the additional Networks uh as soon as the network is not used by any of the photos, I should be able to just delete it right, that's something that definitely and then I I create a new one, uh but for the default, since it cannot be removed at all.

C

We have to do something about it. uh Toggle.

D

Yep, uh so they are the uh in the the section of the manual creation. At the end of this uh paragraph, the you mentions that the uh if the uh default network is not created at that time, the cube will send in their cni not ready, but at that time the uh maybe the uh we should specif the clearly mention the default network is not created. I mean that so this, so this issue is happened because the default network is missing. So this is the CNN not ready.

C

Well, maybe that network.

D

Not ready I think.

C

Yeah, that's that's a implementation detail. I agree with you. That's probably this is just uh I know there is a network Readiness. We probably have to tweak into the current comma uh yeah I I you're right yeah.

D

So so then, maybe the error message should uh explicitly mentioned what is happened so they especially the uh if.

F

Someone understanding.

D

This this default, the network object, is related to.

C

Cm yeah definitely this.

D

Is not so yeah this.

C

Should be, this will be not only for manual, it will be reported even for the automated until the automated kicks in right. So that's keep that in mind that it will be just common right. Cubelet will expect the pots the default pod Network to be there. If it's not, then it will just report.

C

I would probably have to put this by the way into a separate section, because uh do I have this somewhere here or maybe even copied over here, uh so yeah I will I will I will make sure that it's I think it should be here. Any other comments to this.

C

All right so uh to what Neha kind of uh mentioned is exactly this. This use case right. uh What I was thinking is to introduce something called an admin mode um to the cluster that will allow a deletion of the default, Port, Network and then um Recreation of it right. So, basically, because that's how we for the other networks, that's what we would do, but now uh what we would do for default, Network and I- don't have a idea here.

C

um If anyone can think of something I I welcome any proposals.

J

Completely in the admin mode, so like the way we do upgrades internally is you: you have certain node tools which are running on the older Network and then your live migrating or the cluster to the new cni new new icon. So you add a note pool and then you um move that node pool to the new cnim and you slowly migrate, the old ones to the new ones. That's how uh that's one way of doing one is like uh destructive migration. Where you stop the cluster migrate and then one is light migration, so I want.

F

C

J

Wanted to support both.

C

So the blue, green uh I would assume that's that's straightforward, there's nothing to it right, blue, green. The distraction that you're saying uh I would call it a blue green uh type of deployment, and uh basically uh this is straightforward. You just recreate the cluster right, that's easy! The problem is with the migration that you, uh the kind of the.

J

The life migration, the.

C

Live migration: it's a problem, because if my implementations fully supports a if my migration fully supports uh the Pod Network, as is Right fully I read through it, I read the parameters of it and, what's that and then suddenly, I want to change from one to the other.

C

How do I now create go from one version to the other version? How do we do that right?.

J

C

Because now, because now um so.

J

That's the implementation detail of the other of the new ipam to also have some back compact policies with the old one so that you not.

C

Exactly let's, let's assume, let's assume, I, have a completely uh ice I'm. Switching from uh let's, let's call light switching from Calico to CU. Let's do that. I'm doing I I'm going completely breakthrough approach, I'm, switching from Calico to celium. How do I do that, because why I'm asking that is because I will have? Let's assume I from both I have a parameters reference for both and they are completely different.

C

uh Kaliko has their own colleague or something parameters and psyllium has their own psyllium parameters, but then the default which all my both implementation uh read throughput Network and the default one currently points to the calculator, the old one right. How do I now make sure the new node pool when I bring it up, uh uses the celium parameters which I want to Define? So that's my problem right because both will have to have a default. Network defined and I cannot change the name for it or.

J

Exactly the problem, uh I'm I'm, trying to uh say that we need to solve and we have internally in Microsoft different flavors of CN islets, that we have cubenet. Now we have, we are trying to migrate those cubenet clusters to Overlay or see Azure cniv1 clusters to Overlay exactly even though they are internal CF between kubernetes. So we have the control of both. uh But there is a migration story there, like in the.

A

J

Is aware of the cubenet nodes so that it does not assign the overlapping exactly.

C

So so, and and and basically I even proposed the exactly I, even kind of visualize, a different problem right. How do I deal with different parameters right between the two and.

J

How do I know are we saying then the default network is per node pool, not cluster White, but then then I can specify for this node pool use. This kubernetes is the running one and when I'm migrating, my new default network will be for the internet. For this node pool. I will have this spot Network right.

C

The thing here is kubernetes doesn't have a concept of node pool. So yes, gke has it most of our Platforms in Google. Have noteful concept, uh you have it, but kubernetes by default has a concept of just a note, not a notebook. If I'm not mistakening,.

J

Yes correct, but.

C

So so so so so, if if we were to do something pair, it will have to be per node, not the notebook.

J

Because notebook doesn't exist, that's where I was going eventually, it will.

C

Right so I think we are at the time I I I, uh let's next week, discuss about how to deal with this admin admin mode. I I I there's some of the ideas. Maybe we can ensures, but still we have to resolve on how we deal with two different configurations. Do we have like in admin mode I can create a default V2 name or something like that, like a special name which basically will be used by the other implementation temporarily and then I delete the default and then I rename the default V2 to default.

C

Maybe that because we need a bucket as well to kind of hold both at the same time right so default for the old stuff and and default V, something that will be created and then, if we have that, then we can migrate between the two. Something like that. uh Maybe that's an idea think about this: let's call it the quiz for today and I'm. Looking forward to someone, at least, if you someone reach out, you can could drive the discussion for the uh mid-may meetings. Please reach out to me.

C

If you want to derive that, um then then I can give you some notes on what we can. What are the next steps to talk about, and, of course, you can have your own ideas on what how to drive this? That's uh it's not I'm. Just give you can give you some suggestions. All right folks um have a nice weekend meet you next week, bye.

B