Antrea LIVE !!!, 16 Feb 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Episode 16: The CNI Specification (Part 1)

Description

Lets look at the CNI Spec in detail tomorrow, in preparation for Mike Zappa's next show on how to build a CNI from scratch !!!

In this show we look at the cli 0.3 spec in antrea, and how it can evolve to the 0.4 spec and beyond, and look at the overall format of the CNI json that containerd/runc use when making new Kubernetes pods.

Containerd 1.6 - HostProcess Container support for Windows
(https://github.com/containerd/containerd/releases/tag/v1.6.0)
- [Kubernetes Policy Management Whitepaper](https://github.com/kubernetes/sig-security/blob/main/sig-security-docs/papers/policy/CNCF_Kubernetes_Policy_Management_WhitePaper_v1.pdf)
- [NetworkPolicy Status KEP](https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/2943-networkpolicy-status)
Conditions []metav1.Conditon
- antrea cni 0.3.0 ... 0.4.0 (CHECK) ?
- `kubectl delete pod -n kube-system -l "component=antrea-agent"`
- containerd ... go-cni
- runc
- containerd (go-cni) cni add
- co-cni imports. github.com/containernetworking/cni v1.0.1
- parses the cni json config
- matches 0.4.0 to the versions of its own cni config parser

A

Okay, we're live, hey everybody, it's andrea, live and I don't know mark rosetti came to say hi to us. I don't know how long he's going to stick around but mark mark. What's going on, you want to introduce yourself.

B

uh Hi, I'm mark, I work with jay and I mean over in sig windows work in kubernetes project. um I work at microsoft and yeah I'm here until my son wakes up.

A

um A meme you want to go ahead, then we'll get started.

C

uh My name is me: I.

A

D

A

On vmware, all right, let's see um so interestingly, I just tried to create a cluster and it failed. So today we're going to try something we're not very well prepared. Today I will warn everybody. We only have um we. We are going to see if we can read through the cni spec and find get clusters find delete. Cluster dash dash name equals andrea.

A

Okay, I'm going to delete this one and I'm going to try to create a new one, see what happens uh the last one totally failed, and I have no idea why. um So what do we got in the news?

A

Let me actually one second jinji's here: what's up chinchi, okay, um if folks are, let's see here.

E

Let me get one second, let me get the show link. Someone just asked me for it: youtube.

E

Here it is all right there we go: okay, cool.

E

E

Okay, here we go.

A

All right, so, let's see if this comes up and.

A

Using default kind network, so we're going to try to spin up a andrea cluster using the same way we normally do um and depending on, if that comes up, docker images failed to load. What's this.

A

Gawker images, so is that not docker pull? What is that is that failed to load? Have you ever seen that before never seen this one before docker pull I'm gonna? I bet that was corrupt. What, if I do docker rmi, can I remove that image and then re-pull it? I don't know just an idea.

A

I'm gonna delete that cluster and try to do it again. I have no idea what just happened.

A

ah Yeah, what's up zack zacchaeus joined.

A

What's going on, I oh yeah one. I can't do one window at a time zach, but I can make it bigger.

A

um Let me make this one, you know what else I can do to make it easier to read. I can do this, I can do how do I make it, so this is only.

A

Can I make this so that it's only like one okay? So while we try to spin our kind cluster up again, what do we got in the news today, containerity 1.6 mark you want to tell us why you're so excited about this one.

B

uh Yeah continuity, 1.6 is the first release that supports properly sports host process, containers on windows, which are the equivalent of privileged containers, which are uh enable us to do a whole lot of things and then kubernetes and windows that we weren't able to do before.

A

B

Such as such, as run your cni's as a demon set uh with, if using continuity, as your uh container run time,.

D

A

A

So that's exciting! So now it's actually official uh privileged containers on windows is a thing now, even though we're not technically supposed to call them those. What's this I mean you want to tell me about this one. You.

C

Just pulled it.

A

C

So someone from from security uh create a document to get the best practices on uh policies inside kubernetes. So there is a long list of supported policies and how they could be implemented. As a good read.

C

This includes like network policies and.

A

Cool um all right: we got this one and then we've got what's this. Is there an update on this.

C

Yeah this was margin. The cap network policy starts from icon.

A

Oh yeah, we never mentioned this. I don't think on the show so yeah the.

A

Network policy status kept is in so now we're going to be able to start adding um conditions for network policies. So I guess what is it the cni providers are going to update those like my cni provider, makes a my cni provider says, sets the status of a for per policy okay, so it has a status field in it and then now any network policy you have your cni provider can optionally. I guess, update it and say I don't support this. Is that the idea.

C

Yeah, that's the idea, so you have particular conditions where you can define if the network policy is applied or not and add these thoughts of events for this.

E

Okay, cool nice. Okay, let me update this.

A

A

Thank you a meme! So let's see okay, so we got a cluster up. So what should we do now? So, let's go see. So if I go into this one, if I go in here and I say: docker ps, docker, ps and then I.

A

Want to make it bigger for you zach here we go if I do docker exact dash d dash, I c1 sh right. um So I'm in one of my nodes now and let's see if we can go to cd etsy, you go to bash, so my terminal isn't horrible. So if I ls etsy ls etsy, where is the cni configuration? Is it in etsy or is it? Is it an op c cni yeah? So, okay?

A

So let's start by looking at this in like an andrea context, and then we can like back out and actually look at the spec right. So all right! um So inside of this etsy cni net d10 conf list, I've got a version.

A

0.3.0 I've got a name, andrea and I've got a I've got these plugins. So I've got the andrea plugin and it defines the type of ipm that andrea wants to use. Then I've got this port map and then I've got this bandwidth plugin.

A

So now there's also an opt. So so so each one of these and then we'll we'll look back at this spec in a second right. So so here's the cni spec. So for folks that don't know this, the way this works is the cni is a specification.

A

So it's a it's a it's a specification for defining the way that your container runtime puts a network on the containers when you start them right, and so your container run time is, like you know, run c or if you're doing linux or like. If it's windows, it's there's only one windows container run time right, mark hcs, shim. Well, it's shim hcs, a shim wrapper around that, but yeah the host compute system system, persistence.

A

Okay, so hcs is the windows cri linux, many cris most commonly container d run c right, so I'm going to just enter that in the comments. So folks who are not listening can see this. So that's normally the way it works and then the way that what you do is your c and your cri calls your cni to make add a network to a container okay.

A

So that's it! So now let's go and look so so how does a cri know what to tell the cni right? So the cri needs to ask the container to make one of these. So the cri needs to. I think, the way this works is when you set up container d right. So when you set up container b, do we give it andrea as an option? So if I go to cat etsy, I guess it's in etsy: is there anything or is it an opt-cat, opt cni.

B

Yeah, you can specify the configure and a plugin dirt in the container deconfig.

B

I think it's usually have to see an ipad.

A

What's in here, nothing yeah, so I think it's in the comments. So if I go to cat etsy container con, oh what what is it? Why is this cat etsy container d, conf pat, and I should have a cni in here right? No, unless you're using the default one yeah, so the default. We can go fish around and see so, let's see, what's normally in my container d, config right so and normally in the container d config. We have.

A

But we have, we give it unpacked layers, we give it a snapshotter, we give it a default, runtime name, that's run c. We give it a runtime type.

A

We give it and that's continuity. V2 we give it a sandbox. We give it a pause image if pallavi is here, she's, probably laughing and looking at this, because she's been fighting with pause images in continuity for like three weeks now and then um missing huge pay. I don't know what that is, and then something here, so we're not actually specifying this. So that must mean that there's a default there must be a default in the container d tommel config tunnel right.

A

So I think I actually looked this up the other day for another reason and I'm gonna try to see if I can find it.

D

um I think it was here.

D

Where was it no? Maybe here.

A

Yeah here it is, I have it hold on one second. Here we go so it's here so in the in here right. um This is the container deconfig and I guess you can specify in here the cni configuration and you can specify the bin directory in the conf directory um of where, where the cni's are called from, um and I wonder whether whether it's I don't see it in here, which is weird, so I guess my first question is: how does how are we giving this to.

A

I don't see a default path here right. I don't see a default default default default.

C

Maybe maybe it's in another folder that choose the.

A

Cni, where, where would it be, let's see.

B

There is a defaults folder, but I don't see these values defaulted. There.

A

It's interesting so what if I clone it? If I clone this down cd source, git clone container d? Well, I already have it a good thing. I already had it okay, so if I get grep.

A

Here we go it's hardcoded in here or no that's an md file, that's a dot md file. So this is an md5! Oh here we go here. It is mark there. You go okay, so, okay, let me make this big so actually in the container d source code here right so in here I guess the default actually plugs in op c and I bin and at ccni net d.

A

Okay, and so that's, I think, that's like those are like defaults in the code, so it looks in for a bin directory in a comp directory and then and then from there when container d starts every time you start a new pod, it's gonna go and it's gonna um based on the cni spec that we're about to look at in a little bit um sort of parse through this. So like, let's, let's go, we can go in here and look right. So uh where is.

E

A

Whoops, so let's assume now, this is our path right that it's looking in less here right, it's going to go so so, when container d starts up right since we haven't set this value right, it's going to go and it's going to look in this default. Hard-Coded thing it's hardcoded in the go for the tomml, and then it's going to see this right.

A

So it's going to read this and then now from here we'll ask ourselves what happens when we when it reads this file, so it reads this file and then it sees that it wants to do this andrea thing and this port map thing and the bandwidth thing now do either. One of you all know what port map or bandwidth do now we're going to go over. Look at the cni spec and try to understand what what what this json means.

A

It doesn't sound like it. A meme is at a loss for words.

C

So I know bandwidth is doing some uh traffic shaping in your pod port map map some uh port ports directly, but I'm not sure how this is being used.

A

Okay, so let's see so these are I mean these are this? Is this idea of plug-in chaining and I suppose, reading this? I think this is so we could see here.

A

This is a list, and so the first thing that's going to happen is andrea, is going to do something and then after andrea does something I think it's going to pump some json out and then I think, then that is going to get plumbed into here and then that gets plumbed into here and then at that point whatever was when you did your cni add operation, which is the thing that container d calls when it's cr after it creates your pod.

A

um After that container d run c, whatever you want to call it run c is going to assume that the network is created and what not. um So so? What is this specification so so here's another example configuration that they give us. So this is cni.dev docs spec version right, so you can read the specification here, and so this is cni version 1.0, so it turns out we're only using 0.3.0 right now.

A

I don't know if that's good or bad or whatever, but it works. The cni is a pretty simple api right. It's just add and delete and check. I guess I don't know what else it does. I mean, maybe maybe a lot of other things that wind up going on under the hood, but it's a pretty for the most part, it's a to to make a working cni that you don't have to implement all that much and and the complexity comes in.

A

I think once you start looking at the way these plugins are configured so in this one they have the same thing. They have a plugins directory just like we have in this in this andrea cluster and then um antonin's here, and who else is here?

A

Oh perva, hey man is that been why it's been a while, since we've seen you good to see you and fang han, where are the plugins defined, says fang han um the.

C

Binaries are on opt cnib, but they.

A

C

In this configuration file.

A

Yeah, so that's a yeah, so it's a good question. So if we go to opt c and I bin phones and I bin right, if we go in here, then we see there's all these programs- and these are. These are binaries right.

A

So if I ls, if I ls dash altrih opt cni bin, okay, each one of these is a binary right and um these are callable and and so like, so like, for example, when it says type, andrea, right, type, andrea says like there's a binary program in this directory called andrea, and when you start when you, when, after you make a product, I want you to plug the network in. I want you to literally do a call so container d or will actually or is it container d that does this or is it the run?

A

Time does container d, send the cni information to the run time and then the run time starts the process and then calls the network stuff or is it? Continuity tells the run time to start the process and then after the process has started, then it goes. I.

B

A

Container d: does it.

B

But I'm not positive.

A

Okay, I'm kind of wondering here so you know I have a diagram I made once that doesn't have it though um anybody know. You know, I see antonin's here he might know so grayson. What's going on grayson since you're here and I like that picture of the what's his name jason state- is that jason state jason stink from that the guy that beats everybody up in the movies fang fang han okay, you got it all right. So what I don't know I don't know, I'm wondering who calls the I'm gonna check.

A

Does cri call cni plugin or is it container d? I think it's container d.

A

um How container runtime, okay this this guy wrote a blog post on this, so config from the netpoint company? Okay, does he talk about the cri in here?

A

uh Listen for requests from the coop in the agent, the scheduler located each node through the cra coolant is instructing the container runtime to spin up a container and the runtime is executed by calling it a standard way.

A

So the runtime is executed in a calling on the run c binary and flannel and flannel the binary that implements the c id above process. Here we go, he has a diagram, so the kubelet cryo container d, oci image, oci runtime runtime, so cni so from the way he's described it here. These are orthogonal to each other, and I think that makes sense that they can uh that the process itself maybe is created, and then the network is set up afterwards.

A

um So if anybody wants to answer that, let us know, um but I think we can keep going, feel free to interrupt me so okay, so this is type bre. So so these are the plugins.

A

So this again is a name there's a name that, in this configuration they're expecting opt cni bin to have a a program in it called bridge, right and and then so so so this so this so the container you would call bridge right and, um and then after that I would call tuning and then, after that it would call port map. So this is the configuration that you get, and um so I guess they've broken the cni spec that if you go to the cni specification they have specification conventions, cni tool, so cni tool.

A

I mean you know a little bit about this right.

C

Yeah, I think you could explore this okay. What do you got? Yeah just go in the examples. I think this.

A

C

A

Yeah, okay, so so you're saying okay, so you have.

C

For example, we have some binaries here we can use. Like uh the example, that's create a network configuration this line and run the binary.

A

Can I can I just, can I not, can I do this without do I need go to do this.

C

You have the banners, I think this is only compiled.

A

When I tool can I just like, can I literally pull the binary down.

C

A

Maybe they do they do releases here. Let's see if I can just literally w get it. Oh they don't do it, they don't compile it for me, um okay, so yeah! I I don't have go on this inside of this. Oh wait. You mean I don't need to do this in a container. Is what you're saying.

C

I think you can call the binaries and pass, but.

A

C

Be in the name space in the network, namespace.

A

Okay- I don't have go on here, so let's, let's avoid doing this right now to keep the pace up. So um what does this do?.

A

It looks to me like.

A

This the way you normally use this is you.

A

You would literally take like this whole json thing here and then you'd echo, and then you write out a cni configuration. You call this a configuration, cni configuration and then you go and you make a new network namespace, and then you set up you essentially yeah. So then you set up a binary path for your cni, which is similar to what we have here right so like it's obviously an ibin, and then you run cni tool and then you have cni tool, makes the cni add command.

A

And what is this thing? This is var run. This is references the.

C

A

Here and then after I do this, then I can run check and I can check whether this thing worked or not. So this is just a way to call the individual cni. So let's go back now and look at the spec, so, okay, so the cni spec. Where do they define the verbs right? There should be an ad section here. It is section two okay, oh add so the cni specification has add.

A

It has delete and it has check- and I think that those are the only verbs right. Oh, it has versions, yeah, okay, so the cni okay. So you have these three things, and so does that mean that every cni plug-in has to implement all three of those? I guess it does so every cni plug-in, um I guess every cni plug-in has to implement these right and so so the entry of binary. Can I find that on here, there's probably an ops, so I can call that right. Can I just what?

A

If I do, what, if I don't send it anything? What does it do? Opt bin opt.

A

So if I do this, what does it do? Okay, just so the default. If I actually just call this the default, is it returns version right.

C

Yeah we can take a look in the code of this is the in the repository command entry cni.

A

C

A

So if I go to the code base and I go into cmd and I go to entry of cni antonin was the last person. What did he do here? What is this all about?

A

This is a pretty straightforward exchange. We cannot use the default protobuf package. Okay, so, and who's been up in here doing stuff, main.go, okay. So what do we do here? So action, ad request, action, check, request action, delete request and then version. So I guess the skeleton is skel, that's so, okay, so skeleton yeah, that's a thing that you can vendor in from the cni people, so the cni people have a thing that you vendor in and it allows you to easily make a cni plug-in and I guess the arguments that it takes are are.

A

These are the it literally takes these in order, so we can probably take a look at container networking cni.

A

Right, okay, so let's go here.

A

Comes here it is yeah right, okay, so if I go in here- and I go add here- we go so that so when I, when I run this plug-in main right, this is something that we implement here.

A

Plug-In main and it takes in functions as inputs, and it takes an add, a check, a delete, and so what we do in entry is we vendor in the container networking sort of factory that allows us to make a new container networking plugin, and we literally just call this command and we send it in order the add, check and delete functions.

A

This is a good idea to look at this. uh I mean yeah.

C

So on pkg cni, you can see what's going on.

A

Okay, all right so pkg cni all right. So if I go into client.go all right, so if I go, oh god, I click on something. Here we go so now. If I go in here, okay, we have these constants.

A

And so I guess so, I'm kind of wondering like. Why is it that? Why is it that we're on o.3.0 in here how come our entry at 10 conflict is 0.3.0? What am I like? What entry of version am I, on pantria.

E

C

E

C

Cni specification yeah, that's.

B

The version of the.

A

Cni spec to use okay, so that, but that's set up for me by andrea when I installed it right, so I'm installing android 1.4.0 right. It's writing this file out and it's writing out 0.3.0.

A

C

Version right: this is the cni version, the spec version.

A

Okay, but why why are we using that version? How come we're not using a newer version, maybe because to limit incompatibility cases we can strive to support multiple releases along with that major version itself. This is harder than yeah, so I'm kind of curious, yeah antonin since you're here, if why is it that andrea is writing out cni version 0.3.0?

A

Is it because we am I asking a dumb question here? I mean I mean it says 0-3-0, but andrea's on 1-4-0 and cni 1.0 has been out for a little while or maybe I'm I don't know how long cni 1. it was, but I I assume the cni spec is, I think.

C

It's near, but not that.

A

If I go to cni releases, I mean 1.0 came out what so we're on 1.0.1. So what came out before 1.0.1 cni.

A

Mark didn't know he was going to get into all this drama today.

A

So 0.6 so yeah odo 3.0, is pretty old.

A

Let me see here I mean. Maybe we just don't need anything from the newer spec.

C

I think you are taking a look in the plugin, so.

A

Well, who writes this file, though andrea, writes this file when it comes up right, yeah like andrea, is writing this file, and it's writing this cni version when it comes up. This must be hard-coded somewhere.

C

Okay, yeah that that's correct. The version is one.

A

Okay, so if I go in here, I open vs code. Thank you, microsoft, for this.

B

Probably whatever is writing the cni configs just using 0.3.0.

B

Sorry say that again, like whatever is setting up andrea for you.

A

My entry agent sets up andrea right on the node right, so when you, when you coob, ctl, create your cni right then at that point like, if I do, if I do the same thing, I don't know so I guess let me see if I can ssh into another cluster and just see so like what, if.

B

A

Go into, I have another one somewhere here, part.

B

Of the setup should be writing those files to those directories on the host, though,.

A

I mean when so you're saying, wait so you're saying that you think, for example, kind. Does this.

B

Not kind like I'd imagine that the cluster comes up with no networking and.

A

B

Able to um run this with like host network and then, as part of running the the cmi demon, set it's going to write files. It's either it's going to map those files in and that's just hard coded somewhere in whatever oh a network load is, is doing that.

A

Wait I got it I found it hold on hold on. I think I know I think I know, because if I look at this hold on, I know I know where to look, because I know that there's a build, I know, there's a build directory somewhere.

A

Where is it it's in right? At the top there yeah build? I have an idea.

B

Yeah, probably in the ammo somewhere for.

E

ah Entry uh here it is this.

A

One no basically.

B

Maybe is it just an intro that yama is it.

A

Here it is look at this right, yeah, okay, so this is hardcoded in the yamls right. So so we hard code. This can I file an issue like what, if I did what, if I filed an issue right now, why is this? Why are we on 0.30, maybe somebody's already filed? One issue is andrea and put in another one of my like issues. That's like half baked here we go hold up. 3.0. Has anybody watched about this?

A

I don't know no.

B

Since the cni configs, I have seen they're all using it at 3.02, so.

A

Oh yeah, okay, yeah, yeah, okay! So um well, I don't know um we'll wait to see. I don't know if antonin or june j and either one of you are here, and you can tell us why we use 0.3.0 in so many of these and so like we should. It would be interesting now like what, if we just what if we checked in another, I have another cluster right so like what.

C

If we change this to the new version and create a pod.

A

A

Change it to the new version and create a pod, okay, yeah. We could do that and see what happens? Okay, yeah you're right like yeah like what are they gonna do you're right, so I have to act okay, so I have to how do I do that? I guess I have to apt to get update dash y and then I have to do apps get install them right. So I can edit that file.

C

Right so one one thing: that's.

B

A cleaner cluster- this is probably just in a config map somewhere, you can probably find it and just edit it there.

A

I could edit it, but I don't think if I edited android agent will hot reload the configuration.

B

You could re-right do the demon set yeah re-deploy, it.

A

Yeah yeah, I could do that yeah, that's actually. Why is it asking? What is it? What did you do how like, why does it want all this?

A

You can't install okay, cool yeah, you're right I could. I could um do that um so.

A

1.0.0 is what you want to do. I mean yeah all right, so here we go. Let's try it. Let's try this okay.

B

So antonin's waiting for a street yard like to join, to tell us that we're doing everything incorrectly, okay.

A

Okay, okay, anthony's gonna, come in here all right. Here we go all right hold on a second, I'm gonna. Send you one: um okay, I'm gonna send it to you in upstream slack, all right, antonin.

E

And turn in hey anthony here he is okay.

A

There we go okay, cool, so um so, okay, so what happens now? So if I do this and then I um ps ax, let's see do I have what do I have do? I have anything running on here. No, let me give my favorite thing: coupe ctl scale, deployment.

A

Or dns okay, so let's scale this up so that we have one running on this node and then let's see if it starts, here's the antonym. What.

F

Did we do wrong yeah? I don't think you did anything wrong. uh It's just. I think I used this version because when the cni's were written, this was like basically the uh the current version and that's kind of the version that was used through cubelet and docker shim, probably okay. And if you look at 0.4 for example, what it does is, it adds a check command, but if no one's using the check command, then it doesn't really matter uh which version you advertise and I'm.

A

Not totally sure.

F

How this version is used by the container runtime? What what kind of validation is done? I think that's something that would be worth looking into um because container d, for example, is going to be consuming that file right, and it's going to be probably checking that version, and it's possible that, based on the version it finds it, it behaves differently or maybe it doesn't behave differently, which also would be interesting.

A

Okay, so it's worth okay, so this is kind of something interesting worth looking into. Then.

F

And I think no one has moved to cni 1.0 basically, so that would be because that's probably a breaking change on like like 0.4 compared to 0.3, which just adds a cni check command and if no one's using the command then nothing's going to break anyway.

A

Okay, all right cool okay, so that so okay, so this one has the says check in it, so we don't use check uh very often does so the kubelet and container d, and all that, like we don't wait next week, we're gonna have mike zappa come on and he's actually writing the check implementation in container d right I mean.

C

Yeah he was looking at it.

A

Yeah so so I guess the reason. One reason why we don't one reason why we don't do this.

A

One reason why we don't care is because container doesn't implement it, but luckily, mike zappa is going to solve that problem next week. So if you come back on the entry alive next week, the guy who's actually solving this problem that I don't know if we'd call it a problem but who's going to make it so that um it's actually worthwhile for us to upgrade to for odot 4.0 he's going to be here so cool. um I forgot: what were we doing.

C

We are taking a look in the pods if the pods come off.

A

Okay yeah, so let's see if this comes up so do I have accordion, so I didn't get a core dns pod to come up. So if I do coupe ctl get pods dash a is it complaining?

A

um No, they all came up, but I don't know if any of them came up on this node.

A

So what if I scale deployment? What if I scale it to one and then I scale it back up to eight and then if this doesn't happen, I'm just gonna. Do a diamond set. Okay.

A

Oops, okay! So if I do ps-a-x.

A

It looks like nothing, it looks like it's not landing on here, so something must be failing. So if I do coupe ctl coop ctl, it might not like that.

E

um Coupe ctl get pods dash.

C

Is not ready, do it get nodes, it says it's not ready.

E

Yeah, it's not happy.

A

No they're all happy so the nodes look happy, so it must be. Where would I see this? I would see it in the kublet logs right, um cto.

C

Container d- maybe full computer d or cube, can.

B

I try describing the node too. It puts node conditions on if.

E

If you already.

A

B

The networking setup, though something would have said, network ready.

A

Yeah so yeah, it said.

B

Because it initially worked.

F

A

Yeah, does it have core dns here? No, it doesn't so journalctl-f-u-kublet.

A

Do we see it complaining at all? Okay, so, let's see what if we scale it back down. Okay,.

E

And then we scale it back up.

E

And here we go eight.

A

I feel like it's not even oh here we go an error. What is that pod delete or container.

A

I don't see anything, I don't see it complaining about the cni.

B

So maybe we should well so like.

A

Marketing, you probably need to schedule.

B

A pod to trigger containing digital networking just laying that config file down, I don't think it's going to have any effect unless, if less container ds, unless.

A

Like the cra is going to, you know be adding or even pods, but it would be because I'm I'm going from I'm scaling, yeah core dns down to zero, basically and then adding 10 pods to the cluster. I mean it's kind of a stupid test, but it would is core dns on host network.

B

Is it using host network there's, I know, there's different paths that continue setting up.

A

Core dns always uses the I mean in general. It always uses the um uses, the pod network. But that is a good question, so we can confirm that so um so yeah see these are all on the 10 subnet.

A

um Okay, so a better thing, maybe we can test mark's thing, so coupe ctl get config map dash a. If I can. I do this antonin. If I chan, can I change this.

A

That can I change this andrea cni conf list and then restart my entry agents.

F

uh So if you change it on the disk, then you don't need to restart the agent and doing and restarting the agent should overwrite your changes uh if, as I think mark suggested, you update it in, I think it it's part of the config map right in the enfield.tml and you apply it uh yeah.

G

Yeah, if you change.

F

It here and then you reapply that yaml no no 1.2, okay, yeah and then you reapply that yaml um then it's uh it should be good. I mean.

G

F

Kill the agents right, you kill the agents, but after reapplying the ammo right, you have to um update the actual config maps right now. You just change. The yaml manifests.

A

Well, I edited the config map. Oh.

F

You ate, oh okay, I thought you changed it on disk yeah, so you just need to restart the agents now so now.

A

I could just do coupe ctl coupe ctl delete.

A

Pod that dash and cube system is there.

F

A way to like yeah, you can use, remove the pod name and then pass the label. It's dash. L there's a label basically.

A

Oh, I didn't know you could.

F

A

F

And the label, I think, is component entry agent. So if you do delete cube system dash, 3 agents and it should delete all those agents.

F

Like this equal entry, a dash agent, I don't yeah, I think it shouldn't.

A

Oh you're right it.

F

A

Oh, that's, nice, okay! So there's a good way to hot roll out everything and.

F

While we wait for that, actually I was looking at the leap: cni go code, which is what is used by container runtimes like container.

A

F

To interact with the cni, and if you look at the implementation for check how they invoke check, they actually do a version check and if the version is less than 0.4, they return an error directly to the container runtime. And so as long as container d doesn't use check, then us giving a version advertising a version of 0.3 just works. Fine.

F

If container d starts using check, then we need to advertise a version of 0.4 and in a way it's like surprising to me that we don't do that, and I think that was for backwards compatibility with older version of kubernetes, because we do implement uh the check call so.

A

So you do implement it, it's just that you don't expect people to call it. So we probably shouldn't yeah no.4 here.

F

uh Yeah, I think, if you can, we file an issue. I think, if you do, four things should keep working. uh If you go to one I think they're backward, I mean or breaking changes. So, but it's it's. It is actually interesting to move to one and to see what's going to be failing right, because I think if you move to 0.4, nothing is going to fail. Everything is going to be fine.

F

I think if you move to 1.0- and I think chan from the entry team is a pr open to do that, but he was kind of like experimenting with it. Oh.

A

Okay, so, okay, we just noticed that.

F

I mean it's hard right because you have to make sure all those pieces keep working right. Maybe you want to support older versions of kubernetes uh other versions of container d, newer versions of container d, so all of those pieces have to basically cube let the container run time and um and the cni they all have to be able to kind of.

F

Like speak, that same version- and I include cubelet here because I think, if you're using, if you're, using docker, shims and that's kind of like part of cubelet and it's going to be invoking the cni docker.

B

Gym is gone now, though,.

F

Yeah, that's why I said older inferential versions, right. It depends on how far back you wanna you wanna go, but I think docker shin, being gone, is kind of like makes things easier years. That's one less thing! You have to worry about.

A

So I'm gonna file as just as a placeholder, so we don't forget this conversation um and I think that would be an interesting thing to look at and yeah so yeah. So this is a very serendipitous time to be having this conversation, given that mike zap zappa's coming on the show next week, so pear, it's good to see you man thanks for coming to say hi, um so.

A

Where are we so now I do now: let's see coop ctl get pods dash a oh coop c. So now, let's scale chord dns down to zero, actually scale it to one, because I'm afraid that, like what, if nothing works and the whole cluster breaks after this, I want dns to still be working.

A

Okay. So now I'm going to scale it down to zero and coupe ctl get pods dash and coupe system. Let's confirm that it's scaled down! Okay! Now none of my agents have an odot 3.0. So if I go in here and I cat etsy cni here, it should be 1.0 right, yeah see so I've overwritten, so I'm using the new version. So now what happens any predictions here.

A

You think it's going to work antonin.

F

I don't know you're using container d right. I would say that, depending I don't know, if container these are the version that's able to use like cni 1.0. um I think it's possible that you're going to see a container the error.

F

Oh, it does not look happy.

F

I don't know, do you know where we would look for core uh container errors, or maybe it's just gonna? It's just gonna report them to them. Yeah.

A

Fail to create so yeah, it's yep, it's going to send him it's going to plum them back up through the container dude to the cubelet, so failed to set up sandbox incompatible, cni versions.

A

So who who returns.

C

This message is that container d is that.

F

I think that would that would be container. The plugin supports those versions right because right now in entry, we don't advertise 0.4, for example, so.

A

Or you do you do advertise 004, you don't advertise 0.1, which means we could actually do zero.

F

Four well right now, what you change right is you change the cni version in the cni config file to 1.0.

A

Yeah, but if I do this, it'll work right. So if I do that and then I do your same trick again right where I delete those right so now, I'm deleting all of the entry agents- and I just added the config map to be 0.4.0.

G

Yeah right now it will work, but this error we're.

F

Seeing is container d or your container runtime. Looking at that cni file we've been editing and seeing that the supported cni version is 1.0 and complaining, because oh I'm container d and I don't support uh cni version 1.0 right now, but.

A

It says plugin.

G

A

But is it referring to itself as plug-in.

F

I mean we can check where that error comes from right. Let's.

A

F

But actually I would say I would say that the the standard cni code, which is vendored by container container d right. It.

A

Worked see yeah, it worked, so it was referring to this file, so okay yeah, so it was somehow andrea. Is advertising 0.4.0? Where would it be advertising that.

A

F

Is that the file you are editing, that's the only file. That's the only place right, yeah.

A

But what is this 0203031.

F

Those are the versions supported by container d, but to be more accurate. Those are the versions supported by the cni library, which is vendored by container d. You know this. um The cni library is like the the the standard cni code that the cni working group meant. Oh.

A

So, okay, so there's a library that we vendor into container d.

F

Yeah I'll put the link in there.

A

Oh okay, I got it in the youtube.

F

Chat right and container d is in is rendering um is vendoring a version of that library and each version of that library supports like a set of cni versions. So.

A

There's this go cni and then that is something that's vendored in so we can go here and then it's vendoring a person.

F

Yes, yes, then it's uh importing a a version of container networking, yeah.

A

It's it's importing version of this, so so so so right now, the version of kind that I'm running is using a version of container d that vendors, a version of gaucy, and I that vendors, a version of container networking cni. That is not this and.

F

What's interesting right now, you're looking at top of three and you see that the version here is v1.0, which is probably aligned with the cni version they're supporting. But if you look at the exact version of container d used by your version of kind and you kind of like walk up that dependency chain, the tag you're going to see here is not b1.0 is going to be v0.4 that something probably.

F

But now, if you, if you were to move to top of three container d, somehow you would be able to do that. In that case, the failure would probably be somewhere on the entry aside, because I think there are some breaking changes in in the api in 1.0 in 1.0.

A

F

From the container d perspective, uh things would be fine. You would not see that incompatible, cni version, because entry, I would say I'm using 1.0 container d- would be using a version of elipse cni that supports 1.0, and so everyone would theoretically be happy, but because entry doesn't exactly support 1.0, you would see a a runtime error in the entry agent, probably okay,.

A

Okay, so container d is vendoring, go cni and then on startup. So you start a process when you start you do run c and then, after that container d calls c and I add, and then um it's doing that it uses go cni to do that. I guess probably somehow, and then that version is yeah.

A

Okay, that version.

E

A

Looking at whatever versions it can handle so then in there then it parses parses, the yaml config or the json parses, the cni json config, and then that.

F

Well, there is one more step in the chain which is that go cni, actually imports. um What's the name, a container networking, slash cni, which is uh agnostic to the container runtime and is maintained by the cni working group.

D

F

So there's like one more step, which is what's yeah, that one exactly this.

A

One right yep, so this imports that and I suppose, that's probably what parses the json correct.

F

Yes, matches and that thing has a hard-coded list of versions that are currently supported.

F

But I think that people will of course strive to keep this backward compatible so especially with people using potentially older cni versions, and things like this, so even when container d is updated in your communities cluster and actually supports 1.0 and maybe starts leveraging check. I would expect that things would keep working. It's just that you wouldn't get the benefits of check. For example, if your cni doesn't support it.

A

So can you tell us how andrea uses check? We only have five minutes left, so this is a good time to learn this. If.

F

Yeah, I wouldn't say that andrea uses check entry, a provides check and we should probably advertise that by at least updating the cni version too. But.

A

How far does it go like when, so I guess what I really mean is: how does it implement it right like? Does it what what is a check from andrea's perspective? Is it like if I go, break some ovs rules, and I call cni check, but well that still is, will check dynamically, like figure out that something's wrong, or does it just check.

F

If there's a nick check that your pod is connected to the obs bridge uh with a vsphere on linux, for example, I don't think it's going to go as far as check that no one has messed with your ovs flows. um I mean entry has should have, I think, that's reconciliation logic where, if you kind of like ip tables right, if you delete an ip table rules manually, you wait like a reconciliation cycle of like 30 seconds or something, and then your role is going to come back right so entry.

F

I should have the same thing for ovs flows. Now I think check just make sure that um your pod is still connected to the ovs bridge, and maybe the ip configuration of the pods interface is is fine, because that's not something. We have a reconciliation loop right. If you exec into the pod and you change the ip address, maybe that's that's going to break things and that's not something you can recover from.

G

Where is the check implemented? Can we look at it.

F

uh So, where are you now you are in, uh I think it's an age agent agent at the top, yes cni server and it's gonna, be, I think, if you look at uh server linux.go and you check and you find check somewhere.

C

The cni plugin contact connects to entry enter agent via jrpc right.

F

That's connect, that's correct, so the cni binary itself is just a thin grpc client, so it doesn't do anything itself really the way it works. Is it talks to the entire agent over grpc and basically forwards everything that comes from the container runtime to the entry agent? I think that's pretty typical to do that, because you want your logic to actually happen in the in the demand set right.

F

So let's look at.

A

The check specification here so check is a way for a run time to probe the status of an existing container plugins consider so so. This was actually really important for us on the windows side, because things actually go wrong in windows, um looks and so mark came back to to laugh at that so um yeah.

A

So things go wrong on the windows side and and for example, like we have a race condition in you know where right now we have a thing that like waits for the plug-in to be up and then like it like waits in a gofunk.

A

I think, like you know it like goes and it tries to wire the windows pod up and it sort of pulls until it's ready and then it like, you know, comes back, and I think that that's like that's like why this check thing is kind of important to it would be really nice. If container. Do you just use this because then you could have all that logic in container d and your cni wouldn't have to do things like that, but the plug-in must right now in andrea for windows at least.

A

I think we do some polling logic um before we return that the nick is set up. Yeah.

F

So check would be useful for tasks that, at least in the entry case, need to happen, asynchronously right compared to the cni ad and for which you want to be able to return errors and do things like retry. If one of those things go go wrong in the asynchronous tasks. Oh.

A

So you're saying you yeah well so, but well I guess what I'm saying is that you would want container d to run a check and then retry an ad. Yes, yes, that right, like so.

A

Plug-In must consult the previous result to determine expected interfaces. Plugin must allow for later chained plugin to have modified networking resources. Plugins should return an error if a resource included in the cni result type was created by the plugin. The plugin should return an error if other resources not tracked in the result.

A

I feel like a lawyer right now reading this, so the plugin should return an error if it is aware of a condition where the container is generally unreachable.

A

Oh okay must handle check being called immediately after an ad, therefore, should allow reasonable convergence for any asynchronous resources. This is the you know.

A

Plug-In should check on any delegated ipm plugins and pass okay. Runtime must not call check for a container that has not been added.

A

um Okay, so here's the contract. So here's how we implement it. So we go and we what do we do? We container access, lock container defer unlock? What are we locking.

F

It's to prevent uh race conditions, I think uh uh well, I'm not the expert there, but I think it dates back to uh kind of like when we first added support for windows and at least at the time there were multiple calls to uh to the cni commands for the same same pod.

A

Okay, okay, so we wait and we say if you just called ad we're, not gonna like do this, while you're calling ad because we'll get confused if you're calling it a second time or whatever we lock. So we check ipm. So this is checking so this is going and I think.

F

A

F

Talking about which is probably make sure that the ip address is still configured for your pod.

A

C

This is a delegation for the ipad.

A

C

A

Okay, so that seems not specific to the pod, though this is just checking that.

F

The ipam is configured in some. No that's specific to the part. I think what you'll have in the args here is something that identifies the pod.

A

Okay, so the so you go to the driver like the ipm driver and then the driver says: okay yeah. I can confirm that this particular pod has been allocated in ip by me.

F

Yeah because if you think about ipam plugins like host ipam, which we use in that case here yeah it's, it also implements the cni interface right. So you just kind of like pass it. The arguments you you get from the container runtime.

E

F

So it can also perform its own cni check.

A

Yeah, so here's the ipam that we're talking about here so okay. So then we see an iacmd response. Okay, so we return that, but we don't so we don't do anything else.

A

I feel like we could do more here, but maybe I'm just making it up. But yeah I mean.

D

Could we get.

A

Clever and like go talk to go talk to ovs and look if the nick is attached.

F

Or whatever something like that? Well, we didn't. We didn't check what the validate prep result is. uh I don't know if you actually do something else here, parse previous results.

A

So this is oh here's your cnio.4.0 hard coded again.

F

Yeah, so you also have that checking intro, which is strange because the only version we advertise in the cni confine is 0.3.

A

I'm going to also note this should be a constant in this issue that I filed so.

E

It seems like one also seems like we hard code.

E

Hold up for I don't know in the check imp implementation should that be a variable.

E

A

Feel free to I don't know if this is really a actionable issue, feel free to feel free to close it if it's not um so, okay, so parse previous result and then validate previous results.

F

Validate prep result will actually do some. Some checking right, parse.

A

Container interface from results, so this is going and I.

F

Would say, check check interfaces here. um The last call, but configured it would be, is probably what you're looking for in terms of doing some checks and check container interface, pc, pod, configurator.

A

Yeah check container interface. So then I talk to this pod configurator and then I go and I check that. Okay. So now I have the sandbox id and then I go in here and then.

A

Ovs hardware offload sandbox in previous result doesn't match. Okay, oh look! So now I'm going and I'm looking at the interface name and then I grab the. I grab the interface based on the container name, and then I check the container's mac config. Oh, so we're checking a lot of stuff.

F

Okay and I think, validate expected interface, ips is.

A

Yeah so then I go and then this makes a call to.

F

So this basically looks at all the ip configured for the container and make sure that you didn't mess mess with them or one didn't vanish through some uh magical operation.

A

Okay, so this is still at the linux sort of os level. It's not at the lvs level. It's at the linux place.

F

Yeah right, I I would think there are some obvious checks, but uh I would need to look at the code.

A

But I think there's not because it's.

F

Right like well that wouldn't be in this function that would be in the parent function, but.

A

No, it just returns, see.

F

That would be in yeah.

G

F

I would expect it to be in check container interface, but if it's not, then.

A

No, it's not so we don't.

A

No, it just goes to the linux os level v pair, okay, cool all right, so this is cool, so we're out of time. But this is um I'm going to um add one more bullet here, which I'm going to say is related, which is that also we should better. We should comment what we do in.

E

Our check implementation, if we are going to start supporting newer, cni cni.

A

E

Versions right so maybe we should add a comment about.

A

I.E, do we check ovs routing rules.

E

Etc or just nick attachment.

E

Just for good measure, cool all right.

A

Anything else folks want to talk about hey. What's this, I don't know.

G

What I wouldn't click on that, if I were you okay, I I.

A

We're out of time thanks everybody for for thanks everybody for coming to hang out today, thanks antonin and mark for for showing up and uh ameem, and I guess ameem and chinchi, and who else is next week mike are gonna do next week? Are you doing the show with mike next week I mean, or is it just cinchy.

C

Yeah me and gigi are going to yeah.

A

Cool all right, so the three of them we're going to be doing next week and we're going to go another level deeper into the exact same thing. We did this week we're going to go into the check implementation in container d. So we're going to look at.

A

A

G

E

A

Look at let's see where the issue is check.

A

um I don't know, I don't see it, but it doesn't matter so next week we're going to go next. Go go next level on this. So come back thanks! Everybody for coming andrea, live every wednesday four o'clock eastern like and subscribe.