Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Introduction to CNI, the Container Network Interface Project - Casey Callendrello & Bryan Boreham

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Introduction to CNI, the Container Network Interface Project - Casey Callendrello, Red Hat & Bryan Boreham, Weaveworks

CNI, the Container Network Interface, is a small but critical piece of infrastructure linking runtimes such as Kubernetes and CloudFoundry to dozens of different container network implementations. This session is aimed at users and developers who have little previous knowledge of container networking. Attendees will hear: - A broad overview of what CNI is - What the CNCF-hosted CNI project has delivered - How CNI relates to Kubernetes - How they can get more involved in the project

https://sched.co/ZewR

A

Hello and welcome to the cni intro session, given by two maintainers of the cni project, if you're new to cni we'll give an overview in just a moment. If you have questions, we are live online so, as this goes out, so please start asking as an introductory session. This is a high level and pretty much starts at the beginning from cni. You may also wish to join us for the cni deep dive session, which is being published on wednesday.

A

So first some introductions, brian.

B

ah Thanks casey hi, I'm my name is brian ballroom. I work for waveworks and uh known at least to begin with for weave net, but perhaps better known for githubs. These days anyway, I've been the lead maintainer on weavenet for over five years now and I'm a maintainer on the cni project, and I I work on a few other cncf projects like cortex casey. Why don't you introduce yourself.

A

Sure so I'm casey calendrolo. I am an engineer at red hat previously core os. I work on kubernetes, networking, openshift networking and the ovn kubernetes project also maintains some utilities that people tend to use like go ip tables, for which I'm somewhat sorry. I like ipv6 and jumbo frames and the kubernetes informer system, and I dislike nat and the kubernetes informer system.

A

So we'll start out with brian explaining what exactly is cni.

B

Thanks casey, um so we put a uh the big picture here. The uh just has three things on it. You your container and the network and um and cni, sits in the middle. So this represents uh the the big picture. The point we're trying to make here is that um cni is intended to sit in a world where people use all different kinds of networks. They use all different kinds of ways of getting their containers working and we'll go into more detail about that as this talk progresses.

B

um But but fundamentally it's it's the bit in the middle. It's uh it exists as a common ground interface, so that all different companies and and people can contribute to this world and have them work together.

B

So here's a more detailed picture, uh the um uh what we've got on on the right uh representing the um container runtime uh yeah. So it's you know it's not literally the person that sets up a container at some software, um perhaps using kubernetes, although cni works with um others, mesos cloud foundry a whole bunch of different um orchestrators for containers um and those go through some kind of container run time like uh podman, cryo container d.

B

um So so that's kind of putting more detail on the uh on the whole um uh ecosystem on the right um and the cni project is everything in this box here. um It's uh uh things that the libraries that the runtime calls its plugins, which implement the actual network connection um and we'll come back to more detail on this in a bit casey. Why don't you tell us a bit more about the um the project.

A

Sure so we're both here as maintainers of the cni project, which is an organization under the cncf umbrella. As a project, we have two major outputs. The first is the cni specification document, which defines the cni protocol and a reference implementation in go of that protocol. This reference implementation, as it happens, is used by both container runtimes and cni plugins, and it's generally widely used, although not the only implementation of cni used in the industry.

A

This this section of the project or this output, has a pretty slow, relatively stable release cycle. As befitting a specification. The second output of the cni project or consumable is a suite of commonly used cni plug-ins.

A

These include some basic connectivity plug-ins such as bridge and some chained utility plug-ins, like bandwidth, limiting host port mapping and some other tweaks most of the plugins are linux only, but we do have a selection of windows, plugins maintained by microsoft and other contributors, and you can find all, of course, all of these on github to dive in a little bit more into the specification.

A

The specification really is the heart of cni right. It defines the protocol between the runtime and the plug-in, as brian showed earlier, because the cni specification is a simple vendor-neutral, stable specification. It's an industry standard, that's widely used, it's not just used by kubernetes by mesos cloud, foundry, et cetera, et cetera. We see we've seen the list before the key part of the specification is how plugins are discovered by the runtime, the execution flow and the data types for both configuration and the data types for responses as well. The specification is not complicated.

A

Generally, it's backwards compatible for the average end user. The specification- it's probably not going to concern you too too much other than perhaps the version, but it's good to know that the document exists and you could probably read it if you needed to uh the next question. For you, brian, then, is what isn't cni.

B

uh Yeah, um so we thought we we'd put this in because, as maintainers on the cni project, you know almost any problem. You're having with your system can look like a network problem, so so we do get a lot of different questions and unfortunately we don't we don't do every part of the the network uh picture.

B

um So cni is really about con, connecting containers or pods to the network um and if you're, using a system like kubernetes, it will provide in different ways things like load, balancing service, discovery, um network policy.

B

There's uh there's all these different things that um go together uh and and cni is kind of. I mean the cni project itself. Is this little bit of glue um and very little? uh uh You know a lot. A lot of problems come from elsewhere. um It's also, as I mentioned already, cni is not kubernetes specific.

B

So even if we could perhaps make a change in cni, uh that would um you know we we have to. We have to be even-handed across all those different uh systems and um and the cli project, although we do have some plugins that that we're going to talk about uh in our um repo um a lot of people are using uh third-party ones that that you know I work on one. As I said, we've met, calico is very popular silly and there's there's a whole um bunch of them and that's great.

B

You know that is the whole point to to allow uh different people to to implement things so um yeah. So the cni uh that we're representing as maintainers of the project um is the uh is the bid in the middle and there's all this other world. That is not the cni project, although we, you know, we, we talked to those folks um and generally work together, pretty well, but it's not the whole world.

B

uh So, let's take a deeper look into: how does it all work, casey.

A

For context, we can go and dive a bit more deeply into this diagram, as we had said before, cni is the protocol for container runtimes, which we see here on the right, which is podman et cetera, et cetera, to request that a network plug-in such as weave or flannel, which we see here on the left, connect a container to the network.

A

Your container runtime is going to first create the container. Then it's going to call out to its cni plugin, which will create and configure a network interface or more than one network interface in the container's namespace.

A

We see here libcni, which is our reference. Implementation, usually mediates, both sides of the protocol, how the cni plugin actually works, is not defined in the specification. Likewise, the plugin doesn't need to know anything about the container runtime, so it is abstracted over that.

A

Note by the way on this picture, this is just an aside that, but it's an important understanding to have the cubelet is not present on this slide. So remember, the cubelet doesn't actually create containers itself. It talks to its container runtime engines by something known as the cri or container runtime interface. The cri implemented by runtimes is what actually creates the container and the runtime is responsible in the kubernetes world for doing the lib cn for doing the cni uh next up, brian is going to tell us a bit about the execution flow.

B

Yeah, so um so, if you keep that picture in mind the the runtime calling the plugin, it can call it with one of four different commands. We only have four at the moment, so it's pretty simple um add adds an interface delete, takes it away and then there's some check and version a check to check that things are still the way you thought they were and and version just uh you know administratively finding out what version the thing you're talking to is: um plugins are executable. So that's the way you call a cni plug-in.

B

Is you run a program and the program is the plug-in? It's a really simple interface. um The details about what to do are passed in on standard in on this program. That you're running uh in an adjacent format and standard out uh comes the answer. The response from the plugin, so the whole thing is just extremely simple and.

B

That is the whole mechanism of calling a cni plug-in. Okay, so casey take us through what that looks like when you actually.

A

Make so this is the configuration format, uh it's the cni standard, it's a json based configuration and it usually lives on disk. uh It is consumed somewhat it's consumed by both the runtime and the plug-in, and unfortunately, it looks like the highlighting hasn't carried through I'd like to point out so the first three configurations fields here, version name and type are consumed by both the runtime and the plugin. They are used for version negotiation and plug and discovery.

A

The remaining sections of this sample configuration file are plug-in specific, so they are not consumed by the runtime they're, not even really part of the cni specification. So as such, they are open for you as a plug-in maintainer or a system administrator to use. So it's a really simple format and uh it's a little bit of a hybrid of both the runtime and the plug-in side configuration in a single file.

A

However, a simple configuration file has ground up to not be flexible enough, so in specification.

A

Version there we go okay, so uh the spec version 0.3 has we've defined something called dynamic configuration or, in this case we're calling runtime configuration.

A

This is a way for a cni configuration file to signify to the runtime where it might want some container specific configuration to be inserted, and this means that configuration files can mostly live on disk and mostly be static. However, the actual interaction between the runtime and the plugin can be dynamic.

A

So on the left, we would see a json configuration file on disk. It's very simple, but we see here it has the section capabilities, and it says that I support the port mapping capability well for each container invocation. If the runtime so desires, it may substitute a port mapping capability with some port mapping, runtime configuration where we see that this particular container happens to have a host port configured a container report.

A

This is to say we would like a very specific chunk of container specific information substituted into the configuration and then the other exciting thing inspect 0.3 is plug-in. Chaining and brian's going to talk to you about that.

B

Yeah chaining, calling a chain or sequence of plugins in a row allows us to compose functionality to to add things in um so not every plugin has to kind of paste in all of the features that people want. um So this example on on the slide has has two plugins listed, the the bridge plug-in, which is, is um you know, just just uh bridging all the traffic from several containers um and then followed by a port forward plug-in so it's kind of the same example or same area that casey was just talking about.

B

um If I want to uh bring out some ports from my container to the host, the port, that's what the port forward plug-in does and um and the bridge and weavenet and uh calico, and whatever do not have to re-implement that functionality, because they can, they can put it in a chain or, more specifically, the person uh configuring, the network, configuring, your own cluster, can can construct that chain in exactly the way you want to do it.

B

So um so that's a powerful feature. Each plug-in in a chain operates on on one interface, so this is. This is not about running like the same plug-in, multiple times to give you multiple interfaces, um you you just you know if you, if you want to do that, you just do it with with uh separate configs um and we'll talk a little bit more about that later, but yeah, that's plugin, chaining.

A

So next brian is going to talk to us about how the cni project actually works as a open source.

A

B

um So we have uh the the parts that we've talked about already the the spec. um This is actively maintained. We we have a maintainers meeting every week and we we look at questions that have come in, or issues raised against. The spec people submit pr's against the spec, but it tends to happen quite slowly because the this making a change in the spec might require a lot of plug-in authors to to do rework, um so it tends to evolve very slowly over time.

B

We uh are really close to um announcing 1.0, which a lot of people are looking forward to, because that'll be a declaration that we're really going to try even harder not to change anything fundamental in the spec. um So that's that's close month month or so is what we're thinking right now.

B

um The plugins, however, uh go much faster.

B

You know they're they're, individual bits of software features are added, bugs are fixed, just you know, as as they come up and we we have even more contributors there, a lot of people from around the world using cni software and they come across something they'd like or they come across bug, and they can, if they know, go code, they can submit a pr um and, like I say we do a maintainers meeting once a week that tends to be where we we sweep through things that come in and make sure we catch up on all of them uh and at the bottom of the slide, uh cubenet cube proxy.

B

Anything like that is is the kubernetes project, it's not the cni project and um they have an organization sig network uh which, which kind of owns all that stuff. um So, although we're very good friends with those people, it's it's a different project works on in different way, and it's not part of what we're talking about in this talk.

B

So we have um seven maintainers uh right now and, as you can see, they come from um a bunch of different companies.

B

um Typically, software companies- I mean you, know we're not really network people, uh we're not we're not network engineers by trade, um so uh we're mostly kind of application, developer type people who happen to know a bit about networks and wish to contribute. It's really a volunteer effort.

B

um You know we're, there's no kind of billion dollar business on the back of cni. uh So um so those are the people that uh um you have to thank for uh making this thing move forward each week.

B

um So talking about what's happened recently uh in the spec, not point four. That was the most recent version that went out. We um we added this check command, which took a lot of of debate and thinking about how to do it. um We added static ip support, changing support for changing the mac um layer, two bridges and uh uh yeah a little thing that was needed to play better with the kubernetes device plugins.

B

So um I won't read the whole list, but uh you know you: can you can go over to the github, slash container network, cni or plugins, uh see everything that's been going on um cves? That's that's! A kind of colloquial term for for security.

B

Vulnerabilities do come up and those generally cause us to to jump, uh although we tend to have to do that in secret, um because we're trying to get the fix out before the problem is widely known um anyway, that the deep dive session, which, if you're watching live, is tomorrow, um I go into quite a bit of detail about some vulnerabilities we've dealt with recently. So, if you're interested do come along to the deep dive, cni deep dive session, all right, I'm going to hand over to casey to talk about. What's coming up next.

A

So uh trouble advancing the slides if someone could help me out.

A

In any case, oops.

A

Alrighty there we go so uh what's next right, as we said 1.0, we essentially consider it to be feature complete. The spec is essentially stable. We we essentially need to crystallize that, in order to cut 1.0, we need a bit more precise specification language that just needs to be tightened up a bit and we'd like a more secure or signed release. Binary is a more secure release mechanism.

A

What's next, after that, we're starting to dream about spec version 2.0 and what that might look like. Are we going to do something like grpc, about which we'll chat a little bit more in a bit? Do we want a richer life cycle? Create and delete are, of course important, but they're, certainly not the only thing in a container life cycle, as we all know, but the other ques. The other question for the community is: what do you want? What can cni do to help advance the state of container networking?

A

It's an open source project, it's a public specification and we would absolutely love your input a bit later, we'll talk about how you can get involved in the project. Okay.

A

So now we're going to quickly go through some frequently asked questions that often have come up when we've done these sorts of sessions before or in slack and maintainers, meaning in other emails. The first is: why do some cni program plugins, especially those in kubernetes, come with the host side demon and the answer has to do with life cycle.

A

Cni is only concerned with the life cycle of an individual container. That's pod, that's creation and deletion. You need some sort of long running process to maintain routing tables. If you have a higher level system such as kubernetes, you need to react to events that happen across the cluster that are not container level events. The demon might, for example, set up routes or react to node changes or something like that.

A

A question for you, brian, then, is uh brian. Can I have multiple interfaces in a pod.

B

uh Yes, you can casey, um so cni has always supported multiple interfaces in a pod right right from the beginning. um All you need to do is call it multiple times with the different config that you need, um and you can call the same plugin multiple times. You can call different plugins with different configs on the same container, and you will have multiple interfaces in your container.

B

The reason why this question gets asked so much, I think, generally stems from the fact that kubernetes only supports one interface in in a pod, um the the plumbing, the the number of concepts in kubernetes that um they're based around the idea that there's only one interface is, is large and a huge amount of work to um shift from that position. So um so what people tend to do to work around that problem is, um is they run a cni plug-in, which itself calls multiple cni plug-ins?

B

So multis is a good example of that um there are. There are a couple other. There are cni genie dan m, uh coming from predominantly a telecoms background. So um so, if that's you, uh if you're in the telecom world, definitely suggest taking a look at some of those. um Also, you know if, if you have ideas, uh you know back to what casey was saying.

B

What would you like next, um particularly if you want to uh contribute code to help make those things happen? Please do um you know show up in uh in the github issues or in our slack channel or some other way, that we publish to contact us we'd love to hear from you. um But yes, so, back to the back to the simple question. Cni itself supports multiple interfaces.

B

Kubernetes really only deals with one interface for a pod, um but you can generally get around that by by faking it with something like maltose, well, gee that took a long time so casey you can uh take the next one. What about deployment.

A

Sure right, so, uh if you have a cni plugin, how might you deploy it right? Well, the good news is that all you need to do is write a couple files to disk. You don't need to set up a demon unless you do you don't need to do any complicated host level configuration.

A

The best practice in the kubernetes world is to use a daemon set that leaks these files onto host, but any mechanism by which you like to configure your nodes. This will work. This will often be provided to you by your vendor. We would like to come up with a better story for how this is done.

A

Perhaps the cluster add-on sub-project as part of sig cluster lifecycle will be the answer. So we have an answer- it's not great, but it works reasonably well for now, so brian, do we support ipv6.

B

uh Yeah we do casey wells again, uh cni um supports ipv6, uh we added into spec a couple years ago and the the plugins about a year ago, all the plugins that we own anyway support uh as the cni project support ipv6 for over a year um again, uh it's really complicated to plumb that into kubernetes, and um so they made the first step uh which, which is, if you want a cluster, that's only ipv6, um so that works pretty well.

B

uh Most people probably want a dual stack, ipv4 and ipv6. um That is still marked as alpha, in other words, not not really completely ready in in kubernetes.

B

um I think it came in in version 1.16 of kubernetes um and is is still uh still got a few rough edges, um but check it out um and again they need help from people checking it out and people making it better.

B

um So casey what about this? uh You know. Why are we executing binaries.

A

So the cna protocol does kind of seem a bit old-fashioned. Why not grpc like every other, kubernetes and container related api, like cri, csi and so on and so forth? Well I mean it's a long story and somewhat historical, but ultimately it's because cni came out of rocket. The container runtime engine from coreos, which has which is always demonless executing binaries, is a really simple api, don't need a daemon and it just fit in with the entire ecosystem. The simplicity ultimately led partially to cni's adoption.

A

However, the world has moved on. Things have changed and we would like to think about opening up the possibility for grp ac in cni v 2.0 we're running quickly out of time, but brian, would you like to highlight some cool chain plugins.

B

Yeah, uh so we talked about chaining earlier calling one thing then another then another on the on the same interface and- and so people have really run with that idea. um You can do your port mappings, you can tune things like tweaking the mac address and and uh linux system settings. um Actually that reminds me. One thing we wanted to stress: uh cni plugins exist for linux and windows.

B

I in my job I tend to work on on linux, but we shouldn't uh lose sight of that. The um the ecosystem works on on windows too. So um sorry, back to back to the question I'm supposed to be talking about, um another kind of tuning is, is bandwidth, but it's actually separate uh plug-in, uh doesn't matter. You can call as many as you like.

B

um We have a plug-in to try and interact better with uh firewalls on on linux and source-based routing, choosing where exactly your traffic is going to go based on the um where it's coming from, which again is a cool feature that some little niche, but some people like that.

B

Okay, I think we are done on our list of uh frequently asked questions. So, uh let's put up some links um the deep dive session I've mentioned already, if you're watching this live it's tomorrow and the github links where the code lives and the cni is the spec and library, that's the bit that moves a bit slower and the plugins.

B

Well, that's the plugins moves quite a lot faster. We have our own slack channel in the cncf slack, um but do remember if your question is really about kubernetes. uh They have their own area, in fact they're on slack. So I put a link up for that too casey. You want to wrap up.

A

Sure I thanks everybody for coming uh we're here to answer questions and please join us, of course, in the slack and on github, and I will end the presentation with this slide just to leave it up, so we can reference it if there's any final questions, but thank you very much for watching.

B

Yeah thanks bye.

A

Oh hello, live.

B

Looks like we're live online, and um uh so thanks for all the questions, great questions coming in, we answered a lot of them uh by typing. I wanted to pick up uh one that came in earlier, uh which is basically how how what are some things to think about when you're picking a cni plug-in and um uh that which is obviously a great question. um So I think I would. I would highlight one key difference there.

B

There are some of them that work as what's called an overlay, so they they kind of make a whole virtual network inside your real network, and there are some that run on your real network, and um so basically uh you you might have constraints on your real network. You might have not enough address space or limitations or or something like that. That might really drive you in the direction of an overlay.

B

So that's going to narrow the the choice down. Overlay has a little bit of extra work, so there's a performance hit to to run an overlay. um So if you don't really need an overlay, maybe you're going to choose one of the ones that doesn't do that. um But that's that's a really uh pretty fundamental divide um between the ones that work by being an overlay and the ones that don't um then, beyond that there are some features uh you might look at like. Do they do encryption? Do they do network policy?

B

um Can they handle, for instance, multicast traffic um that that if you had a specific need, that's going to narrow down your choices? um After that you know, sort of personal preference, or uh uh you know brand recognition or something like that. The the um uh that's one of the great things that we really try to create with this whole world is is a lot of competition, and um I'm sorry that makes life harder because you have to choose, but we do think competition drives the bar up in general for everyone. So there you go.

A

Right and if you're using uh something say like multis you're doing something where you are not using kubernetes and you care very specifically about the networking technology, so full disclosure, I work on openshift and openshift makes it relatively easy to use multis to install extra networking plugins and for those that, if you're doing something advanced like that, you probably care a lot more about the specific technology. That's being used right. Sometimes you just want to bridge.

A

You want to use a bridge where you're going to bridge your east, 1 or east 2 into something else, or if you're, very performance sensitive you might want. You might want to be using something like sriov, where you use a device plug-in and so a little bit more duct tape to pull it all together, such that you have a dedicated pci lane for whatever it is that your advanced networking configuration might be right. So in that case you do need to care about the specific plug-ins networking implementation, as opposed to just thinking of that.

A

As a unimportant detail.

B

uh So a couple of more questions: um we could look at uh as long.

A

B

A

That's a dream: yeah! Let's do that, one right, so uh yeah, so dynamic configuration is like, like I had mentioned in the presentation. As for anything, that's plug-in specific. I'm sorry that it's container specific, so the two most widely used instances for this would be bandwidth and host ports, but there are others as well right. Kubernetes itself has some glue that uh puts together uh host ports, which are a fundamental part of the pod spec and bandwidth, which is a magic annotation.

A

um In addition, so for example, in the multis world that we have built uh the multisand castle that we have built in openshift, we also use dynamic configuration, so you can give pods static ip addresses on their secondary interfaces. um So one of the dynamic configuration conventions or standards is to request that a specific pod have a specific ip address. I'm sorry that a specific interface have a specific ip address right. So that's like a classic example of dynamic configuration.

A

Since you don't want every container to have the same id address unless you're totally crazy, which some of you may be.

B

So there's a question: does overlay reduce the ip address management? I I don't entirely follow what that means, but it maybe refers to something I said about if you were short on address space, um so the the point being that an overlay network uh is is isolated. It runs effectively in its own address space.

B

um So, for instance, you you could run 10 overlay networks, uh all reusing the same set of addresses. This is more of a problem in ipv4, um but then that's that's. What a lot of people are using um so uh yeah an overlay gives you some flexibility, because, um as long as you don't want to talk to an ip address uh outside of your cluster, you can use whatever address you like, because you're you're in a little bubble.

A

I think we are out of time.

B

A

B

And hand over to casey.

A

uh Sorry, uh my laptop decided to drop out your audio, but uh in any case uh thank you, everybody. Thank you, everybody for coming. I think we have to end the presentation here. uh I hope the rest of you have a great uconn and I'll see you in slack. I will see you all virtually in all the places in which we gather.

B

A