Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Deep Dive: CNI - Bryan Boreham, Weaveworks & Bruce Ma, Ant Financial

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Deep Dive: CNI - Bryan Boreham, Weaveworks & Bruce Ma, Ant Financial

CNI, the Container Network Interface, is a small but critical piece of infrastructure linking runtimes such as Kubernetes and CloudFoundry to dozens of different container network implementations. This session is aimed at implementers of network plugins and runtimes using CNI, as well as anyone interested in contributing to the project or becoming a maintainer. In this session we will take a close look at the Tuning and Firewall plugins from the containernetworking/plugins repo. We will also have plenty of time for discussion and Q&A.

https://sched.co/ZexP

A

Hello and welcome to the uh cni deep dive.

A

um We've got a couple of the cni project maintainers here um before we get started. I uh just want to put up some information. This is this is the slide that usually appears at the end. I will put it up again at the end, but uh if you um are totally new to cni, I'd really recommend you start with the intro session uh which which, if you're watching this live, streamed yesterday, but uh there's a link there.

A

um If you want to look at the software, the spec anything like that, it's in the those github repos and we do have our own slack- um although I I hope, uh if you're live, I hope you're on the kubecon chat and uh we do have uh we're here bruce and I should be live at that time. So please get asking your questions. If you have some this talk, we're going to go through very brief.

A

What is cni just kind of set the scene bruce is going to talk it's a deep dive, so he's going to go really deep into the technical details of cni extensibility and then I'm going to talk about some security issues that shed a different light on the project.

A

um Then we should have time for questions at the end, so let's get going.

A

B

Yes, hi everyone. This is bruce, I'm a software um software engineer at san francisco from china.

B

My major work is, you know on some container networking, including the you know, building some network plugins for our communities, clusters, the ingress controller and the best operator, or something like this uh for the cni team, I'm a freshman. I joined the team since september last year.

B

My github id is mars 1024. So if you have any questions, just ping me online, hey brian.

A

Thanks bruce um so my name is brian borum. I work for weaveworks, uh which is is probably best known as the gitops company right now, but we were very early into the container networking scene with weave net. I've been lead maintainer on weavenet for about five years now, I'm also a maintainer on the cni project and a couple of other cncf open source projects like cortex, for instance, I'm on github tube, although it's probably more entertaining to find me on twitter beborum is my handle most places.

A

Okay, um let's open up. uh I'm just gonna give you the the um very uh high level view of uh what is cni, um so the basic idea uh is that uh that you, or or probably a container runtime run by you, uh has some containers uh and you wanna attach them to uh the network, which is at the the bottom of the screen. um So the uh cni is the kind of the glue in the middle.

A

um A lot of the the hard work about setting up the containers is one is done by what we refer to as a container runtime. So that's very commonly kubernetes, but cni is used with mesos with cloud foundry with rocket with podman, so cni is agnostic to all those different runtimes and it's equally uh wants to be even-handed, about which network you're using so cni defines a specification.

A

That's the most important part how you talk and make a request, um and then it we also have some library code to help you implement that and some some um base plugins that uh are used by many people.

A

But uh it's important to say that that a lot of the um cni kind of sits in the middle and a lot of questions. People have are really about the upper layer about who's calling cni and a lot of the questions about the lower layer, um uh who's implementing cni. But um that's that's where it sits. That's kind of sets out the uh overall position.

A

Okay, I'm going to hand over to bruce now and bruce is going to tell us about the whole subject of cni extensibility.

B

Yes, first, let's take overview to the singing accessibility.

B

As we know, the singer has defined the specification between the container runtime and the plugins, but this is not enough, because sometimes we want to pass some. You know dynamic or custom information to our plugins uh for some case, for example, if we want to assign some ip address for a container if you want to set batteries host, formatting, mac or mtu for our container interfaces.

B

This is this. Is this cannot be done by spec? So to handle these user cases, we need to override the impulse to our plugins. This is called ci accessibility.

B

You know beyond spec, and we have been thinking this question from the beginning of c9 and actually they say I have evolution way from the senior arguments to arguments in conflict to capabilities.

B

uh In my opinion, I think this way is, you know from easy to wise from free to customized.

B

So, let's come to the first answer of: let's come to the first answer of cnn accessibility to sing arguments. The senior argument is born with the original scene aspect. You know it appeared long time ago and just as its name, it was passed by the environment variables.

B

You know when we're calling the plugin and this format is some key value pairs separated by semi columns in a single string- and you know uh seeing arguments is you know it's an easy way? It's easy to be implemented and easy to be used to be big to be debugged, and because we can cameras, we can carry several cubed pairs in a spring in one time it's flexible, but scene arguments has a worst limitation. It's because the string is flat.

B

It cannot it's not easy for for it to hold the structure data so such like jason, so cena arguments has been deprecated and takes the lowest priority.

B

So here's the question: if I uh do not use events to pass our arguments, what should we use instead? So with this question we come to the second answer of cni accessibility. The arguments in configuration you know please take a look at the right picture. We can see the arguments is a special field in the network computation.

B

So this is because we think the json network config is a better place to hold all the dynamic information, and this way is easy to hold the structural data, because it's just in a structured, editor, jason and the history. The argument in configuration is introduced in spec 0.2.0 and is implemented in code release 0.4.0.

B

But this way we still have some limitations. The first limitation is, you usually.

C

B

Get the network configuration from disk it's a fixed value, but if we want to fill the dynamic information into it, we have to read it pass it and fill it. You know override it, so the override function has to be implemented by ourselves. You know by the user.

B

The second limitation is, you know, just at the format. You know we carried all the dynamic information in a single map, so this map will be passed to every plugin. Whatever the info is needed or not, so I don't think this is a good way. uh The senior accessibility is designed for carrying extra information.

B

You know the ability beyond spec, but should it be fully customized? I don't think so, because this will cause some controllers. We want to pass specific arguments to specific plugins.

B

You know this is what we want and with these questions we come to the latest answer. The capabilities, yes, which has another name, the runtime configuration and the capabilities, take to two main advantages over the previous ones. Two ones- and the first advantage is you know, plugin can define, is additional fields in network config. You know this can be used as providation and, secondly, the we can use this additional failed to to you know to filter the information pass, specific information to specific plugins.

B

This will help to remove related information, and the second advantages advantage is the libya similar has implemented uh override function for us.

B

We we don't have to implement a config override function ourselves and then the picture below shows how the capability works. You know here is a plugin named podmapper and it declared in the configuration that it needs a capability named port mappings. Then the libsima will gather the dynamic information from streams and into a map. Then the libsaner will search the map with with the key for the mappings, a font, it will generate a new json which is filled by the value.

B

Yes, you can see the runtime config field is a new field filled by filled by the lip scene eye. Then the the website will pass the new json to the plugin. You know with extra information.

B

So, as we know, the c9 accessibility has, you know a long journey to the capabilities, but is capability. The final answer for the dynamic configuration uh for now, in my opinion, I don't think so, because capabilities still have some limitations.

B

The first limitation is, uh if you want to introduce a new capability, it needs some cost. First, we need to change the network config. Secondly, we need to do some code changing container runtime. This is, you know, cost uh the second limitation is now.

B

The capabilities is only come from continue right time, which means that the actual information can only be passed from upstream to plugins, but in some cases we think that you know the extra information can be generated by the previous plugin, which is expected to be passed to the next plugin in this case, capability won't happen, so I I think capabilities still need some improvements in the future.

B

First, I think capability needs self discovery and, secondly, the I think capabilities should be passed over plugins. You know not just from the upstream uh that at last I think you know for the exam team. We should provide more utility package about the c9 extensibility. So if someone want to introduce some, you know extra information to their plugins.

B

We want to make it easier.

B

In the end of my part, I will show you a comparison summary. You know the the three kind of ci accessibilities uh from different perspective. uh I think the capability is the most recommended way for the ci accessibility. So if you want to extend this in life, use capabilities, so next part brian will give you to something about the recent cves hi brian.

A

Thanks bruce um that was really interesting and uh just thinking while, while you were talking uh you know if anyone listening to this is is interested in in that idea of extensibility. um Please do a message in the in the chat here or come over to our slack on the the cncf slack.

A

The cni project is, is pretty small and basically volunteer run. So if you do want to engage with us, you have ideas for enhancements or or whatever. um If, if you're willing to do some work, uh please please come along and engage with us. If you just want to request it that we do work, uh might take a bit longer.

A

um Okay, so I'm prepared some slides to talk about cvs. What do we? What do we mean by cv? We mean some kind of security, vulnerability and they're they're, given uh numbers they're given unique identifiers, so that when we talk to someone uh we know we're talking about precisely the same issue. uh If we have that identifier, so I I picked a couple um uh first, one I'm going to talk about, has the uh glorious name of cve 2020 10749., it name doesn't mean anything like.

A

I said it's just a just a tag, just a unique identifier, but it is about this router advisory message um and then I'm going to talk about a totally different issue. After that, um let's look at the the first one. um What is the scene? I've got? uh I've got one host here. um The blue box represents your ordinary code that you're running some containers uh in your system um and the red box. uh Somehow an attacker is running code there. Now you might think you know my my system is protected.

A

Attackers can't get in and run code, um but maybe they maybe they attacked the upstream supply chain. You know, maybe maybe they coerced one of your developers to put the code in, or maybe uh some library that you're using some tool that goes into a container really difficult to to ensure that the whole supply chain is is pure and perfect. So um so maybe the attacker can get some malicious code sneak it in in the the guise of something that you want to be running anyway.

A

uh If that has happened and um the attacker sends this message, forges a fake rooter advisory message uh and and what they say in this attack is, is they they basically tell linux to send all the ipv6 traffic here um many many systems are running with the capability of ipv4 and ipv6, but really only ipv4, configured they're really only expecting to use v4. So if the attacker sends this fake message, saying hey, send me all your ipv6 traffic. Well, linux will obey that.

A

So the next part of the attack is when your innocent software tries to open a connection. You know. Maybe this is making a request. Maybe uh maybe an http request. Something like that. um There's this cool thing that called happy eyeballs, it's kind of a weird name.

A

You can look it up, I didn't invent it, and the basic idea of happy eyeballs is is that when you open a connection, it's gonna open uh or attempt to open uh a ipv4 and an ipv6 connection simultaneously, um and it's gonna pick the fastest one to to come back and the the reason for this. The reason this exists, this happy eyeballs algorithm uh is, is we're kind of in a transition from ipv4 to ipv6, and we we want to be even-handed.

A

We want to uh you know not not favor, one or the other. So literally those requests are sent out simultaneously as as far as possible. um So uh in. If you have this uh malicious code, unfortunately running in your system, then the um the v6 request is going to hit the the attacker uh because they're running right next to your real code, um their response is very likely to be the fastest one.

A

So what we end up with is that your client is now talking to the attacker and running a fake service. They do have to um run the entire tcp stack uh inside their code. The linux won't kind of legitimately root packets um to to uh a fake attacker in in this way, um but it's it is perfectly feasible to to implement a tcp stack in in user code. So um so this attack can work. um They can make fake responses, they can mount a man in the middle attack.

A

You know, take your take your real request and maybe change it slightly and pass it on. Send the money to their bank account instead of your bank account um this uh this. This vulnerability was um reported to us and we had to take action. So what uh what did we do? I've drawn a few uh different mitigations on on this slide. First, one over on the far right is, is the code change that we made.

A

This was to change a setting in to linux, saying, do not accept router advisory messages except ra equals zero, uh like like many of these things, the actual code changes like two lines just set that setting it did have to be deployed in every different implementation of cni, so so the base plug-ins had to do it. The uh I had to do it in weave net.

A

You know maybe calico had to do it all. The different implementations had to make that change to disable it and there's no legitimate reason. Why why linux would be accepting router advisories from a from an ordinary container? So that's one mitigation that we put up um the next one coming left. uh I, I would definitely recommend that you turn off. What's called cap net raw, this is a capability um which is like a linux permission to send raw packets. This is what allows an attacker to forge anything.

A

They like uh router advisory messages, the entire fake message, the entire tcp stack. None of that is possible. If you just don't allow people to to forge packets, um if you insist that they go through the operating system to open sockets and make requests so um so very much recommend that it is. It is on by default, and uh so you have to turn it off. um But I it's for me. That's a recommendation and the the last one I'll mention if you had some kind of authentication on your connection, I've put tls here.

A

It could be something else. um If, if that is the case, then the attacker has to work harder, because not only do they have to get the fake code and send the fake messages, but they have to um have your credentials to participate in an authenticated connection, whether it's tls or something else, so um uh so definitely think about those kind of mitigations not only for this vulnerability but for future vulnerabilities that we've not discovered yet um having described that one. Let's, let's look at another one, um this one's about demons listening on 127.0.0.1.

A

This is this: is the local host address or the loopback address, um so in this picture, we've got two hosts. um One of them is, is listening on this address and and most people. I would certainly have told you this before I knew about this vulnerability. Most people will believe that that address can only be accessed from the local host, so they think it's safe. I just listen on that local host address and nobody can come in for another machine.

A

um If somebody does try to do that. Well, they can an attacker on another machine can say uh you know. Please will you route packets to that address if they have control of of this machine machine b in my picture, um then those packets will go out and if they're reasonably close, if they're on a local network to the host they're trying to attack, I just made up an address here: 36 whatever.

A

um If that's close, the packets will arrive, uh but linux will will drop them. um They're called martians, uh martian packets. They appear from outer space, so.

A

That's the normal, the expected state of affairs. Most people will tell you, that's what happens. You cannot root to that address. Surprise.

A

There is uh ins in the cni software there's a thing called the port map plug-in um and same thing happens in in other bits of software. Cube proxy is an example. It changes a setting. It says I do want to root that local address um and it it does it for kind of intricate reasons to do with with how traffic is redirected inside uh for for the mapping of services, but anyway, the bottom line is, if you have that setting, if you have root local net equals one um that connection is possible.

A

Attackers can now talk to those demons that believed there could only be accessed by somebody already on the host.

A

So what to do about this? um Well, the code change that we we shipped in uh cubeproxy, for instance, and in again put out a release with this change in it was, was to add a rule of iptable's rule blocking those connections, um so that makes that safe again, um but once again I if those connections were authenticated, you know tls, is an example could be something else. uh Then it makes it that much harder for the attacker.

A

So there you go quick walk through of uh two different, very different vulnerabilities, but I hope it gives an insight into just how these things interrelate, how how a bunch of different layers can kind of stack up and come up with a behavior, that's unexpected um and, of course, that's what attackers do they? They do the unexpected, because, uh because they're trying to hack your system, so um that's the end of the prepared, slides uh I'll put that information slide up once again.

A

um If you uh are in too deep, the intro session link is up there. The video should be up shortly after the original streaming. um Our software is in two repos one for the spec and libraries, one for the plugins the base plugins that we support.

A

um We do have our own slack uh lovely, to see you there if you want to head over, um do not uh kubernetes has its own slack. So so do note that difference. um Cni is not part of kubernetes. It's. It's works across many different container runtimes. So so, if your question is really kubernetes specific, please head over to their uh their slack instead, okay, we're done with the prepared section.

A

I hope you all have questions and are have been typing them already, but if not, please start now and bruce, and I should be online to answer them. Thank you.

C

All right, uh brian here I hope you can hear me. We had a. We had a few technical issues. uh Hence I'm not on video, but I am here live, um so we had a few questions already. uh A number of them were kind of touched on. By that last thing, I said that the cni is an independent project of of kubernetes and a lot of uh obviously kubernetes being a very large dominant project.

C

A lot of the questions that people have relate to kubernetes, um but we're not uh really in a in a position to speak for kubernetes. um They they have their own sig network and uh uh yeah you're, really better off asking them um question just came in from hirotaka. I hope I said that right.

C

Are there any plans to add grpc interface to cni and observing that many cni implementations do run a server on each node? um So I think we touched on this in the intro session I mean the answer is broadly yes, um yes, we've talked about this, um so uh cni was was created uh uh originally the very first project to use it was the the rocket project rkt um and rocket had no demon it. Everything was demonless.

C

It's just exact program so that sort of followed through to the architecture of cni, where you just exec a program and um yeah you're, absolutely right. So if you've got a demon running then talking grpc to it is kind of more standard in in today's world. More expected, um it's also easier to do kind of global state things.

C

Cni works container by container interface by interface in its interface in its api, um so that would definitely be an advantage of moving to grpc um that you, you could have kind of system-wide questions like is the network up at all.

B

C

Yeah, we definitely talked about that you can find in the in the repos. Actually, let me see if I can uh move back to that slide. That has the links on it. Does that work. I hope that worked. um Yeah take a look around. You know just search for grpc in the issues on the the cni repo you'll probably find a little bit of discussion about that, and you know we love. We love help quite a small project and basically volunteer run.

C

So uh if you have great ideas about how to do grpc, uh please get in touch. uh You know you could do the coding and uh we'd love that um okay, let me try any new questions. um I don't see any new ones that have come in.

C

So let me turn to a couple that I answered um earlier on on the on the q, a typing um which were about like extra attributes, passing information kind of specifically from kubernetes, but but, as a more general point, there's there's kind of a fine distinction between what's interesting to you and what's useful to everyone, and we try and walk that line with the cni project.

C

We try and keep the specification really minimal, really down to the things that apply in in many many many cases, and but on flip side we don't want to stop anyone from doing anything. You know, that's that's kind of uh anathema. The the whole point of cni is to allow everyone to play on a level playing field. um All the all the container runtimes and all the networks can come together with this very simple core interface. That's the idea.

C

So you know if you come across a situation where you say gee cni is preventing me from doing this thing that I want to do. Then, then, by all means, get in touch. Raise an issue talk to us, um because the requests and the responses are json.

C

uh Fundamentally, you can add anything you like uh you know we do. We do not barf on unexpected fields, um so you can add, you know uh bruce, and his bit was talking about specific ways that we've tried to um style the way that people add fields. But fundamentally it's jason. You can say whatever you like, you know you can.

C

You can put a megabyte in there saying uh you know whatever you want to say in your json um and the only requirement really is that you get at least one runtime implementing it, and at least one plugin implementing that I would say if you find you know, lots of plugins implementing it or lots of runtimes, um then that that would be a time to come to the cni project and say you ought to standardize this. uh We actually have two levels of standardization. We have.

C

We have the spec, the formal spec that everyone has to conform to, and then we have a document called conventions which is exactly to get out of this compelling everyone to do something.

C

So we can add things to the conventions document very easily, but you know you would need uh some evidence that it is actually a convention that there's a you know more than you using it um in order for us to publish it to the world uh so yeah, you know we're we're, definitely interested in playing our part, but don't uh don't kind of confuse the cni project which goes across everything with one specific project, even kubernetes, big as it is, but we love kubernetes.

C

So I'm uh that I'm gonna wrap up I'm on slack um in the uh cncf slack. um You can ask there and uh or you can ask in the github issues- and I hope to see you around. Thank you very much and goodbye for now.