Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2018 (Copenhagen), 14 Nov 2015

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: The “Silk” Road: Building a CNI Plugin from Scratch - Usha Ramachandran & Angela Chin

Description

Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18)

The “Silk” Road: Building a CNI Plugin from Scratch - Usha Ramachandran & Angela Chin, Pivotal (Any Skill Level)

CNI promises container runtime systems the ability to swap in different third party networking plugins. With many 3rd party plugins available, it can be difficult to determine which, if any, are the best match for your system. Angela and Usha will highlight their journey through the CNI ecosystem from adding support for CNI in Cloud Foundry to building “Silk” - their very own CNI plugin. Attendees will gain insight into the process of both deciding to and building a CNI plugin and considerations that must be made about how to integrate the plugin with existing platform concerns about networking and security.

About Usha
Usha is a Staff Product Manager at Pivotal and the Product Lead for CF Networking. She is responsible for prioritizing and delivering core networking capabilities for Cloud Foundry. Usha has over 15 years of networking experience, having worked at Cisco, Alcatel and Ericsson. She has worn several hats throughout her career including software engineer, technical marketing engineer and product manager. A seasoned conference speaker, she has spoken at Cisco Live!, VMworld, Open Networking Summit and Cloud Foundry Summit.

About Angela
Angela is a software engineer at Pivotal, currently working on Pivotal Container Service (PKS) team. Previously, she was the engineering lead of the Cloud Foundry Networking team, which develops the container networking and policy enforcement systems inside Cloud Foundry Application Runtime. Angela has previously given talks at CF Summit (North America and Europe), Open Source Summit, and Spring One Platform.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Okay, everyone um so we're gonna get started today. Thanks for attending the session, my name is Angela Chen I'm, a software engineer at pivotal and with me, is a Ramachandran who's, a product manager at pivotal a little bit of background. Why are we wearing these hats? Why are we giving this talk so we're both have worked on the networking team for cloud foundry, which is a platform as a service on the networking team.

A

We've really been focused on, of course, container connectivity and communication between containers running in cloud foundry, and so as part of the journey of working on the networking team, we decided to build our own C&I plugin. Hence the Silk Road, building a CNI plugin from scratch.

A

So we'll just do a brief overview of what we're going to talk about here today. The first thing is just to make sure that we level set on what is CNI. It's a very high-level overview, just make sure we're all on the same page, before delving into what is Cloud Foundry again to understand why we built our own C&I plugin.

A

Look at C&I +, Cloud Foundry before talking about the main motivations behind, why we decided to build silk before talking about the design decisions that we made at a high level in order to meet the requirements that we were looking to, enable we'll look at what's next for the future of our networking plugin before finishing off with some key takeaways.

A

So we'll start with what is C and I. So CNI stands for the container networking interface right and it's an industry standard API for container runtimes to call out to networking plugins, and this is sort of just the bare definition. So why exactly do we have CNI right like? Why do we create it? Why is it part of the since you have like? Why do people care so much?

A

And it's really because you're going to have different use cases you're going to have need different capabilities for your networking stack, depending on what hardware you're running on what your use cases are, and so, instead of having your container of runtime also deal with setting up the networking stack for your container, you want to be able to have it pluggable. You want to be able to choose your own solution, maybe add additional features and so see and I really allows for that extensibility.

A

So, for the purpose of this talk we'll be focusing on CNI plugins. So what actually implements your networking interface? What sets up the networking stack for your container, so the container networking interface right is an API. It's a specification and what it deals with for CNI plugins is two main things: one. It deals with creating the network interface for containers so again analogous to in the C&I specification, C and I, add and also deals with removing these resources.

A

When you delete a container so making sure you clean up after yourself, it's also important to note that delete is idempotent. So this means that you could get into a partially torn down state. You can run delete many times. Maybe you actually did clean up should ever fail and at its core C&I plug-ins deal with network connectivity. So we're really looking at making sure that our containers are able to be reached that we can send receive packets. You can do all kinds of fancy things with cni plugins.

A

You can do things like traffic shaping, you can do things like host port mappings, but at its core, CNI is really about the network connectivity aspect of it, so CNI plugins normally come in the form of binaries right that you run and they are invoked during a container, create and delete. Obviously, because you're doing the addition and deletion of your networking stack and arguments just at a high level very quickly for C and IR is passed in through a mixture of environment, variables and standard in, in the form of a JSON format.

A

To say how exactly you want to set up your networking interface and again. This is all documented in the cni spec.

A

So we talked about how we have the use case for CNI, because we want to bring many different solutions. So here's just a list of many different CNI plugins that are out there, as you can see it's extensive, but for the purpose of today's talk, we're really going to focus in on silk, which is if this ever moves. No, can you hit next I think this died.

A

It's a CNI plugin, that's designed for Cloud, Foundry and so to understand our journey to understand why we decide to build silk. We need to look a cloud foundry first, because we designed it for this use case and it's all handed over to Isha.

B

Thank you. Alright, it's like this click or maybe a little temperamental that will work with it. So I'm gonna spend the next few minutes just introducing cloud foundry, there's a lot about Clare. Finally, in fact, there's a whole conference dedicated to it so I'm gonna try my best to just give you a quick overview of what what the platform is and what are some of the networking related aspects of it that affect what we did with our CNI plugin building, just a quick show of hands. How many of you here are familiar with cloud?

B

Foundry, ok, looks like maybe about half the room, alright, so just to get started cloud foundry. Is that there's a simple haiku that basically describes some of the principles of cloud foundry and it's coined by Anzhi Fukui from payroll, and it basically says that here is my source code run it on the cloud for me: I do not care how and this principle kind of forms the basis for a lot of the activities or the or the features a cloud foundry supports. So at its core cloud, foundry application. Runtime is your building block that space?

B

That's gonna run and manage your applications, so the idea is that, as an app developer, you can concentrate on writing code and everything around the life cycle of the application, such as patching. It scaling it routing to it security policies. All of that is taken care of by the platform, and it runs on a infrastructure layer called Bosch that manages your infrastructure and it's responsible for the lifecycle, management of the components that are gonna host. All your applications, as well as the control plane for it.

B

So both of these platforms put some specific requirements on building a C&I plugin for this platform.

B

So just to give you a quick idea of what happens so that maybe you can draw a parallel between, say, a kubernetes life cycle and a Cloud Foundry like application life cycle is that as an app developer will push code using a very simple command called CF push.

B

It includes usually an app manifest that has properties that describe the application, and the next step is if it'll this will go to our Cloud Foundry API, at which point it'll go to a container Orchestrator, which will then schedule it to one of the hosts that are going to host the containers. We call these host cells. There is a container runtime that is responsible for the lifecycle of the container and on the host while it exists. Typically in Cloud Foundry, the container runtime was responsible for the networking for all the containers.

B

It's also responsible for setting up file systems etc. With C&I, we actually extract out some of the the networking functionalities. The last part for an application is to register a route so that it's accessible from clients that are trying to connect to it from outside, and the component that does. That is a core router. So, in terms of networking, there are a few things that applications on Cloud Foundry have to deal with. One is of course, being available from the outside to connect in and then connecting from an application to a service.

B

That's sitting outside the platform. The typical way for applications on the platform to communicate outside through the use of NAT and we have egress security, firewall rules that can control that traffic through a feature called application security groups. So that's a very, very high level view of the lifecycle of an app within Cloud Foundry.

B

So let's talk a little bit about where Cloud, Foundry and CNI intersect, and to do that, we have to take a couple of steps back to just understand how legacy Cloud Foundry networking worked.

B

So, as I mentioned before, the the weight application traffic goes out on on platform V. Today's is that everything is knotted out, and so that meant that the app identity is basically lost. If you wanted to do direct container to container communication, you had no way of doing that in the traditional stack, and this made it difficult because you know it that there are some inefficiencies of all these hair pinning through and the ingress gateway and also in terms of applying policy.

B

It's difficult to do that if you don't actually have a PI identity, so our ideal state was basically to have a direct connection between containers. So that you had containers that were identifiable, you could apply policy between them and you could have more efficient traffic parts within Cloud Foundry, so I'm a p.m. I love. You know coming up with requirements so in terms of requirements from the networking stack.

B

The first thing was, of course, that the legacy features of Cloud Foundry continued to work, so we wanted to provide backwards compatibility, and so what that meant was that the existing NAT based work flows would continue to work. Application security groups continue to work. The second part is connectivity, as I mentioned before, we wanted to provide every container with its own IP address, and we also wanted to be I, as agnostic is the way so I found he works is that you can deploy it on any any cloud.

B

So we really wanted this container addressing to be. You know to work across any any any different cloud and finally, we wanted it to be extensible, mostly because there are other solutions.

B

You know Angela showed a whole bunch of C&I plugins that are available as well as Sdn solutions that you might want to deploy, and so extensibility was another big requirement in terms of connectivity.

B

Okay and the last part, oops was policy, so policy is basically allowing specific applications to talk to each other having it dynamic, as well as being self-service.

B

Since we're talking about CNI we're gonna focus, this talk on the connectivity aspect of our requirements and the main the main things we wanted for connectivity was that it was I as agnostic that there was a unique IP per container and that it was extensible.

B

And so what this turned out to in terms of the solution was actually to use a VX land based overlay, so that it would work in any environment and to allocate a subnet per cell from which every single container could get an IP address and then finally, to leverage CNI so that it could be extensible and you can replace the batteries included, C&I plug-in with anything that you choose.

B

So maybe you are thinking that this sounds a lot like funnel and you're right. So we started off with actually you implementing funnel as a solution and along the way we found out that we had some motivations for building our own CNI plugin. So I'm going to talk a little bit about what led us down the road to sock.

B

So I'm gonna walk through some workflows like Dennis slowly and then in your mind, you can imagine that this is happening across the board for like thousands of application instances and in a system where things are changing all the time. So the first thing is that application containers can be added or deleted at any time. So you can, you can delete containers, add new containers, scale-up containers. The second thing is that your Tico cells that are running the containers themselves can be destroyed and recreated, and this is done by Bosch without any operator involvement.

B

So when that happens, all your containers running on a cell are rescheduled to another cell. They evacuated and your once once you clear out all the containers the cell is deleted, and similarly you can. Okay, all right. The cell is gone, the cell came back so similarly, once the cell comes back, you're gonna have to reschedule containers back on it and you're again. There is some just in terms of the life cycle. You know, when can you start scheduling, containers to a cell?

B

What, if I schedule a container when the networking is not yet ready to set things up, so these were some of the things that we ran into and had to find out solutions for. The other thing is: how do you keep your networking running when you have network partitions, if you're, storing state somewhere outside? How do you deal with actually losing all your networking state right? How do you? How do you recover the system?

B

So a lot of these requirements ended up around you know just making the system available and reliable across many different lifecycle events, so that was the primary motivation for trying to build our own plugin, which was to make sure that networking was stable across many different by cycle events, a secondary consideration was also extensibility, so, starting from an app to app policy, we wanted to extend security to also include things like services or the core router, and in order to do that, we'd like this, the RC and I plugin to start supporting other types of entities, not just at containers.

B

So now that you know the, why behind why we wanted to build silk I'm gonna hand it back to Angela to walk you through? How exactly did it.

A

Great thanks Usha, so we'll just start again with a high-level overview. So silk again is the C&I plugin that we built at its core allocates a unique IP per container and it does its networking using a VX land, overlay, Network and so silk is composed of three main components that we can see here in red. We have the silk controller, we have the silk daemon and we have the silk C&I plugin.

A

So the silk controller is the control plate component, not very great, at naming that deals with being given an overlay network allocating and distributing that overlay, Network splitting it up into subnets and giving the subnets to each host that wants it. The silk daemon is a local process running on each host machine. It deals with first acquiring the subnet from the silk controller, renewing the lease it has on that subnet from the silk controller, and also deals with sending up all the routing for the state of the world.

A

And lastly, we have the silk C&I plugin, which is the binary. That's called during the container creation and deletion which sets up your network interface and also allocates the unique IP per container. So when we decided to build silk, we had several design considerations in mind, first and foremost, reliability and stability, so just to make sure that we're all.

A

On the same page again, we made several decisions around the concept in silk known as Elise Elise is basically a subnet given to a host VM, so the silk daemon will ask for a lease the leases, the subnet from which it can allocate container IPS from, and so the first design consideration we decided to do around leases was that alesis, given only if there are no containers on the sell and the reason we did.

A

This is because you can imagine a use case where you have containers on the sell that have some container IP and silk daemon needs to acquire a brand new lease for some reason or another. Maybe there was a network partition and it comes back up and the lease have had before the subnet had before I can no longer acquire so I asked the SIL controller for a new subnet lease.

A

Well, then, you're going to have a mismatch where the host thinks that it can allocate subnets from a specific or allocate IP use from a specific subnet, but there are containers still running on the cell that don't fall within this range anymore and then you're going to have a loss of connectivity for these containers, and so the first thing that we cited was that you can't acquire a new lease. Unless there are no containers running, you can renew leases, but you can't acquire a new one.

A

The second design decision we made was that the network ad would fail if the daemon didn't have a lease which seems pretty straightforward right. If you don't have a lease, if you don't have a subnet, what are you allocating container IPS from in the first place? And lastly, the final design decision we made was that the lease should be kept for as long as possible, and this is truly to prevent against the case of long-lived networked partitions, because let's consider that a host VM is partitioned from the control plane for a long period of time.

A

The best case scenario would be when the partition is fixed, that the host the silk daemon running on the host would be able to renew the same lease because it has containers running on it. So it can't get a brand-new lease and the containers would be connected again. So we wanted to make sure that we are protecting leases. We have an idea of lease staleness where a lease can be stale, but we won't allocate it out to another host until we've run out of leases that I've never been allocated before.

A

So how does this all fit together? Let's consider the case of a sudden that assignment we have a new host coming up for the first time and we hit the silk Damon running. The silk Damon will reach out to the silk controller, to ask for a subnet lease, because it has no containers running on it.

A

It happily is given a subnet range from which to allocate ip's from so that when a developer decides that they want to create containers, they're able to have their container runtime initialize the process of creating the container before calling out using a CNN client library over the CNI API to the silk C&I plug in the silk scene, I plugin will create the network interface and we'll ask the silk Damon what subnet range it can allocate a container IP from choose a container IP I think in the container creation can be completed and, of course, this can happen from multiple containers on a given host machine.

A

Now, let's consider the case where you decide to restart your host, maybe you're upgrading, and so in this scenario. The first thing that happens when you decide to do a restart, is you need to evacuate all the containers off schedule, the workloads elsewhere? Then, when it comes back up, the silk Damion will once again need to acquire a new lease, because this host machine has been around before it's not brand new. It has an idea of what lease it had in the past and so will initially ask for that same lease.

A

If that leased wasn't available, it could acquire a different lease because there are no containers running on the cell, but in this case let's say that it's able to acquire the same lease happily does so and again is able to successfully create containers on the given host.

A

So those are some of the design considerations that we made around reliability and stability. Let's also talk about the actual architecture of the network. That cell creates I guess after this I totally forgot. The final thing that we did in terms of reliability and stability was to add the ability to call a teardown from the silk binary and the reason we decided to add ability for a teardown again goes back to fact that we knew how we were going to orchestrate our control plane, how we were going to orchestrate our container creation.

A

We knew we had a certain lifecycle management process, and so we are able to add this ability to call teardown so that whenever a new cell or a new host was restarted, the first thing that we would do is call this teardown to make sure if there is any left over networking state that would be wiped away clean.

A

So these are design considerations where, if you're designing for a specific use case, you can really take advantage of knowledge of how your runtime system works and knowledge of how you orchestrate it to make sure that you're adding some of these benefits and with that we'll get to the network setup. So, as assha mentioned before, one of our main goals was that our network set of had to be I as agnostic, and so we decided to use an overlay network with VX LAN. How does this look?

A

Let's consider the use case of having two applications on two different hosts and for a sake of switching things up, let's say app. B actually wants to talk to FA this time, instead of the other way around. In this case, we see that we initially send out this container to container packet with the source and destination IP s being the source and destination IP s of the host or of the source and destination container eyepiece.

A

In this case, we actually then have this packet virtually routed to the silicon VX LAN, where it's encapsulated in an utterly packet, with the source and destination being the source and destination IP s of the host machines. It will then be correctly routed through a switch to the destination host where it's D encapsulated and then correctly virtually routed to the destination and forwarded along note here that this is a distinction from flannel which deals with a bridge. Here we don't have a bridge.

A

We deal with everything in a sort of curve through virtual routing and so from the containers perspective. It's a single flat layer, 3 network, and this protects against things like broadcast storms and our spoofing.

A

So that was some of the considerations. There were many more that we've made far too many to go into today when building silk, but you might be wondering, did you just build a C and I plug in and leave it alone? What's what's next right? What are you actively working on so I think I can't walk too far away from this.

A

What's next, so the first thing that's next is C&I chaining, which is really hot, really cool.

A

It's the ability to chain your plugins so before in the previous day of the world, with with our setup, which is still the current state of the world, I guess we would have the Guardian external networker, which is our CNI client library, call out over the CNI API to the CF wrapper C&I plugin that we maintain- and this deals with the legacy state of cloud foundry that wooshin mentioned previously, and then the CF wrapper C and I plugin then had to delegate out to silk.

A

So we were nesting plugins in this scenario, but what C and I chaining you now have the ability so that your C and I client library can call out against the CF wrapper, C and I plug in and then two silk C and I plug in, but have that result, return back and use that result, that's returned to call out to some other C and I plug in we we're, even in visioning a world, although this definitely is not set in stone where we no longer have to do the nesting, so we can call it to the CF wrapper, C and I plug in have the result, returned and then chain out the silks and I plug in, or vice versa.

A

So chaining has been a really popular talking topic. It's really great for additional features that aren't part of the core network connectivity aspect. So maybe you have your first plugin that sets up the networking interface and you want to do some additional feature to it. One such use case that we actually had in silk was the ability to do bandwidth limiting, and so we actually extracted out that code and construed and contributed it back upstream to the cni community in the form of a bandwidth limiting plugin.

A

So this is now accessible for everyone to use it's in the cni plugins repo and it's actually being currently merged into kubernetes as well for use, and so we're looking to also extract that code out from the silk scene, I plug in and do plug-in chaining with the bandwidth C&I plugin that we've recently added.

A

The second thing we're looking at for the future of silk is, if I can go, is having it used by other container runtimes. So at its core, you know we said: silk is a CI, a--, plugin and designed for the cloud foundry use case, and this is true, but we think that it could be useful for other container runtimes as well. We've done work in the past to see that it runs as a scene I plug in for kubernetes, but there's still some work to be done before it would be something we consider production ready.

A

So this is on our radar as well. These are just a couple of things that we're looking at for the future of silk, but with that I'll, let you should take it off with key takeaways.

B

Thanks Angela, so in terms of the key takeaways, what we covered here today was basically just an overview of CNI and how we used it with Cloud Foundry. What we found using Siena is that it's a really wonderful and simple interface to provide networking capabilities and extensibility for your platform. We certainly don't recommend that everyone go build.

B

Your own CNI plugin, look at the CNI plugin repo there's like there are a lot of options available, but there might be cases where you would have to build your own C&I plug-in to get some capability that you need for a specific environment such as cloud such as cloud foundry. We also covered what silk does and some of the unique characteristics of it in terms of better lifecycle management around when containers come and go and your host VMs come and go.

B

And finally, we spoke about the bandwidth limiting plugin and C and I plug-in chaining new capabilities in C and I that where they were trying that we're starting to leverage and will continue to contribute back to the community, so definitely CNI is really awesome. Try it if it's actually not that difficult. I was in a in a talk earlier today on C and I, on, like creating small plugins to do additional, capable additional things than setting up network connectivity. So with that, we thank you all for attending. Are there any questions.

B