Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2019 (Barcelona), 23 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Treating Network Assets as Scheduleable Resources - Vinothkumar Siddharth, Amazon

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Treating Network Assets as Scheduleable Resources - Vinothkumar Siddharth, Amazon

Several cloud-providers now offer a managed Kubernetes solution. Customers launch their worker nodes within a Virtual Private Cloud (VPC) and in majority of these environments, Container Network Interface (CNI) is the preferred networking solution. While the default scheduler tracks CPU and Memory as resources and allocates pods to nodes based on the availability of the required resources, it does not natively track network resources such as IP addresses. This session will describe a centralized cluster networking solution built using custom resource controllers, admission webhooks and CNI plugins to enable network assets to be treated as cluster wide Kubernetes resources. The controller and webhook are re-usable platform agnostic components that can support Linux, Windows and ARM64 worker nodes. They also serve as the foundational blocks for building special purpose extensions.

https://sched.co/MPb3

A

Can you all hear me?

A

Okay, hi, my name is Siddharth and thank you all for coming to my talk is called treating network assets, a schedulable of resources, so I am an engineer on the Amazon eks team and in the past, I have to work for the Amazon ECS team, which is yet another container orchestration framework, and even before that, I used to work for a company called Citrix when my Alice, who work on the Xen hypervisor and the open source jennifer tool, strap called sappy and I have a bit of an agenda for today's talk.

A

I'll briefly introduce CNI and container scheduling before I move on to describe the challenges and the solution. We've come up to address these challenges.

A

Let's move on to describing CNI see. I stands for container networking interface, it's a highly overloaded term in today's world because it means different things to different people.

A

The CNI project itself has multiple provisions of the spec spec voice in 0.3 gives her three commands add delete and version add is used to set up the containers network, namespace, delete user, teardown and content network namespace and work here is used to return a list of supported versions of the CNI plug-in.

A

There's a new version of the spec we 0.4 it defines a new method called check that used to describe the containers connectivity, whether it works properly or not.

A

The CMA project itself also hosts a bunch of plugins. They go all the way from the simple loopback plugin to advanced plugins, like flannel and port map. We even see windows plugins now on the cni project, like wind bridge and when overlay.

A

The Siena project also has the library and provides a skeleton to build and develop your own schema plugins so to quickly recap: on CNI, the CNI project defines the spec, has a library and also defines base plugins to build your own cni plugins for development and testing purposes.

A

The Siena project essentially aims to provide connectivity and reach ability for containers. The CNA program also has capabilities called plug-in chaining, wherein you can make one plug-in call another plug-in to get set up. The network net should set up a network stack for the container.

A

The scanner project is very useful for developers because it exactly calls one method to set up the stack and one method rotate down the stack since the cni project decouples the network's development from the container runtime an orchestration frameworks, it's actually very useful for customers and developers.

A

Let's move on to see how cubelet and CNI work together, the cubelet is not exactly aware of the pods per se there ever or something called as sand boxes and the sand boxes and ports are loosely coupled and not tightly coupled.

A

The only guarantee we have today is that the cubelet guarantees at least one invocation of the CNI plugin per sandbox.

A

To enter that CNI and cube network together, we need a pass in these mandatory flags. You need to set the network plugin to be CNI, you need to point it to the CNI binary directory and the configuration directory and in cases wherein you have multiple CNI setup on the same node. The use lexicographic order to determine which plug-in to call to set up the stack and if you now typically see CNI deployments, they are typically daemon sets which have three components.

A

There was a CNI plug-in itself and then there's a CNI configuration and then there is a node level demon set. So the daemon is a long-running process on the node that the plugin invoke to set up the stack. So if you take the case of the Amazon, you see a solution, so we have the ec2 instance and then we have the cubelet, and then we have the plug-in itself. So when cubelet invokes the add command, the cni plug-in talk to the network, local control plane to get an IP and plumb, the routing and untrim.

A

The routing entries on the node.

A

Let's now move on to contain a scheduling to see how it works.

A

So this is the typical architecture of kubernetes, so the control frame consists of three major components: the API server, the scheduler and the controller manager, and they have a data store called HCD.

A

This is just a flow of how parts get scheduled in communities, so when users create a pod, they send a request to the API server. We send savestate onto HCD the scheduler watches for unassigned quads and then tries to find a node for this corresponding pod and it binds the pod to a node and the state is saved back onto a CD. The cubelet then picks the pod sets up the container net stack and then runs the pod for you.

A

The community scheduler essentially finds nodes for posture on its complex and feature-rich. It has a bunch of features you can set. Node affinity and anti affinity for definitey and 90 affinity also supports trains and toleration x', so the default scheduler typically takes CPU and memory into accounting for scheduling.

A

So as an example, if you see the pod spec, it has CPU and memory limits that you can set on them so with the Linux kernel. Both of these come from Signum subsystem. Typically, CPU is the compressible resource and it can be oversubscribed.

A

You can use TP of shares to set up your CPU limits. You can also use a combination of CPU quarters and CPU period to get fine-grained control of CPU and typically with memory, it's not compressible and running, and when you run out of memory, it's going to kill some process or a container. So how this thing happens is there's something called as an ohm score. That's computed and the more higher and the higher the score, the more likely you're supposed to are likely to be killed.

A

Kuban. It is also supports custom schedulers. You can have your own scheduler for special purposes and special needs, and the way to use them is to set the scheduler name in your spec.

A

Let's move on now because some of the challenges we applying to address.

A

Typically, customers use VP C's to launch the workloads, and the me pcs can be simple or complex me. Pcs can have public subnets, private, subnets or multiple siders.

A

One of the key challenges is: how do you account for networking assets in the scheduling process in communities follows an IP per pod model and with this at scale, how do you ensure that it happens.

A

The other big challenge is that with typical CNI deployments, they use n3 networking and we also have a long-running node level daemon, which is used to set up the routing entries on the host. Is there a way to get rid of them and.

A

Typically, the I M permissions on the node are really exhaustive because you need to like create and delete interfaces assign and D assign IP addresses on the interfaces. So is there a way to reduce the surface area? Is the next big challenge so.

A

Let's move on to the solution: the solution we built uses external resources, it uses custom resource controllers, it also use the admission web hooks and then we now have CNI plugin to factor all of this into account.

A

So the first component is called the BBC resource controller. This aims to provision and manage BBC resources for the cluster. It's built using go so its platform, agnostic and architecture agnostic. It can be built for any platform and for any architecture.

A

The controller also has a flexible resource model and has a simple provider interface. The provider interface makes it very easy to add new resources to the controller.

A

Let's look at the provider interface, the provider interface basically has five methods. The first is called get resource name. That method just defines the name of the resource, and then you can basically tweak your warm pool settings for the resource of planning to create and delete. So the method called inert resourceful is for the initial setup of the resource pool for this particular resource.

A

Create resource is used to create a give a number of quantity of a given resource on a given, node and delete, let's just use a free app or to free up the resource. On the given node. As of today, we have two implementations of the provider interface. We have the eni provider and the IP address provider.

A

So how this thing actually works under the hood, so the controller watches for node objects and then advertises the nodes capabilities like for example. In this case, we get no objects and when we handle them, we get the notes, capabilities in terms of networking capabilities like how many a nice there's no supports and how many IP for interfaces the Kansas node support and with an advertise them at the extranet resources.

A

For example, in this case, we see that this node has one eni and that eni supports protein IPS.

A

The controller also watches for pod objects.

A

So what the controller now does is that it tries to find an IP for the port to consume and annotates the ports back with the IP to be used.

A

We also have a V PC admission webhook. It is a mutating admission web hook that basically injects extended resource requirements for the pod object.

A

For example, in this case, we see the limits and requests for the pod objects.

A

The C&I plug-in itself is fairly simple. It's a simple executable that reads: support annotations: to set up the pod Network to set up the pod Network stack. The plugin also provides connectivity and reach ability.

A

Let's go for a quick demo.

A

Can you all see the screen.

A

Is this better.

A

This is good all.

B

A

Okay, it's be hard for me to see okay.

A

Okay, give me a second.

A

Just bear with me.

B

A

Oh in my cluster I have a 1.11 cluster and I have a bunch of nodes to make the demo more interesting.

A

We have a bunch of Linux nodes and Windows nodes.

A

They still hug. Is it hard to read on the back.

A

Okay, so onto this cluster I'm deployed by BBC admission, webhook and the controller so.

A

So we have the controller and the web book running for this cluster. Now, let's do some work loops onto this cluster.

C

A

Can you read this.

A

So we are deploying a simple server core web server on to this cluster. This is a Windows pod.

D

A

We now see that all of these quads are running and if you describe any of them in detail.

A

You see that the admission web book has injected a limit of the private IP and a request for the same. For the same, you can also see the controller injecting annotations with the pod IP address, and this should match the pods actual IP just set by the cni plugin for the squad.

A

Since this cluster has both linux nodes and windows, note you can also schedule linux notes on them.

A

This is a simple, simple pod, : ax. So if you apply these.

A

All of these should be running and.

A

We can also scale the deployments, for example, let's scale this to ten odds.

A

You see a lot of them are running and if you know describe one of these quads.

A

You now see that this node has advertised an opaque resource, called private ipv4 address and three of them are being consumed by the pod they have on the cluster.

A

E

E

E

Is it COO net or the web hood? Considering the amount of IP addresses available in your cluster, like, for example, in our slash, 21 cluster, you will submit, you will be able to have a limited amount of IP addresses so.

A

This IP address limit come from the node itself, like in Amazon. Each instance supports a fist, say a set number of en eyes and eg and I has a set number of IP addresses, for instance type. So the iPad, as you see, is based on the instance type of your worker node. So.

F

Also unknown known as unknown eni situations, it's very common to have like a limit of 255 addresses per per node, so I.

E

Just wanted to point out that, for example, when running any key, as you have to be careful with this subnet that you decide to choose all the way, then into a limit of IP addresses. As you can allocate interview PC and then you are practically import, never work because I won't have a networking. So.

A

Your BBC configuration is different from the limits you see here, so your subnet can be bigger or smaller. If you have really really tiny subnet, the IP limits come from the instance class and not from the subnet itself.

F

A

I'm sorry I just.

G

If we add a syrup here has come from the.

A

Ip address comes the IP address. The private IP address from the VP see.

G

From the VP see yes, okay, so.

A

Each part in eks and gets a private IP from the BBC and when.

G

I changed the network, so it's doable on the fly. I'm. Sorry when I want to change the the networks, because maybe I'm I'm running out of IP addresses I switch into a different network scope. Is this configure configurable or is it static? What configurable.

A

G

I change in the subnets or subnet sizes, or something like that so.

A

The limit is, for instance, and not for subnet.

D

H

Other obstacles in this design to supporting public IP addresses of really globally routable IP addresses for pods I, mainly think about ipv6 for the future, but globally, reachable ipv4 as well. So.

A

It's gonna be easy to scale this model to support IP addresses you can just have a new provider that gives you public IP address instead of EPC private IP addresses and that's possible with this model.

I

If I understand your design correctly and I feel so, I can see that the the local ipam is not really responsible for assigning the IP address. So so is it possible that, with this with this design, we are completely able to ignore the IP address range assigned to a node and just and just rely on the controller and the CNI for IPAM.

A

Essentially, what happens is now we know near I, Pam D on the node, because the IP addresses comes from the controller itself, so you can do away with I Pam D. With this model.

I

C

I

Only thing I wanted to confirm, so that's really cool, it's really cool. Okay! All right! Thank you.

F

Any more I missed it just.

J

C

J

Clear on the run, the limits and the requests, obviously the same things or there's a one-to-one relationship there, but does that impact their scheduling on to that? So, if you've got a node, that's got a ton load of small pods on there and you run out of your your eni and your IP limits for that node is I mean the new pods won't be scheduled on there.

A

So if the nord can't support new IP addresses, then you have pods will be stuck in pending, but they will know now know that you cannot run the ports because they don't have an IP address available for the for any of your class. Any of the nodes in the cluster.

J

So just so clear they will schedule. They'll just feel they'll sit in a pending state until some other external actions. You need to have like an alert so to do that.

A

So once you add more north via cluster, and then you have enough IP capacity, the port will then continue to run on on the available nodes that you have on the cluster I. Think.

F

The answer is: if they're, pending they're, not shed jeweled I, think that's the question. You're asking they've not been allocated to a node if they're, pending.

J

So I've got two nodes: one that's got two massive pods on it. One that's got tons of small pods on it and the node with a lot of small pods exhausts. All of the IP addresses that are available for that node type and I go to try and put on another another pod. Will it shed you'll it to the one? That's got capacity to be able to do it. You know it's ignoring the CPU and memory capacity stuff. So.

A

With this model, we advertise the new opaque resource called IP address for the whole cluster. So when the scheduler tries to find a node for your pod, they'll try to find a node which has enough private IP capacity and then put the pods on that particular node.

D

Is there a concept of a persistent IP request, for example, if you have a elastic IP bound to a private IP, can you request that that that you always be assigned to that node on that of IP I.

A

Don't think that's possible out of the box, because the kind of design works with private IP addresses from the BBC.

D

Ok, so you don't consider IP requests when scheduling just the amount of IP address that APOD requests. Yes, ok. Thank you.

F

Anymore, we're going over there. Are you just retching, okay, as you're gonna make me run to the back.

K

So what I need to change if I want to start using a secondary IP siders secondary cider IPS for the the pods.

A

Could you just repeat the question so.

K

If I run out of IPS in my current cider and add a secondary cider can I start using them in in eks and work. What I need to change in the cluster.

A

So the current controller does not take into account second two ciders, but if you have a node, that's in the secondary side arrange and then the IP addresses would then be able to come from that range. For you.

K

C

Disappoint about the question that the they hit it is it when you want to add a secondary CIDR to to the V PC so to the e case. Cluster also, because you are running out of IP, addresses the V PC high V changes. So I was running into that. That issue with the KS, so I was creating the cluster with terraform and since the IV changes of the V PC, you need to recreate the cluster because you need to assign the the new I you'd have to assign the new IV to the eks cluster.

C

So I had to reassign to recreate sorry biggest cluster. Is that possible through the Amazon console to just configure the V PC? So it will be reconfigured so I mean? Is it necessary ready to recreate the case cluster when, when you are the secondary CIDR to the V PC, so the ID changes.

F

Might be a little detail, maybe you should just go to the Amazon booth and asked that question a little bit down in the weeds, I. Think for for the big audience the.

A

Current controller does not take into account ii, decided ranges yet, but will add support for that. So on.

F

B

Had a question in principle with the your current code be adaptable to some other cloud platforms. Are some native I mean VM based or clusters, or is it sort of Amazon specific at the moment, so.

A

I think the controller is a very general purpose and extensible can just add a new provider or whatever resource you want and then make it register with the V PC controller, and then things should work for you. Thank.

B

I

We can I find it. What do you mean? We can I find the code on github.

A

So as part of the windows public preview for Windows nodes, the controller on the web book are already available on github like at least the way to set them up. I've been there on the instruction set like on the readme file. So let me bring that up.

A

So if you look at this is a doc 4ek support for Windows nodes. So this walks you through setting up the V PC controller and the admission web book and how to set them up in the cluster. For you come.

I

F

It possible to get that URL a bit bigger.

A

F

Slash AWS slash containers roadmap.

F

Any more questions: okay! Well, thank you very much sit off all right.