Cloud Native Computing Foundation Kubernetes Community Days Bangalore 2022, 11 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kube Proxy Next Gen

Description

The Kubernetes Service Proxy is a critical extension point for businesses wanting to expand the scope of cloud native applications to more sophisticated networking models, larger scale clusters and for designing better services mesh technologies. Some might even say it is "the thing" that differentiates Kubernetes networking from other container platforms. For several years, extending it has required copying and pasting code from Kubernetes, or rebuilding its extremely sophisticated caching logic piecemeal.

Kube Proxy Next Generation (KPNG) is a Kubernetes SIG Network project which decouples the dataplane (backend), from the Kubernetes specific logic of the Service Proxy, allowing for innovation in the way networks are built on Kubernetes without the need for adding more technical debt to Kubernetes core. The KPNG project supports multiple operating systems and allows users to easily plugin high performance, in-memory service proxying technologies that are entirely powered by a GRPC integration layer.

This reduces load on the APIServer and creates a service extension model for Kubernetes. KPNG project is rapidly growing and the community is engaging and passionate about its overall mission. In this session, we will showcase how KPNG is structured to tackle the pain points around current implementation of Kubernetes Service Proxy along with a demo of KPNG in different backends like nft, iptables, ipvs and how to reason about kube-proxy’s logic in any mode, using a generic model. KPNG being a nascent project, we will also talk about what lies ahead and give a sneak peek of the project roadmap.

A

Hello, everyone welcome to our talk, cue proxy next gen. This is anusha. I am a senior software engineer at vmware and I am the maintainer of cluster api provider for bring your own host. I started working on kpng as a site project and as a means to decode kubernetes networking.

B

Hi everyone, I'm rajas and I work at vmware. I got involved in cupping in mid 2021 and I've been hooked to it ever since. In this talk, we're going to show you cube proxy next gen, also known as cupping.

B

This sub project comes under sig network in the kubernetes ecosystem, we'll get into the details of cupping in a bit, but I'll give you a peek into it right away. Cupping tries to redesign cube proxy by decoupling, the kubernetes bits and the networking bits. Cupping interacts with the cube api server and stores the global state of the cluster in memory and then provides a local diff state to the back end, implementing the routing rules of a service.

B

These backends can be nft ip tables, ipvs windows, etc. All right, that's a lot of spoilers. Now, let's zoom out and see how cupping fits in the kubernetes puzzle. The first piece in this puzzle is q, proxy and anusha will walk us through it.

A

So what is kyproxy as you'll probably know? Q proxy is one of the two kubernetes components that runs on every node, the other one being the cubelet. It manages service abstraction within kubernetes. It is deployed as a daemon set on every node and can be configured using a config map. You can run q proxy in several modes. Iptables, ipvs and user space user space mode is now considered as legacy. Iptables is the default mode and ipvs is the most recent mode.

A

Now, let's take a closer look at the diagram whenever a service is created or deleted, there are changes to the endpoints resources. These are persisted in the api server and the information is synced to queue proxy. Whenever a client sends a request to the service, say an nginx service exposed as cluster ip note that this is a virtual ip.

A

It is useful because this value does not change and, unlike pod ips, which are ephemeral, the client has to store information of cluster ip alone and not worry about the actual backend ips, where the services are running.

A

So now, when a client sends in a request to the nginx service, depending upon the ip tables rules created by cube proxy, the request is routed to the appropriate backend port.

A

Now, let's take a look at where the q proxy code base lives. So this is the kk repo and under package proxy you can see uh there are different modes like ib tables, ipvs user space, main user space. So this is the entirety of q proxy code base.

A

A

Time for some demo and hopefully not burn anything down to the ground for this demo we'll be using a kind cluster. I've already pre-created that let's look at the config file, so we have a kind cluster with one control, pane, node and two worker nodes.

A

So all the notes are ready. Let's also look at the pots.

A

So here are the q proxy parts. We have three q proxy parts running because we have a total of three control plane: three nodes, one control, plane and two worker nodes, and we know that q proxy runs on every mode. Let's take a look at the logs on one of them.

A

So first it retrieves a node ip and then it uses ip tables proxier. We just learned that q proxy runs an ip tables mode by default. It is also possible to run q proxy using ipvs or user space modes.

A

It then sets contract max values, some timeout values, and it also starts the endpoint slice. Config controller wait waits for caches to sync, so a single q proxy process does both network related things and also kubernetes api related things.

A

Now, let's look at the notes again, so the notes are ready and they've been assigned an internal ip address.

A

I've also created an nginx deployment for this demo.

A

So there are two replicas of the nginx deployment running and I've exposed the nginx deployment via a cluster ip service.

A

Let's also take a look at the end points, so we have the nginx deployment exposed via cluster ip 1096 231 184 on port 80, and these are the actual backend pod ips, where the nginx service is running.

A

Now, let's exec into one of the worker notes, let's execute a kind worker and look at the ip table rules.

A

We use the command ip tables, save.

A

Now that's a lot of ipad ip table rules, so you can spot something like default engine x. This is what we should. This is what we want, but let's grab for it, so that it's easier to read through.

A

So yes, this is much better. So, let's start over here, because this is where we see the destination is 1096 231 184, which is the cluster id. We go back to this view.

A

We know that the cluster ip is 1096 231, 184 and on destination port 80. It says jump to this particular service, so this is a service ending in l6n, and we have that over here. So there are two entries for this: let's read through so the first line here states the default nginx with a random probability of 0.5 jump to this service endpoint. So 50 of the requests end up going to this particular service, endpoint and 50 percent- goes to the second line over here and jumps to this particular service endpoint.

A

Let's look at one of these. Let's look at the service endpoint ending into th, which is over here, so this line basically adds the masquerade bit on the packet. But let's look at this line. This is what is of interest to us, so it says the service endpoint that is default. Engine x has to be routed, that is, to destination 10, to 44 2.3 and on port 8000.

A

So this is nothing but one of the endpoints listed here, which is also the pod ip. Let's quickly list that.

A

So we have two parts with 10 to 44, 2.3 and 1.3, and this service endpoint is routing to 2.3, whereas this service endpoint is routing to 1.3. So this is how q proxy does load balancing as well all right. That's it for the q proxy demo now coming back.

A

Yep, that's a lot of ip tables so now, while q proxy has been doing its job since the beginning of time, it does come with its fair share of limitations. Let's take a look at those.

B

Q proxy does the networking and the kubernetes api server watch bits in a single process. This causes a significant load on the api server as the node's increase in the cluster. This blog post from open ai highlights these pain points of q proxy. When the cluster is scaled up, it provides data on how the bandwidth required for responses from api server grows by order of n squared extended q proxy is complicated because of its in entry implementation.

B

Cni providers like entry have their own proxy, which replaces q proxy, but the implementation requires bits and pieces from cube proxy code.

B

In this snapshot we can see how andrea has to copy parts of q proxy code, there's a similar story with calico as well, where the proxy is based on cube proxy, which requires syncing with the q proxy code.

B

Very few people in the community understand current cube proxy code and have context about the history of it. This tweet is one such example where we wanted to know why service boards require names. Tim hawkins then had then replied to us to fill us in with the context and pr from 2015 mapping to this feature.

B

Now that we have seen cube proxy and how it works and how what the limit and the limitations that it carries. Let's see how some of these are dressed in cupping.

A

Cupping to the rescue we are finally getting into the meat of this talk, but first a little history and context. Jan 2020 was when mikhail gave us cupping for christmas october. 2020 was when the kept for cupping was out in march, 2021 jay and a few others decided to start a working group for cupping and since may 2021. Until now we have managed to make quite a bit of progress, rajas and I joined somewhere in this time frame.

A

The cupping team has managed to separate nft ip tables and ipvs back-ends as individual and vendorable plugins. We also added github actions and enable ci that gates all rpr's. We have also added windows user space as a back-end. So far, all right, brace yourself. Now we are going to take a closer look at cupping.

B

Cupping architecture has two parts to it. First is the front end, that is the cupping server, and this is responsible for getting data from the kubernetes api server and then the back end, which is responsible for implementing routing rules. So there can be different backends like ip tables, ipvs and ft, etc.

B

Copying decouples the data plane, that's the back end from the kubernetes control plane. So the cupping server has an api, that's decoupled from the kubernetes api and it stores the global model in memory and then tr, and then any backend can connect to the server through grpc calls.

B

This model allows for innovation in the way networks are built on kubernetes, without the need for adding more technical debt to the kubernetes core.

B

uh Let me just uh dig a bit into the back ends model that we have over here, so back ends have their own go mod and dependencies so that these are pluggable entities and they they interact with coping server using something called as a syncs object, and this is the thing that every backend has to implement and backends, and so the important detail from such a model is that back ends and the cupping server can grow at their own pace, independent of each other.

B

Let me show you the repository of cupping, which is under kubernetes 6 org in github. This is where the different back ends are housed, so every back end comes with its own, go mod and its own dependencies, and things like that. uh This is where the cupping server resides and here's how you can interact with coming. So, let's take a look at this, so here I have.

B

A kind cluster- this is a three node kind cluster, similar to what anusha had it's just that this is having cupping instead of q proxy, and I got this kind cluster up by running something called as hack, e2e, dot sh with ipv4 and the back end is nft in uh in development mode. So this is a script that we run in ci and you can try to run it to get cupping up this lives under the hack directory in copying repository. There is also something called as uh hack, cupping, local up dot sh.

B

So this has iptables backend by default, but that's something that's configurable all right, so this is how you can get cupping up uh on your machine.

B

Okay. So now, let's look at the cupping parts uh that we have over here.

B

So there are, these are the three cupping parts uh and you can see every cupping part has two containers, so this is because it has one container for the cupping server in the other container for the back end uh that's used uh in in the in the cluster.

B

So let me just describe the config the demon set of cupping, and that is how cupping is deployed.

B

All right, so here's the demon set of cupping and as we can see there are two containers. This is the one that is the server and and then uh there's cupping nft container over here. That's listening to the copying server and uh using it to write nftable rules, because that's the backend that we're using so uh nf tables are similar to ip tables and in nft back end in cupping. uh There's something called as a diff. Different object, that's used.

B

So what happens is that cupping server sends the entire state of the kubernetes network to the cupping nft backend through something called as a callback api and then the backend recalculates, the new state of the network and then rewrites the nf table rules. If they are necessary, uh we uh there's a great blog post by jay, pointing out how nft implementation works in keeping and uh we'll come back to it in some time all right cool uh all right. So I have uh cluster cuppings running. Everything is fine. Now, let's see how cupping works.

B

So what what we'll try to do is: let's try to create a simple nginx deployment.

B

All right, so I've created a simple nginx deployment over here and we'll create a nginx service. But before that, let's try to get into uh the cupping part and try to hack with it.

B

All right so now.

B

This is where uh so here, I'm inside uh the cupping part, and what I would like to point out is something called as cupping node log, so here I'll, try to call call this utility and try to connect it with the cupping server. So this gives me a state at a node level of kubernetes services and endpoints.

B

So this is what uh cupping server tries to send to uh the back end, and this is the information that's required for the back backend to create the routing rules, all right so now, uh so this is the current state of the cluster right. So now, let's try to see what happens when we try to create a service for our nginx server, so I'm trying to create a simple cluster ip service and once I create a service, the state changes.

B

Now I have a diff which points out to the new set of end points that are needed for this service. Let's try to uh let's try to update this service and see how that that gets changed real time all right. So let me just change the port or to maybe 8081 and before I enter that I'll. Just make space for the new div that will be formed.

B

All right so here it is so this gives me the state that now the port has changed to 1881 and before that it was 8080.. So this is sort of the information that's coming from the cupping server to the cupping back-end through and this this is the node level information that's required.

B

uh There are other utilities which are similar to node log there's something called as global log, so I can say cupping global log and uh all right, it is cupping hyphen global log- and this gives me the the state of kubernetes networking from a controlled plane perspective and not from a node perspective. So this has endpoint services as well as uh parts of uh kubernetes that are that are relevant right.

B

uh So what this uh does is that this gives us this gives a sense of debugging on at a node level as whereas at uh at a global kubernetes control, plane level. So uh this will help in something like bug reports wherein, uh if someone tries to add a service- and you know or implement service proxy and uh and can then grab the state at the global level and also at the node level and then try to see what sort of uh things are changing and how things are getting communicated so on and so forth.

B

So this is sort of the flexibility flexibility, that's given by cupping and then through through these diffs diff states, a particular backend is able to write its own networking rules. So in this case, nf tables rules will be written. If we had to use iptables backhand, it had to write iptables rules so on and so forth. So this is how the decoupling works in copying from uh from the kubernetes control plane to uh the to the to the back end, that's responsible for the networking aspect, all right.

B

Okay! So now that we have some context on how cupping works and how uh how it tries to decouple and and the architecture of cupping, uh let's see how we can get involved in cupping right. uh So this is a mirror from one of our pairing sessions. So I'm not trying to scare you off it's just a tool that we use to learn things around cube proxy and cupping, and things like that.

B

So uh with this sub project, one of our main goals is to make you proxy for to work and easy to understand, so that we can make your proxy extensible from a command line and a back-end perspective, and we want to do it through cupping. So we want to learn as much as we can about kubernetes service networking and also build an upstream community around it.

A

I know we are just barely scratching the surface for this talk, but if you want to dig more, you can go through the complete cap. There are videos and blocks on nfts and iptables and how nft is implemented in cupping.

A

There are also a couple of tgik and entria live talks on cube proxy and cupping and, of course, if you're adventurous- and you want to write your own backend, there is a blog post for it too, so make sure to check it out as well time for some recap, what do we know and what have we learned so far?

A

The cup split, that is the control plane and uses plane split, makes it possible to develop separate data planes or back-ends without forking off of cube proxy. The cni's no longer have to sync with the q proxy code. They are free to implement their own back-end, while still using the cupping server, as is the kubernetes. Api server is hugely benefited with cupping reducing the number of watch requests.

A

Cupping can be used as a library to create proxies outside of kubernetes. While this is useful for cni plugins, it opens the possibility for anyone to create an independent, plugable proxier for a special purpose.

B

uh The cupping working group is small but dedicated to the project uh and here's the team, so mikhail kind of started cupping and is also behind the nft back end of cupping.

B

J is everywhere. When it comes to cubbing, uh he was also helping out vivek, uh who was behind the ip tables implementation of cupping, and now anusha is helping out with iptables implementation as well.

B

We got the initial implementation for ipvs from ricardo and then uh lars also helped with ipvs and ip tables. Implementation hanuman is now driving the ip ipvs. A backend of cupping and naya is helping out with ipvs as well. Neha is also the founding member of the cupping ci with friedrich, and we got valuable inputs from antonio as well when it comes to cis packs of cupping.

B

um Doug is has been also valuable when it comes to ci and he's also helping out with the windows kernel and user space back-ends a meme added the windows user space back-end to cupping uh pair has been helping us with ipv6 support and uh I've been helping out j with the user space port, uh the linux user space board for cupping.

B

uh So this is this: is the team that's behind cupping? And um it's it's a it's a hard project. Your first comment may take weeks or months, especially if you're new, to cube proxy or to kubernetes networking. However, we will pair with people wanting to join and make an impact. We pair uh we pay program every wednesday and friday at our meetings and also informally at other times as well. So please don't hesitate to join us or ask us questions when meetings or on entry or live stream.

B

That happens every wednesday, uh but be ready for a lot of debugging and uh be ready to take up complex problems when it comes to cupping feel free to join us on slack as well on kubernetes slack on sig network kpng channel all right. So thanks for listening to us and hope to see you all in the next coming meeting.