Cloud Native Computing Foundation KubeCon + CloudNativeCon China 2019 (Shanghai), 5 Jul 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Cloud Native Load Balancing From Scratch - Fangyuan Li, VMware

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Cloud Native Load Balancing From Scratch - Fangyuan Li, VMware

Load balancing is required by all modern distributed systems, which can be done in many places and multiple layers. With all the options on the table, eg: gRPC, Envoy, IPVS/BPF, have you been curious of when and how any one of them will fit into your stack and scale your system as traffic grows? Aiming to “demystify” load balancing in K8s, I’d like to share my experience utilizing these cloud-native load balancing techniques, discussing their tradeoffs and applicable scenarios. The journey starts with a simple VIP iptable implementation. Then we dive deeper and examine from client to server, kernel to userspace, going up from IP to HTTP. We’ll draw the big picture and demonstrate the use of different tools in plain Bash scripts. With these building blocks in mind, we’ll summarize by comparing them across K8s(clusterIP/nodePort), Docker(routing mesh), Envoy, and a number of CNI plugins.

https://sched.co/Nro2

A

I'm a software engineer from vmware. I've been working on a project called pks um since its beginning for almost for almost two years now um so today, I'm going to talk about load balancing, especially in the context of cal native.

A

Okay, um so here's what I'm going to do. Firstly, I'll give you a brief introduction of all the concepts related to load balancer and then I'll walk you through the life cycle of a simple packet and point out where and how load balancing is happening.

A

Also, our list of other techniques can be used to implement some concrete solutions.

A

I do think that sometimes it really helps to get your hands dirty to gain a better understanding of all the concepts. That's why I prepared a demo at last um to show you how to build a reliable and scalable l4 balancer by running a bunch of very simple bash scripts.

A

Okay, let's get started so I'm pretty sure everyone here are already familiar with the term load balancing, but what does it mean it can't native contacts right um so around five years ago I mean even for now, some big companies are still depending on very expensive box boxes, to implement this or to fulfill this load balancing functionality.

A

They are great, but they do come with a bunch of disadvantages. First of all, they are very expensive.

A

The upgrade is very costly and it takes lots of training and money to get the internal working of the very fancy boxes, also as our as our application scales up, it's very hard to meet the aha requirement, because the redundancy model here it provides is one plus one, which means you have one active low, balancer running to serve other requests and the other one is standing by um it's also syncing states with this active one.

A

um Also it brings itself up once it detects a failure of the active one, but it's just good, not good enough right. Also, um if we're using hardware or some software in the hardware to implement this, it lacks the program programmability and flexibility um for quick iteration. um The same way we do for our cognitive applications.

A

Okay, so we like to have our load balancer defined in software and to scale our load balancer itself the same way as we scale our applications.

A

We want to keep this software as stateless as possible right then we have this scale lab model which provides earn plus one redundancy. Also, the department itself is extremely thin simplified because we can deploy the load balancer instance. The same way we deploy our application. We can just add more load balancer instances to provide a better look balancing functionality.

A

With that help, we can also short the load balancer instances um to provide performance, isolation to implement different sla, for example, right container native, if you think about a vanilla, kubernetes deployment, where you have at least three microwaves and you deploy this one container, that is serving one service. Now you expose this service and you have corresponding cluster ip service ingress and before the traffic hits the target container.

A

If you really think about it, it's very likely that it'll hit the bm firstly and firmware the routing decision will be made again and the request will be routed to the other vm, where this container is actually running. So it just adds a bunch of extra latency and overhead, and we don't want that right. So by being container native um from can load balance load balancers perspective, um other containers are the same endpoints just same as vms right, then we can count all the latency and overhead.

A

It's also very important to be the load dancer to be able to um interoperate with other cloud services by being cloud native, for example, um some security related features provided by cloud provider like ddos mitigation right or caching, like cal cdn or some identity-based authentication.

A

It's also very important to not ask our clients or customers to do any manual provisioning of platform, related resources, in this case loop, dancer itself as customer's application scales up. We want the load dancer cell to be automatically deployed and scaled to be consistent with our application, also in cognitive world, as we know that our backhand containers they come and go. So it's a very um it's a highly um dynamic environment right and to be able to run requests to the endpoints on currency endpoints.

A

We need our load balancer to be highly configurable to adapt to other changes spontaneously.

A

So with all this being said, let's take a look at the basic traffic flow or the lifecycle package. Here. This picture demonstrates a very classic model where we have kinds um all the packets are generating the kind and before it hits a target. Backhand server it'll reach, middleware called load balancer in cognitive world. We tend to model independent modules into separate microservices and depending on how they interact uh with each other. It can be very complex, but if you really think about it, the service that is making a request is actually acting as a kind.

A

So we still have this old classic model where three phases low balancing, can happen right, client load, balancer and service. So now, let's take a look at them, one by one on client side. If you have config control or if you have partial control over kinds, you can start low balancing already right. In this case, you can choose to use grpc by making calls to grpc library in your application. Application code and grpc itself is smart enough to directly routing requests to different back-end servers.

A

If you feel like that's too much, okay, grpc shouldn't be responsible for this load. Balancing decision, that's fine. They also provide something called look aside, load balancer. So that means this grpc kind can talk to a third party entity that implements grpc lb protocol, so it accidents entity for rotting decision.

A

Then this responsibility can be fully dedicated to the third party.

A

It sounds all good, but the problem is not everyone understand this grpc lb protocol, so in istio, the way they expose the data plane is called something universal data ping api right, so it abstracts out all the data pane in all aspects.

A

So grpc is also considering implement this universal data pane api, and in that case the grpc kind will make requests directly to let's say istio control ping for routing decision, and after that the request will be automatically routed to different backend servers.

A

If you don't have any control over clients, let's say you have an application. That is an internal service that talks to other services and in this case it's a kind itself, but you just don't have control over this this um this code or, let's, say a jar package whatever it runs just fine and you don't want to touch it um in this case we can adopt a model called sidecar proxy. I believe everyone um are very familiar with this. um It's actually the way istio is leveraging online policy right.

A

You debug proxy, together with all the clients they talk to each other through some local networking or sockets, and then the proxy will make the writing decisions.

A

But most time we just don't have any control on clients and most time load balancing actually happens in the middleware.

A

Of course, our old friend dns can be used here. You can define a a record to map your on survey's name to different ip or even one ip. In that case, any cost can be used. This ip itself can be backed up by different servers that deployed across the world actually and depending on where the server is deployed.

A

The request can be routed to the closest location. I'm going to show you later um how to how to do this um in terms vip and ecmp.

A

So vip here means virtualized ip right, instead of thinking as something that um we assigned to a physical nic or a virtualized nic on vm, specifically, um it's more like a abstraction right in l4 networking, it's like a name. It's like the dns name. We can actually map this ip address to a bunch of clusters or a bunch of servers and ecmp itself or equal cost. Multiple path is way to scale one ip or one virtualized ip. So the idea is quite simple.

A

Now the cluster is backed up by this router right, that's serving this vip and we have a bunch of routers that are still in the same vip.

A

We asked these routers to broadcast this vip to the upstream to the same upstream router, then, from this upstream router's perspective, this vip is actually associated with a bunch of next hops right. So it's able to pick up anyone from the routers and just send the traffic to that, because they are all serving the same vip.

A

The challenge here is how to reduce the connection disruption as the number of load, balancer instance and back-end server number change right. If that happens, we want to keep all the packets from the same stream or sync connection go into the same. Backend server and consistent hashing is a technique we use here. So it gives you this guarantee that.

A

The number of disruptive connections will be at most as a one over n if the total number of servers is n, it's good enough, sometimes, for example, in google cloud. um This is actually is a mac that the picture here shows that software called macdab. um It's actually the paper published by google, and this is the thing that's supporting google cloud networking right to them. Load balancing is very important, so they don't care that much about connection disruption but depending on application, sometimes it's very important.

A

So in gitlab, for example, they took some other techniques together with consistent hashing to minimize the number of disrupted connections.

A

With all the tasks with all these techniques, um it's all good, but sometimes we still have the limitation that the load balancer instance itself has to be deployed in the same l2 network, with our back-end servers. um I'll, give give you more details why this is a limitation, um but for now, let's just assume this limitation. So now we want to like scale beyond this l2 networking right. We don't want all our backhand server to deploy in the same l2 networking or on the scene switch. So the way to do that is very simple.

A

We use encapsulation or just tunneling right. We wrap the original packet use, another packet and directly download the connection to a remotely robbed boat server.

A

From service to service, of course, dns can still be used in vip, virtualized, ip or cluster ip in kubernetes term right, it's mostly man using a bunch of ip table rules in l4, as we know, except ip. We also have part number right so here. The port number itself is also part of the address, the abstraction right.

A

So it's very important sometimes to be able to expose the same port number across a bunch of servers and when you send requests to either one of the vms, they can be routed to the same set of containers right, so it's called node port in kubernetes or writing match in docker swarm, and they are mostly implemented by something called ipvs which I'm going to demonstrate how to use it later with a bunch of details.

A

Of course, service mesh can be used here. It provides something fancier like l7, um low balancing, for example, it's able to rot your request based on some fields or, for example, url path right in http protocol. It's able to route different requests based on your path to different backend servers.

A

So now, um with this concept being said, I mean this concept is just very dry here and really talking is very cheap.

A

So, let's just get into demo right as we can see from the previous list here all these techniques listed here, the l4 load balancing um is actually the most fundamental building block right and that's what I'm going to do, or I'm going to show in this demo, um is to help teach you how to extend one ip to make it an name or an abstraction in terms of l4 networking, and I'm going to do this by using some widely available open source softwares like ipvs or just dummy interface or and bgp or ecmp,.

A

Before I start demo, I'd like to give you a very brief introduction of the ipvs yourself, so ipvs is called ip virtual server. It's actually something that has existing linux kernel for over a decade. Surprisingly right and it's widely used now in kubernetes and docker to implement the things I just mentioned, like cluster ip and running mesh, and no port like that and what it provides is essentially just l forward balancing with many algorithms like round robin this connection, um destination, hashing, blah blah blah.

A

It's also able to stay on to sync states with other ips instances so that if one of the ipvs goes down, um we can still keep the states. They are doing that through multicast, but it only supports ipv4. Now now, let's start a demo, so in this demo I'm going to start from the very simple department here.

A

Can everyone see the see the graph here or I can just uh describe what it is? Okay, so the top box here this guy here is the roger.

A

So it's actually my look my laptop here right and deploy two other vms. The first one has ipvs running on that that serves as a load balancer and this guy here as a server. It runs a very simple golden hp server and the client here is actually also my laptop I'm going to make a bunch of requests to this guy here and the first mode supported by ipvs, is called net mode or networking address.

A

Translation idea is very simple that when the request comes to the ipvs, the destination ip will be translated to this real server's ip and the default routine, for this server will be set to this guy here. So when the response, so the response will be routed back and the the net will be armed down, and so the response can be sent right back to client.

A

Now, let's see how it works.

A

A

So here I have the first vm um that has ipvs running on that so and the same box. Here um I have the second vm. Then I'm gonna start this server right. um So what this step does really is, as you can see here, u2 step, one, a step: two sorry, it's too small.

A

It removes the default routing which is 33.1, which is my router. I want to change it to the 33.2, which is the ipvs vm right and also I started server, it's quite simple. Actually, then, I start this ipvs on the first vm and I add this server into ipvs.

A

As you can see here, the virtual ip is configured on this ipvs and we also add the server into the ipvs. Now, when a request goes to this ipbs, it's supposed to send a request to the backend server here, which is the other vm I configured.

A

Now let me start the clients.

A

So what the class does is it's just making curl requests to the endpoint with the virtual ip and it slips one second, every every time for for this one thousand times, which is uh hopefully this demo can end in 18 minutes. Okay, let's see.

A

A

As we can see, the server starts serving requests here right. That's all good! Let's go back to the ppt and see what's wrong with that.

A

So net is very pain and simple, but it comes with a bunch of disadvantages.

A

For one thing, all the packets for this connection have to go through the same ipvs right. That's a load answer: it's not like something everyone send traffic. To that. uh Let's say you are download a huge file. So in that case the request itself is way lighter than the response right. It doesn't make any sense for your response to all go. Go through this loop answer. So that's one thing. The other thing is, we also add extra overhead for the critical path here, because we need to modify each packet right.

A

We are doing dna here for each packet. Oh thank you for each packet, so we would like to let the server send the response directly back to client and how to do that. Here comes the second mode supported by ipvs, which is called direct routing. The idea is this server here. This ipvs here will send the packet in l2 directly to this server.

A

It doesn't do any ip level change. It sends this packet in l2 right a switch level directly to this guy here, and we have a dummy interface configured on this guy with the virtual ip.

A

Then we configure routing to the default routing to this guy, so the request can be uh sorry, so response can be sent back to the kind.

A

This technique is called directly server return, with the assumption that, with the assumption that the server has to deploy in the same l2 network with the loop answer, let's see how it works.

A

Now I'm gonna start a second on vm.

A

What this step does is it sets up this virtual ip on the diamond interface? Also, it starts this golden server and then I'm going to add this server into my load. Balancer.

A

Okay, as you can see here, this server is added into the banner right 33.4.

A

If we go to this this page here we can see it starts serving requests as we waste that, but you might be wondering why this only is only the second server. That's serving the request and reason is the default. Algorithm is called wlc or weighted list connection, so the load balancer itself will send all the requests to the server with the list connection, let me let me do a watch on the load, balancer stats and see what happens.

A

As you can see here, the active connection for the first one is 64.. The second one is um 55, so the second one is still catching up and before we reach the same number, the load answer will send all the incoming required to the second one.

A

With this direct quality mode, we just liberate ourselves from the limitation that puts on us right. The response can be sent directly back to client and we are okay to download some huge files in this case, but still with the assumption that we have to, we have to deploy loopbanger and server in the same l2 network, and we want to scale beyond l2 the way to do. That is a third mode, supported by ipvs called tunnel mode.

A

The idea just like what what I described before it encapsulates this packet with another packet and it tunnels this connection to a remotely rockball server. So here the vm is deployed actually in another subnet. If you can see that it's deploying 34.2, these three are deploying 33 subnet right, so it's just remotely rubble, it's not in the same l2 network and the way to configure that is to set vip on this tunnel 0 interface, which is the default linux default, ipip um interface.

A

Okay, so ipip is a protocol used by ipvs here for encapsulation, which means to wrap one ip packet with a ip packet.

A

On that step they stopped.

A

Now I start a third server, um so what this command does? Is it just sets up tunnel mode right on the server like I described before, and it starts this guillain server? Okay, then I'm going to add it into the load. Bancer.

A

As you can see, 34.2 is also added into the load balancer. If we go to the client here- oh my god,.

A

Well, I don't know what's happening here. um It looks like there's some failure, but that's fine, because we are going to have health check on the load balancer later in the demo. So now um the requests are serving by these servers, even with some temporary video, probably because I'm not working in this building. I don't know if you look at this picture here.

A

They all have the active connections around 10 or 20..

A

A

So now everything is good right. We are able to scale beyond l2. What's the problem, then, if you look at the department here, the load balancer instance itself becomes a single point. Failure, it's a bottleneck here. If this guy goes down, it brings down all the other servers.

A

So we want to scale beyond this ip, this virtual ip that is supported by this servers right. So the packing used here is called ecmp.

A

um The idea is, this: guy will run, go bgp to broadcast this virtual ip to upstream router, and we are going to replicate this deployment here to have a second load. Balancer instance with other servers and the second load balancer instance, is going to broadcast the same vip to the upstream router so that from upstream router it act. It actually has have um two routers with the same supported by the same vip. Let me first replicate the department here.

A

So now I'm gonna deploy two more vms.

A

A

What okay, oh nice, um I'm going to deploy two more vms here um in different subnets and then I'm going to have a second load. Balancer instance.

A

um Then I'll set up the second little bandster instance.

A

As you can see here, the second little dancer. Oh sorry,.

A

The second little bouncer is here with two other servers configured um now, let's say if it works, I'll change, the default routing on kind to the second load, balancer.

A

So now we just change the default lock into 33.5, which is second load balancer and, ideally, request will be routed to the other two servers: okay, nice. So, as you can see, these two guys start serving requests.

A

Okay and now we want to scale beyond this virtual ip, these two load balancer instances and we're going to start go bgp on both instances.

A

So what this command starts is oh, okay um is, it starts, go bgp on both ipvs and ideally it should add.

A

It should broadcast the virtual ip.

A

Let me run this manually.

A

Okay, so we just started: go bgp on both uh load balancers and they broadcast the same vip to its peer, which is the upstream router, and on this host you can see the routing table here. It actually has this vip associated with two next hops, and this this guy will pick up one of the two to send all the traffic and to make this guy work. We need to change the default routine to to the upstream router, which is 33.6.

A

And now, if you go to here, let's see okay cool. um So all the requests are started serving by the first little dancer. But in this case, um even even even if even if the first one fails, um we have no problem because we just use go bgp to backup this virtual ip by two routers. Okay. Let's then mimic the failure here to by shutting down the vm and before that I'll add some health check.

A

Okay, um so health check is very important to the bouncer, because you need to detail the failed server to stop sending incoming requests to the failed one, and in this case we can only support. We can imagine the scalability and availability we mentioned before. So let me add some health check on a load balancer.

A

So the health check here is also very simple: some batch functions. What it does really is. It makes requests on this endpoint. If it fails, it's going to run this command. If it succeeds, it's going to run the second one. The first one is to remove this endpoint from the load balancer. The second one is to add it back. Okay, so it's going to keep querying the end point and either add it or remove it.

A

Now we just started the health check and we can see the house check. Request is showing up on these three servers. Let me mimic the failure by shutting this guy down.

A

Cool, as you can see here, the sec, the third one is removed automatically and from clients perspective, there's no disruption at all. All the requests are served right by these two. The third one is, even though it fails, it's removed automatically by the health check function, and let me just remove the other two and see if this traffic switch to the second guy here.

A

Okay, let me shut this guy down.

A

Now it's done and you can see it's also removed from the from the ipvs. That's here. Let me also shut this guy down. Oh nice, so there's no disruption at all. As you can see when I shot this guy, it's 619 or something like that, and all the requests are automatically routed to the second load balancer right. They are served by this cluster servers.

A

So if you go back to here, you can see the first load. Balancer is automatically removed from the upstream router okay. So in this way we just build a very stable and reliable l4 load balancer. Since we are running up time, I'm going to stop here feel free to ask me any questions.

A

Also, the setup is available on github. If you have any questions, just ask me or send me an email thanks to everyone for joining and bearing with me for this very long demo. Thank you.

A

So, what's the difference between between 20ws and the ros.

A

So, what's the difference between ipos and the rms? uh Sorry can you see? I I uh second one was the second one: l, yes lvs, yes and another open source. Note balance obvious. I'm I'm not aware of that. Sorry: okay, yeah lbs!

A

I I think it's just linux virtual server or something maybe they are the same thing right yeah. They are same thing just the name. Names are different, it's the same thing ipvs or lvs. Sometimes it's obvious. Okay! Thank you.

A

Any question: okay, thank you guys.