Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2018 (Copenhagen), 14 Nov 2015

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: gRPC Load Balancing on Kubernetes - Jan Tattermusch, Google (Intermediate Skill Level)

Description

Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18)

gRPC Load Balancing on Kubernetes - Jan Tattermusch, Google (Intermediate Skill Level)

Load-balancing is a must for everyone building scalable services. This talk will give an overview of gRPC's load balancing options and show how to use them in a containerized environment.

About Jan
Senior Software Engineer at Google. Working on the gRPC project for the last 3 years.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Okay, so hello, everyone, my name is Yan I'm software engineer at Google and I've been working on the GRDC project for the past few years. I think it's been almost four now. My talk today is called G, RPC load, balancing on kubernetes and here's. What you're gonna do so initially I wanted to give you an overview of some basic load balancing concepts and tell you how they apply to G, RPC traffic and kubernetes environments, specifically because there are some specific customers mentioning Oh choosing the right load.

A

Balancing solution for your application depends on many factors and there's no really one-size-fits-all solution. So, instead of just telling you what load balancing solution you should use for your application, I'm gonna give you an overview of what's available. What's out there and I'll mention the pros and cons of those approaches, and then you can hopefully choose for yourself, even though most of the concepts are actually relatively simple, I think it's always useful to get your hands a little bit dirty, so I've prepared a little demo. That's gonna demonstrate some of the basic load. Balancing scenarios.

A

I now also show a few useful tricks that relate to specifically G RPC traffic and specifically the G RPC environment.

A

Okay, clicker doesn't work as always.

A

A

Yeah, okay, anyway, so I'm pretty sure everyone knows what load balancing is. Oh, so I'll just do a quick review. So, first of all, load balancing is one of the key techniques how to build scalable services. Obviously, oh, and the idea is that you have a client and you have multiple possible backends to talk to you and you just obviously are balanced log between them by like sending some requests to some of the backends and sending other requests to other backends.

A

We do that too, improve the overall throughput of a service and also quite possibly to decrease the latency of our request. Oh and one of the side effects is that we also avoid overloading one of the backends because, like if all the requests went all the way, always to the same back end. You know it's not gonna end up. Well, oh one of the or some of the perhaps a little bit less obvious things is when you set up load balancing correctly.

A

Oh you're gonna get some of the interesting features you want for your application for free. Oh so you'll get improved tolerance for banking failures because, with correctly set up load balancing if one of the backends goes down or all the traffic, it's is seamlessly rerouted to other backends and you'll also get the possibility to basically update versions of your service on the fly by or using techniques like rolling updates and similar things know specifically when it comes to the kubernetes environment. Oh kubernetes is really useful for deploying applications with the micro service architecture and.

A

Because, with micro service architecture, you usually have multiple components in your system, and so many of those components actually have multiple replicas. One way to think of it is if you're deploying able to get application to kubernetes. You can't really not use load balancing it's just question of like if you're doing it right or not.

A

So one of the key distinctions when it comes to load balancing is so the distinction between the so called l, 4 and l, 7, l, 7, sorry, l, 7, load, balancing these labels are based on the OSI network model and they basically described at what level in the hi model, the load balancing is happening, so the L 4 load, balancer, is so-called connection based and the l 7 is happening at the application layer and the key difference is.

A

The difference is basically the granularity at which the actual active load balancing is happening so with the elf 4 load balancer, the load balancing is happening any time you create a new tcp/ip connection and for some of api's that actually might be fine. So if you're talking, for example, about very common HTTP 1 rest api. So because we create new tcp/ip connections very often because of some of the problems of HTTP 1 protocol like head-of-line blocking or the load balancing actually ends up working quite well.

A

But this is not necessarily the case of G RPC, because G RPC is basically a thin layer on top of HTTP 2 protocol and.

A

Every RPC call is a separate stream on the same tcp/ip connection, and this is one of the techniques how G RPC achieves good performance, so performance is great, but the bottom line is, if you create a G RPC channel, which is basically an abstraction for a tcp/ip connection and you're talking to a back-end or like once, you once you connect the channel. All your our pcs are gonna, go to the same back-end, so there's basically gonna be no load balancing happening and obviously we want to do the load balancing at the per RPC level.

A

One potential problem in kubernetes environment is that kubernetes has a built in load balancing feature, but this feature only operates at the at the l4 level. So when you set up a service of type cluster, IP or load balancer, it's only gonna be l4, and that might be a problem.

A

Another important distinction of load balancers is so basically we're in the data path. You've put the load, balancer and there's so-called proxy load balancing, which basically means you have a client and you have the backend. And then you put the proxy right in right in between those.

A

What's good about this solution is that you get a simple client which doesn't have to be a variable load balancing at all and then, although possibly complicated logic or Stace into proxy, or you also don't have to trust your clients, oh, but on the other hand, he'll get higher latency and overhead.

A

I'll kubernetes there's there's a good way how to deploy the proxy and it's called the so-called citecar proxy, which basically means in the same pot. You deploy you deploy the proxy right next to your service. The other approach is so called client load balancing, and this one is especially useful when you really want to achieve low, latency and low overhead, and in this approach the client is fully aware of what all the backends are and and implement some kind of strategy to choose which back-end to talk to you for every given RPC.

A

So one of the problems with so the client side load balancing is that if the logic of load balancing gets too complicated, you basically need to make it into your client and with GRP see we have implementation in ten languages.

A

So if you wanted to have a complex logic that, for example, involves deciding which back-end to route to based on what's the current server load or you know which, which back end has already handled the most traffic, you would have to implement all this logic in a simple client in in all the languages that G RPC is available in and this this might be a problem, so there's actually workaround for this kind of problem, and it's basically that you take all the non-trivial load balancing logic and you move it out to an external component.

A

So another picture, you can see that you have so-called leukocyte loo controller and the way it works is the G RPC client before it connects to all the back ends. It works. Its first establishes a connection to look a site controller or, and it's going to be a streaming RPC and the leucocyte. The load controller tells GA RPC client, what are the available backends and what's the right strategy to balance load between them?

A

This is actually an approach. That's used quite heavily internally at Google, and it's been battle tested on many big services. I.

A

Also have to mention a model: that's gaining a lot of popularity recently, so it's the service mesh model, oh I, hope everyone is familiar with what the service mat service mesh is. Can you just like raisins just to get an idea? Okay, so yeah good portion, I'm gonna, talk just specifically about the load balancing part because service mesh can give you a lot of other features that are useful.

A

So in this diagram you have the G, RPC, client and all the traffic from that client transparently goes through a proxy and the proxy talks to a so called cluster manager, which can be, for example, a steel pilot and.

A

This cluster manager basically decides what's the load, blending strategy gonna be and the proxy implements that, and it talks to all the backends without the client knowing so it's so. This is basically a proxy load balancing solution a little bit on steroids that the disadvantages is that you are basically paying two times the proxy cost, because the the call needs to go to through proxies to proxies are not necessarily you don't necessarily need two proxies for load balancing itself, but for some other things like encryption in the service mesh.

A

It's something that you you actually need.

A

Okay, so that was the theory. Let's look at some of the load balancing solution that you can use with gr PC so from proxy load. Balancing we have the like the two most popular proxies that you can use our envoy and nginx. O envoy is so relatively new project that saw recently has so gained a lot of momentum and some of the differentiating factors about it is that it it's high performance but at the same time, it's possible to configure the Envoy proxy dynamically.

A

So, and this was basically the example, I showed with the service mesh. So in your deployment, you can have a bunch of proxies that when they start up, they talk to the cluster manager and they get their configuration from there. So it actually works a little bit like the DHCP configuration in your network.

A

Nginx is another very popular project that's been around for a while, and they recently added a full G RPC support. So you can actually specify some of the nginx rules. Oh specifically for JRPG traffic for doing proxy load, balancing in a service match or you can use. The combination of envoy and sto sto is basically the cluster manager and envoy is the proxy or you can use the linker D project and they're.

A

Obviously, like many other service meshes but I'm just trying to mention, though the ones I think are having the most momentum- or you know, are the most popular currently o.

A

For simple or for client load, balancing G RPC, like the vanilla version of G RPC, has a implementation of a simple, around throbbin load, balancer in, like all the clients, so energy RPC go G, RPC Java and the ERP CC core or you have you, have this implementation and.

A

A

Okay, sometimes happens but yeah, so you have the client load balancer and you you have the round-robin resolver, which basically works based on so there's a resolver and the resolver results at the NS record and based on, like all the DNS records it gets back. These are going to be the backends that the round-robin load balancer is gonna load the lens to so.

A

The other solution is the load balance, leukocyte, client, side load, balancer that I showed before and G RPC actually implements the so called GRP CLB protocol, which is a simple protocol buffer based G RPC protocol, which can be used in the scenario where there's a load. Balancer is moved to another component that next, oh, that's, basically not part of the backends. But it's it's. It's a separate service.

A

One of the problems with the ERP CLB load balancing solution is that, even though the client is so fully available in all the gr PC versions that are all currently available, that we actually don't have a implementation of the load balancing controller server, because this protocol is mostly used internally at Google right now.

A

So there are actually two ways we could we could handle this. One of them is basically implement the ERP CLD server that the client can talk to you, but because this protocol is mostly used internally at Google and it's not a de facto standard like in the open source world. We actually think a better way is to adopt so called anvil. Universal data plane API, which is also protocol buffer, based it's it's kind of a similar protocol, but it already has adoption by or some of the but some of the like existing open source project.

A

So it fits better in the in the whole ecosystem, with the universal data plane. Api support, there's basically two possible deployment scenarios, and one of them is that you still use the proxy, but the proxy does the load, the leucocyte load, balancing by talking to the cluster controller and that's available right now, and we're actually going to see it in one of our examples and the other approach is that we will implement the client for the universal data, plane, API right into G RPC and that's so basically work in progress. Okay,.

A

So that was the that was the theory, and now I will need to do a little bit of setup and I'll show you some examples. I prepared. All of these examples are.

A

Available on github so.

A

After the talk, you'll be able to check out like all the code and see what all the settings are. Oh, we don't that much time here so.

A

Okay, so the first scenario I'm gonna be showing is the simple round robin load balancer, or what do we need to do in order to set this up in kubernetes is so when you deploy a service to communities.

A

You basically have multiple service types you can deploy and in this case we're gonna use the headless service, and the reason for that is that we want the kubernetes dns to expose multiple DNS records for each of the replicas available for the service, so that g RPC can basically resolve that dns record all those dns records and then see that there's more backends to load lines.

A

We need to set the load balancing policy in the G RPC client, and we basically do that by setting a channel argument and we connect to the service as usual, so we're going to use basically the default name. You use for naming services inside kubernetes.

A

Okay, so I've done some prep work, so I basically have a simple gke cluster. So google kubernetes engine- oh it's it's empty, like nose, so no services are running. So it's cube. Cg.O parts.

A

Okay and you can see that the cluster is empty, so what I'm gonna do I'm gonna, deploy or as a service first and then I'm gonna, deploy a client and I'll show you that the load balancing is actually happening. So the client is a really simple client written c-sharp and the only thing it does.

A

It basically creates a a G RPC channel can actually everyone see that even in the back row sounds good, oh, so you have a or that that creates on only one channel and then basically keeps doing a simple RPC on that channel, and what's that supposed to demonstrate that, oh, when you're, even when you're on the same channel, oh your calls are going to reach different backends at each time, Oh for the server source code. It's basically a really simple service that starts.

A

The server listens on a port and then replies with its own IP, so that we can see that we actually what replicas we actually reached.

A

So I have a humble definition of of the server I'm gonna be deploying and you can see the cluster IP is set to none. So that means we're gonna be setting up a headless service. Oh it's exposed to port 8000. Oh I've already pushed all the necessary doctor images to the container registry, so I can just go ahead and deploy the service. Now.

A

Service is gonna, have three replicas and.

A

Now I'm gonna deploy the client that demonstrates the round-robin. Oh I'm gonna use the target. Basically the regular kubernetes DNS name I'm, going to use an environment variable to set up what the current load balancing policy is.

A

Okay, so the client has started, and now I'm gonna quickly show.

A

Okay, so as you can see or because I said, there's three back ends and then, though, all the calls basically round-robin around the three back ends that are currently available. So if I show you what the service replicas are.

A

Okay, so you can see like these IP addresses correspond to the three end points that I've deployed for my service, so that was the.

A

So there was the simple round-robin client site load. Balancer, oh good thing about it- is that it's really simple to set up and works out of the box or the bad part is it doesn't take server load into account and handling those scale-up functionality currently requires a little workaround on the server side, because DNS is not really dynamic, and so you need to periodically every resolve all the backends in order to get updates about what replicas are still available.

A

Oh, the other example I want to show is so load balancing with sidecar proxy I'm gonna use, envoi and I'm gonna shoot to a variation of this. The first one is so it's a non-void proxy. That's configured statically, which is one of the ways you can configure envoy. So I'm still gonna use the headless service to expose, although all the replicas as a DNS entry, the end of a poxy is going to be set up as a really simple side. Cont container.

A

So this is the configure of the greeter client with static envoi Oh.

A

In this scenario,.

A

You can actually see that there's two containers or in the in the gamble file and one of them is the actual greeter client and the other one is the Envoy proxy that's deployed as a sidecar. So that's the most important thing about it. The static configuration of envoy is is basically, it uses the round-robin load, balancing policy and it's subtypes trig DNS, which happens to be what we need right now and it's gonna round robin or among all the backends.

A

Okay, so these sub locks for the client I just ran okay and because I have I now have two containers deployed in the same pot. I need to specify with China which, for which one I want to see the locks. So this is actually the greeter client, because the other one is basically the sidecar proxy and again you can see that I'm reaching multiple backends.

A

The next configuration is is basically using envoi as a dynamic proxy. So what I've done in advance? I've installed instance of Steel in this demo, cluster and I.

A

Can also deploy a.

A

And voi so dynamically configured and I'm gonna use the sto CTL kuben check command, so that basically means I. Take a a llamo definition that only contains my my simple greeter client and the sto CTL group. Inject command is gonna edit. The sidecar definition into into my mo file.

A

Okay, interesting.

A

A

Thank you. Actually, someone stole my printed out notes right before the talk. So oh.

A

Thank you so I see where some sto experts here.

A

So I I have the dynamic envoy running aside my greater client.

A

And I'm going to show you the locks.

A

Okay and I actually forgot to deploy the the service for that. So.

A

Oh, no, it's actually fine.

A

Okay- and you can still see that we are a lot of Lansing to multiple backends or another thing we could do is I can show you that if I reconfigure, the number of replicas.

A

So, for example, I change the number of replicas of the creature server to just.

A

1X and adjust one replicas, and basically the configuration has picked up that we shouldn't load blind student on existing replicas anymore.

A

So that was the Envoy, with we're studying and in a dynamic configuration in a similar way. We could set up the whole scenario in our sto service mesh.

A

So for that we would also use the sto CTL kuben check command to basically inject the sidecar next to both our client and both our server and.

A

I've also prepared an example of.

A

How to set up the GRP CLB leukocyte load balancing, but we're actually running out of time, so I'm gonna skip that. But all the examples, as I said, are online, so you're feel you're free to check out how to set that up and experiment with your experimental kubernetes cluster as well.

A

I have two more interesting things, so one of them is the challenge of balancing streaming. Our pcs, besides from unary calls G RPC also supports so-called streaming. Also, you have an RPC that supports multiple requests and multiple responses, and those RPGs can be a relatively long-lived. And that's that's one thing that that you need to keep in mind when you're designing your API, because once you start a streaming call there. So even if you have l7 load balancers set up, you basically like you, can't really change the backend that you're talking to you.

A

So some of the strategies how you can now approach. That is that you can restart your streaming, calls periodically to basically make sure that you load balance to our different backends each time or you can also use the max connection age setting and on the server side to make sure this happens automatically for you, the Mexican ection age can be also used in scenarios where you can't really do anything else than L for load, balancing to kind of make L for load blending a little bit less worse for your pc's purposes.

A

Okay, oh I'm. Almost out of time, so I'm gonna. Thank you for attention. A few things that I wanted to point out is so tomorrow, I'm gonna have office hours in the CN CF booth, which is in hall C now and I'm gonna be already there to answer any questions about GRP, so you might have not necessarily about just load. Balancing and Michael III will be having a talk birthday in the afternoon and it's gonna be a gr PC deep dive.

A

It's gonna be oh about life of a gr, PC call and what were some interesting API choices in G, our PC or it's mostly for everyone, but it's gonna be especially useful for people that want to that our power users of G, our PC, or that maybe you want to become contributors. Oh I'll also ask you to fill out a little sorry. Oh I, don't know if you have time for questions still, okay, so any questions I need to grab the mic. Oh.

A

B

Yeah. Thank you. It was very good. Just one question about the examples. In your examples, the client is on always within you know the same environment as the server's I.

A

See? Okay, so you basically mean okay, so the question was about that. This is basically only internal load, one thing within the cluster: yes, so if you wanted to do external load balancing you basically and you want to do l7 load balancing you basically set up so called ingress in kubernetes, oh, and so you can also use things like unveil proxy for that, but unfortunately, I didn't have enough time to cover that in the talk. But that's something that that can be done and the thing to look up is the kubernetes ingress. Okay, thanks.