Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2019 (Barcelona), 23 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: gRPC load balancing and Service Mesh - Vishal Powar, Google

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

gRPC load balancing and Service Mesh - Vishal Powar, Google

Service mesh architecture proposes a unique approach for control plane management (including load balancing). This talk goes over the options available for gRPC load-balancing in the context of a gRPC integration in a service mesh. In this talk, we will cover: various load balancing options available for gRPC and the tradeoff associated with each one of them. load balancing at scale with and without gRPC lookaside load balancing. proxy based load balancing and Service Mesh along with considerations when using them as an alternative to lookaside gRPC load balancing. how unified endpoint management can be achieved for Health check and load metric reporting for both gRPC load balancing and Service Mesh.

https://sched.co/MPbI

A

Hello: okay: this is working hi I am Michelle I'm, a software engineer at Google I work in the load, balancing team, where we try to solve and the load balancing polish problems for Google services and also for Google Cloud customers, some of the learnings that we've had over years, we'll see in this presentation, but this presentation will primarily focus on load balancing for a large service. We are not going to cover a lot of basics of grpe load balancing, because this is something that has been covered before in queue con 18 by my colleague liane.

A

If you have, since we can talk about it at the end of this presentation, yan is here, so you can ask him those questions. So what do I mean by a large service? A large service is generally a service which has a lot of clients, a lot of requests that the clients need to make and generally clients which are distributed pretty much globally.

A

So when we talk about the problems- and we talk about the things that we want to care about, will Jen when I say large service is the aspect of the service that I am going to talk about. So this is we have a small agenda for this presentation? Will look at load balancing in G RPC at a very high level go quickly over it then talk about centralized load balancing?

A

What does it mean and how all that goes into load, balancing at scale, and then we look at service mesh and how does that work with G RPC services? So why load, balancing I believe I? Don't need to talk a lot about that, but generally load balancing is used to improve a throughput of a service.

A

So if you have a very large service, you would need multiple, endpoints or backends, for that particular service which can serve those clients so that you can scale, and it also means that more endpoints you have it includes this: your service, reliability and availability. These are important for load balancing, but outside of that, there are a couple of other things that load balancing needs to worry about. One is the client requirements. You can't really have clients connect to all your endpoints, for example, or the client is written. C is sensitive.

A

So how do you need to load balanced? Looking at the client characteristics? Also, you need to look at endpoint or the backend requirement, whether it needs isolation or protection of certain format, which, for most large services, is true and also if there is a lot of connections from clients. How do you really make sure that only a portion of clients connect to them and don't overwhelm the endpoints?

A

So, let's briefly, look at client side load balancing in G RPC in G RPC. We have to load balancing algorithms, which are round-robin and pick first, which basically pick a set of backends or a single back-end based on the discovery or the resolution of that endpoints. So, based on the load balancing policy, the client can decide to pick one endpoint and just use that endpoint for all its requests or round-robin over all the endpoints that it has received optionally. The client can also connect to a sub set of endpoints.

A

We don't have it in G RPC, but I. This is one of the aspects that can be completely controlled on the client side. So at the client side you have certain control over what endpoints you can select and how you can select it, and that can basically help you to limit the number of connections a client can make and can be used to enable full of client behavior that we've seen previously, but just making decisions on client is not really helpful in the bunch of scenarios.

A

For example, if you do decide to use pick first now in in this particular case, if because all the clients are taking decisions independently, a bunch of clients or maybe all of them end up connecting to a single endpoint which can overwhelm those endpoints and bring those endpoint down, this can eventually cascade to bring all the endpoints down. No that's. Definitely not desirable also round-robin is not really great.

A

If a single client has a lot of requests to make, and maybe it is buggy or maybe it's it's- a attack eventually bring out bring down your whole service. So just making decisions on the client side is not enough. You need more. So this is where a centralized load balancing really helps centralize load balancing can essentially determine what endpoints offline can connect to now.

A

This is more than just the discovery that we've seen previously for client lb and in this, the load balancer actually knows about what clients are available and can tell each of the clients what endpoints to connect to and the clients can eventually take their own local decisions.

A

To connect to those endpoints in this particular end pause in this particular example, you can see that the load balancer actually ended up, giving even number endpoints to one of the clients and odd number to the client to so it can help provide isolation to those backends and protect them in case one of the clients really starts overwhelming the service and still allow the client there are that line to continue to serve.

A

Now, let's look at example of Sandler's load, balancing I'm sure most of you have already looked at G, RPC or GCL B, now, GRP CLP. The way it is right now is a protocol to provide endpoints to the client where from centralized, ERP clb server. Now this is called look at a leukocyte load balancing, because this is purely a control path, load balancing where your actual connection from client to endpoint is directly between them and there is no control apart aspects of this.

A

There is no hop and we'll see another example after this, which is the proxy load balancing which has a hop in between which is slightly different, but in gr PC, the GRP CLP essentially picks at the endpoints that is sent to the client right now. The default for client with GI PCL B is the round robin and the balancer essentially provides a weighted endpoints to the clients in the form of repeated lists and we'll look at a little bit more in terms of how those weighted end points impact the clients and why they are needed.

A

So J, RPC lb is the first example of your centralized load. Balancing another example that is commonly used is proxy, based load balancing for this presentation. I am a purely going to talk about envoy as a proxy and I know. There are a lot of other load. Balancing proxy is available, but we are talking about on why?

A

Because it is one of the most popular ones, for this presentation will only worry about the load, balancing aspect for Envoy and not of other capabilities that our proxy provides, and you might want to use it for those reasons, but from a load balancing point of view and on why proxy connects to management server.

A

In this particular case, I have shown you an example of Ischia pilot which serves an ax TS load, balancing API for Envoy, so the on OA connects to management server to get the endpoints that it needs to talk to and and then uses that to route the request to the right, endpoints the client over here.

A

In this case, the GI PC client doesn't really know about all these endpoints and dip, and it basically the the connection, gets terminated at or at the Envoy proxy and subsequently, the on web proxy does the load balancing for those endpoints.

A

The Envoy itself can be deployed in middle proxy or a side proxy mode, and in this particular example, we are using a sidecar. It's again one of the most popular methods of deployment.

A

Now we've talked about how a centralized load balancer can allow you to to provide isolation such that your servers are protected, but that's not enough. There are more deployment considerations for a large service. One of it is your endpoints themselves for heterogeneous and they can be heterogeneous and various aspects like their capacity. We'll talk about what capacity means a little bit later in when we talk about it in more detail. Also in terms of location of endpoint, could be in geographically different location than a client or endpoints can be globally distributed.

A

But they're headed in heterogeneous to the point where a request made from a client we'll have a different behavior when we do when you reach out to different end points also end points. Health and capacity can change over time. For various reasons, you are abrading the your service or you are there's a hardware failure when you are actually dealing with physical machines which can change the capacity over time. There are other aspects too, like request could not mean not. Every request is the same and the search, and that could impact your service capacity and availability.

A

So, looking at just the endpoints and making decision without really taking endpoint information into consideration is not really great, and that is where we mean we need to start thinking about, centralized load, balancing having to get those information to make better decisions. So what we're essentially trying to say is for inform load balancing. We want the back ends or end points to start reporting, information to load balancers. Now what are this kind of information that you would need? We talked about capacity. What is capacity capacity is essentially a certain specific and can be various aspects.

A

It could be number of our pcs that your particular service scan or the endpoint and serve, and that can differ based on how big your endpoint is in in compute or in memory. It can also differ. It can also be something like like your utilization of and aspect, and one of the good example of this is the G RPC load reported protocol, which allows you to end report. All of this back to the balancer and I believe open source implementation, for this exists, at least in C++, I'm sure about other languages.

A

But this is something that you can use to provide more information to the balancer, such that the load balancer can make better decisions based on endpoint capacity and do and provide those to your your clients. We saw in G RPC example that the GRP CL b just gives you a list of endpoints that clients like round-robin over using your capacity, the load balancer, can actually give weighted information. It could repeat some servers indicating to the clients that you can do more our pcs on tho. Then it is, as it has more capacity.

A

One other important information that a load balancer can use to a centralized node can use is the health of an ED point by that the end point is up or not so, instead of giving an end point which essentially doesn't serve anything, it could really give end point which is serving right now, and these are more timely information. That's allowing the clients to be able to do their requests and get a successful response.

A

Second aspect of informed load balancing is locality. Now what is the locality? I talked about log, a large services where clients are distributed all over the world for for the behavior of the service that you expect. You would want those endpoints to be also distributed across the globe, which means that is an advantage for your clients to reach out to endpoints, which are more local or close that geographically to them. But to have to actually do that.

A

You need a construct to allow the load balancer to inform the clients that these are the servers that you need to connect to. So I've shown an example of couple of locations over here. So the construct of what we're and endpoint resides needs to be made available to the load balancer and also needs to be made available to the client in some format.

A

Either it can be encoded as part of the protocol like in G RPC lb, or it could be done as part of a protocol which XPS does wherein it provides locality as an explicit member in the protocol which, where the client can take more informed decisions about where it wants to go, and the advantage of this is the client can take advantage of spatial locality and have Layton sees ready latencies for the request.

A

This kind of locality is different from the temporal locality associated with pic server that we've seen before, where a client would want to reach to the same server again and again, also the other advantage that a locality gives you is. It can give you an abstract concept of locality, capacity capacity that can be used for smarter balancing at the load balancer. For example, a bunch of servers capacities can be aggregated for a particular location, and you can say a particular location has a certain capacity.

A

This allows the load balancer to basically balance based on those capacities and we'll see how this becomes more relevant when we move on to the next slide. So we talked about a bunch of concepts till now about what is required for load balancing at scale, how to make centralized load balancing have more information such that it can make do smarter decisions now, assuming we do all of this. What what? What does the real architecture? Look like this?

A

The mocked up load balancer, but it tries to illustrate what exactly we've talked till now we have you, have multiple locations a to Z and you have a bunch of services all are reporting, they are load back to the load balancer and the load. Balancer is basically making global decisions for each of these services and, at the same time, sending those information to the client where it's telling client what to do based on those capacities, weights, localities, etc, trying to serve it closer to the clients. All this calculation will take.

A

A lot will take some time because it and it's purely depending on your scale and number of services, but is this the right way to do? Maybe not maybe there's something additional that we could do, which is. We could divide the responsibility between the centralized global load balancer and the client. We could have the centralized load balancer take some of these decisions, while the clients can take more decisions more locally, based on the information that they have. The global decisions like what's the capacity of a particular locality and where is for clients in particular locality?

A

Where should they directly go to those can be? Decisions made at those decisions can be made globally at the load balancer, while the clients can take more local decisions of which servers to pick and which localities to pick be in case certain localities fail now. This is this is not really possible in G RPC lb protocol PCL, but whatever doesn't really have the information of locality encoded in it, but this is something which is all which already exists in XDS protocol and is already available with in the form of locality and endpoints.

A

Now, let's look at service mesh and load balancing and how it happens. This is a small example of what it would look like for two services service, one and service to service. One is deployed in two locations: USA, central and Asia South and service to is deployed in Asia, south and USA West service. Two is a client which wants to connect to service one and based on locality, all the server's, all the server's.

A

All the services, sorry are sending load reports to sto pilot and it's making load balancing decisions and giving the information to the sites are proxies for load balancing.

A

So whenever a service from whenever the client from USA West wants to connect to a service s 1, it is redirected to its a server which is closer to it, which is in USA central, while the servers from s 2 are redirected to Asia South, except for one server, because the capacity in asia south are indicated by the number of servers which is just 2 is not enough to serve those requests and for that particular which in our case is server s, 2 3, it is redirected to us s central.

A

So this way we achieve, we can achieve multiple level of load, balancing where clients can can locally go to closer servers, while globally, the sto pilot or a management server. Can basically redirect them to a different locality? Now we've talked about G RPC services and we've looked at service mesh. Now this presentation and service mesh. So how do they come together? The same infrastructure that we've talked about for service mesh using you using the sidecar proxy can also be and used to do.

A

G, RPC, load, balancing and we've looked at how X J's provides various constructs for load, balancing flexibility, including locality and proximity information. These information can be used to do G, RPC load balancing using using the sidecar, and we look at that in our demo, which we are going to do next. So this is my demo setup in my demo. Setup I have something similar to what we have seen before.

A

The only difference being only the clients are the only ones that are using the sidecar, while the endpoints are do not are not using sidecar and the only information that the end point is really sending. Is the health information associated with that end point I am NOT configured capacity for this demo and the health is the only information that is going to be used for global load. Balancing now, in this particular case, similar to my previous service, mesh architecture or deployment, I have a greater service and a greater client.

A

The greater services actually deployed in USA Central and South Asia, each with two instances of service running and this deployment is set up in GC with traffic director. As the management server. We can use East EO as a as the management server. True traffic director was just easier for me to configure for this particular demo and it's easier for for me to write up our demo script, which also made available and is present in the last slide. So but it's the same X J's API, which is to your users.

A

So for this particular case, I have greater service with two endpoints running each in USA, Central and South. Asia and I have greater client1 running in USA vest and one running in South, South Asia. So by default, whenever the demo starts, you'll see that the greater client basically reaches out to read a service in USA, Central and greater client will reach out to service the endpoints running in South Asia. So let's look at the demo.

A

So these are my machine names. They are slightly different from what is it in the? What is them the demo from a configuration point of view they are managed. Instead, they are, each of them are all of these are under one particular service and I'll show.

A

This is not the right version.

A

Yeah so I have I have this one service, which is configured which is right now shown as healthy, because it has all the end points associated with it as healthy and I have two clients which just open another console for us.

A

So I have two clients, one in Asia and US West. What I will show? You is. Let me first start the clients on each of my notes. So all the server does is respond with its name, which has its continent, which region it is from and its actual hostname.

A

On the on the client side, outside of making the actual request, I have a HTTP server running which basically displays what the last requests last 50 requests where the last 50 requests have gone. I have two of them. As you can see, this is us West it it's for the last 50 requests have all been in US and for Asia South. All last 50 requests have all been in Asia. I need to refresh this, because this is a pretty basic HTML page that I could cook up based on my HTML expertise.

A

So, as you can see, they are all last 50 requests are equally divided on two nodes. Now, let's bring one of the server down and see how the how the client react to them.

A

So I've stopped the service on one of the nodes in Ashiya south and we'll see how it impacts the requests that are being made.

A

So ideally, in this particular case, once the service goes down, the management server over here detects that it is unhealthy and it'll start to redirect some of the traffic to a different node.

A

So it's already started to do it on u.s. central.

A

Ok, I think I have brought down both of them. So let me so I in in my frenzy of typing I ended up dropping down bringing down servers on both the continents. So let me write again so in this particular case, because one node went down on each of the continents. You see that both the clients, the uswest client and the US and the Asia South line and end up making requests or across the continents. You would also see at least in this dashboard right now.

A

It will show only one as unhealthy, because I have bought the other server up. So it's unhealthy, so you can see that the one servers have been healthy. Now, let me start the G RPC server again on on US central that should this should change the way the requests start.

A

Noting ok, that has started routing again to that particular node. So you see as soon as the node came up basically ended up informing the client of the GRP decline to start moving traffic back into u.s. central. This is for the sorry us yeah, u.s. central, so the US clients are not all being served in one region, while the Asia plants still has one note down. So it's moving. Traffic from Asia are to us.

A

So a bunch of these demo scripts are available in the github repo. You can all try this yourself. It just took seven steps essentially to get the setup up and running. So if you really want to figure out, try out for yourself how to do this global load, balancing and trigger behavior and get hands-on experience on doing it, you can use those scripts. The server that I have is pretty basic. It's great a server. It's one of the most common examples of go server available.

A

You could build your own server and replace that and it'll still work the same. You could try it with more complicated setup, as I mentioned before the G RPC specific load, balancing presentation in the examples. If you want to take a look at it, are present in the second link given below and yeah, you could try those out too. So that's all I had today. If you have any questions around things that we have talked, you can ask them.

B

Hi question about long live connections with dear PC, so a load balancing works well on the initial connection. But if you have a long lived, how do you handle rebalancing yeah.

A

So a long live connections are definitely a problem when you are thinking about capacity for a particular server. So for long link connection, unfortunately, the most the the biggest decision maker is going to be the centralized load balancer, because the client cannot independently make a decision. In this particular case, your capacity cannot be derived based on your RPC request, but in essentially how much you resources are utilized at your server. So if you can plug that into your load, balancer you could definitely send.

A

You can definitely use that information to set the load balancing weights in your Exodus protocol to make sure the next set of requests are coming from are going to different servers. So that's definitely possible it's a little more trickier than just single are pcs.

B

Any more questions.

B

Going once, if I'm, if I'm not seeing you wave louder, no okay, thank you very much. Then visual.