Cloud Native Computing Foundation EnvoyCon North America 2021, 30 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Speeding up Istio: Our Journey Implementing Delta xDS - Aditya S Prerepa, Tetrate & John Howard

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Speeding up Istio: Our Journey Implementing Delta xDS - Aditya S Prerepa, Tetrate & John Howard, Google

xDS is the way config is distributed to Envoy. Most of the ways the xDS API is implemented today (and in Istio) is through the state-of-the-world design. If one out of a thousand clusters changes and Envoy needs to know about the change, most control planes (including Istio) will send all of the thousand clusters to reflect the configuration change. On top of that, every configuration type is converged into one stream with Istio (ADS), which does not do the network any favors. This is the “quick and dirty” way, when logically, there should be no reason to send configuration when it hasn’t changed. This is what the delta xDS API aims to solve. Delta (or incremental) xDS is a variant of ADS/xDS, which has a different interface. If one configuration changes, that is the only configuration that will be sent. Istio is having quite a journey implementing delta xDS, sending only the “deltas” in configuration changes. Especially in a service mesh like Istio, which is the largest and most in-use mesh, there are quite a few caveats that need to be covered. This talk will be about the journey of Istio in implementing delta xDS, along with the expected benefits & apparent struggles that we had, along with guidance for future implementers of this amazing API.

A

Cool so uh hello, I'm adithya, I work at tetrate and I'm also a senior in high school and.

B

Hey I'm john howard, I'm a software engineer at google. I've been working on the east joe control plane, which is a non-void control plan for about three years now,.

C

A

Okay, so in this talk we'll cover basically um uh delta xds. So what istio is?

A

First of all, um we're also going to cover what we've done to our control plane and why that wasn't sufficient for uh scaling it up uh we're going to talk about the solution that we think is good, is delta xts and how that helps with the scaling we're going to talk about how we implemented it like the specific details and the problems and solutions we faced um with implementing delta xcs and we're also going to talk about the future of delta xds within istio.

A

So that's a lot to cover okay. So, first of all we have to understand what is istio, I'm sure most of you know, but it is a service mesh that uses envoy as its backbone, so it uses envoy as a sidecar proxy and also as a gateway.

A

We can see that configuration is distributed to the proxies via the control plane over the network. There's a component in the istio control plane called pilot, which does all the configuration management. So that's how configuration is sent to all the proxies and we'll talk about that as well.

A

So then the other big thing is what is xds so xds is essentially. um It stands for, like this wild card discovery service, where, um like you, can have it's a way to like propagate configuration to envoys uh dynamically, uh whereas the alternative would be statically like hot, restarting envoy, with the configuration file it's either implemented over the file system, so you'd set up a file file system subscription or you can do over the network, it's usually the case with the rest or grpc, and it's usually a grpc bi-directional stream.

A

So that's what xds is.

B

So over the years, as ego has become more and more popular, we've seen that users are also deploying it in larger and larger environments and we've kind of had a cat and mouse game of catching up as people are scaling up.

B

So in the early days of istio, if you were to deploy, you know, even a moderate size, mesh with, say, 100 workloads, control plane would often times be using multiple cpus just to service that, whereas our users are trying to deploy in environments with you know, hundreds of thousands of pods and some of the larger cases in many cases, because we support multi-cluster as well, we're not even bound by the size limits imposed by kubernetes, and so there's really no bound on the scale that our customers may require.

B

So over the years, we've invested a ton of effort into optimizing the control plane. We've done a variety of things. To do this, a lot of work we've done is just in micro, optimizations of our code. You know reducing allocations where we can splitting up a loop where we, where it's able uh one of the big changes was. You know some changes to how we encode our protobufs, because all this configuration is sent over grpc. So a lot of time is spent on marshalling and unmarshaling those uh protobuf objects.

B

Some other things we've done are try and make our control plane smarter.

B

So when a configuration update comes in, say, like a kubernetes service changed, we would typically just send recompute all the xds configuration and send it to all the proxies a lot of times. That's pretty wasteful there's a lot of updates that can happen to an object that don't actually change the configuration and so we've added some. You know logic to skip pushes when they're not necessary, which is also made a huge improvement.

B

We've also done some other things. We've changed the actual configuration we send to make it smaller, not for behavioral changes in envoy, but just to improve the performance.

B

uh We've started doing caching of pre-marshalled xds responses, so we don't have to go through the full generate marshall and send we can just skip to the send, and we even started doing some more segmentation of configurations that the proxies don't see all the configuration they see. Only a smaller subset, which is, of course, more efficient to optimize or to send so, despite all that, we still haven't, really met our performance goals.

B

um You know, while on a larger size clusters with, say a thousand pods, uh it's usually no problem for istio, as we start to get into the mega clusters, with hundreds of thousands of endpoints we're still having issues with scalability of the control plane.

B

uh Fundamentally, what that comes down to is just the math of these of the scaling. So if we look at what impacts the load of the control plane, there's really three main factors.

B

The first is the number of workloads so for every workload connected to the mesh, we need to generate configuration for it and keep that up to date.

B

The next one is the resource size, and this is actually impacted directly by the amount of workloads, because the configuration contains information about how to connect to each other workload, such as all the list of all the other p addresses, but it's actually quite a bit more information than just ip addresses, so this adds up quite a bit.

B

Finally, the third factor is the rate of updates. This also is typically tied to the number of workloads just by laws of random chance that, as you have more and more workloads, it's more likely that one of them is going to scale up scale down, be restarted for an update, et cetera.

B

So putting all that together, we kind of have a polynomial scaling problem, which is obviously not good so to put that into perspective on a a cluster with say, a thousand pods. For example, if we had a one megabyte resource size, which is something that is not unheard of in our our control plan implementation, anytime, a pod changes, we would need to generate and send over the wire a gigabyte of data.

B

That is not just a gigabyte of garbage data. Either we need to go generate these protobufs and then we encode them. So there's a lot of cpu spent doing that as then. We also have to garbage collect all that gigabyte of data and send it over the wire.

B

In the case of a rolling update, say we're. You know rolling out a new update to all 1000 pods. It can add up to a terabyte of data that we're sending over the wire throughout the rollout. That is quite a bit of data.

B

You know we're not a cdn, that's designed to be serving a high amount of data like this, and so this is imposing pretty serious issues.

B

So what we kind of realized was that, no matter how fast we make the control plane, even if we really made it instantaneous to generate the configuration you know with the scaling problem like this, it doesn't really matter. We can't keep sending like a terabyte of data over the wire when there's a rolling update, otherwise we're just not going to be able to scale to where we need to be.

A

Yeah so I'll talk about the.

D

A

Type of xds we run on and so configuration is distributed through xds, so the way we do it right now is to the state of the world. um Essentially, what this means is that for every single proxy that cares, we have to send every single resource that it needs to run every single time, so that can be quite a bit of data as john mentioned.

A

So if, in this example, you have three clusters only cluster two changes, you still have to send all three clusters. This is a really trivial example. Usually it's a lot more clusters, um so the control, plane and envoy need to process these configs, so it's kind of like both of them need to process it, even if only like a few of them are changing, and this can cause combinatorial explosions during these updates.

A

uh Trigging triggering data by gigabytes of data, even if only one service is changing, so that's state of the world and that's where we are right now with istio. um But what we're working on is delta xds, which is the solution we think uh so. This is also incremental xds, and this allows us to only send the resources that changed um to envoy. So in this example, if only if only cluster two changes are the three clusters, you only have to send cluster two.

A

uh This seems like the intuitive way to do it like it's just. If only that cluster changes, you should only be able to send that one. uh We implemented it with state of the world and a lot of control planes do because it's easy and b it's reliable.

A

So that's why we did it that way, but now we're trying to move it to delta xds, and so again this dramatically decreases the configuration we have to send over the wire and then we'll go over some benchmarks as well, um but yeah. I just want to go over like some of the key differences in the request response, rendezvous between the control, plane and envoy, so with the state of the world um you use, but you can specify a bunch of resource names that you want um envoy, usually we'll just say.

A

I want all the clusters and it'll just say that that will mean that I want a subscription to every single cluster that I care about, and the response will contain all these clusters and, like john said, this is a lot of data, but with delta.

A

It's basically the same for the request. It'll just subscribe to all the clusters, but with the response like it'll, only send what changed so in the resources field, whereas previously you'd have all the resources. Now you only have like if anything's added and if any resources were removed, uh then you can send those as well. The names of those resources and envoy will remove it internally.

A

So yeah, that's the way. The request response works for these two methods, enabling delta xds within envoy, is really simple. You can see it's just an api configuration change from grpc to delta grpc, so it's really simple. You just put that in your envelope configuration to start envoy and now it starts listening or looking for the delta api um go control plane, which is a control plane that I think one of the uh speakers talked about uh support cell to xds. Out of the box and we'll talk about how that happens, um but yeah.

A

So if you're using go control plane, uh you can just enable delta, xds and you're good. uh No additional configuration needed um yeah. So now we'll go over the go control, plane, implementation um and this is important to understand the contrast with istio's implementation.

A

So the way it works with go control plane is that, like users of the go control, plane, library will provide a snapshot to um to go control plane and that snapshot will be served to the proxies, um and so this snapshot consists of all the resources that the proxies need to care about.

A

And the key point here is that the computation of the snapshot and the serving of the snapshot are completely decoupled.

A

So it's not that, like one triggers the other, the snapshot generation happens and then whatever the product, whenever the proxies need the data, it is served to them, um and so this makes it really easy to either do state of the world or delta xds, because um if you want state of the world just send the snapshot, and if you want delta xds, you can store the old snapshot and then, when you get a new snapshot, you just diff them. You get the deltas and then that'll be the deltas that the proxies care about.

A

um So that's the way go control plane. Does it so incoming istio? uh So we don't store any snapshots in memory at all ever um because it'd just be too many. um If we were to do that, we'd have to store a different snapshot for every single proxy, whereas with go control plane, it's just one snapshot for all the proxies, and so this would lead to like what gigabytes of in-memory data, and that's relying on that. So we can't really do that. Instead, we do configuration generation on the fly.

A

So what this means is that whenever we get a request uh from a proxy saying, hey, I want this resource type or something we will generate that for them on demand and then send it back to the proxy on a per proxy basis.

A

So the re, so the response, like the resources, will be different for every single uh proxy uh because their visibility is different, um and so basically uh because we can't store the snapshots of these resources in memory, we have defined a different way of doing delta, xds and we'll go over the way we do that but yeah. So the key point with istio is that the config generation and serving is completely coupled. So it's when you get a request, then config is generated and then it's served. It's not like it's stored somewhere, okay, um okay!

A

So our initial implementation uh for the delta api was just literally taking the state of the world generation and sticking it on to the delta api.

A

So if you used istio, 110 and enabled delta xds, I'm sorry, you didn't get any performance improvements because well we didn't send any deltas. It was just state of the role generation over the delta api, and this was just in an effort to get the core infrastructure in place for delta xcs, without rewriting the entire configuration generation pipeline so yeah. So here you can see that even if one of the clusters changes on the it'll be on the delta api, all three clusters will still be sent for the initial prototype.

A

John we'll talk about problem in solution.

B

Yeah so once we implemented the initial delta xds, you know we were all very happy, pat ourselves on the back and then quickly realized.

B

There was a lot of issues with it, so the implementation it passed all of our tests, which are reasonably robust, and so we had a lot of confidence in it, but as we started to go into actually optimizing delta xds, rather than just kind of the state of the world over delta xds, we found all sorts of issues, some of them very obvious, and we were very confused because everything seemed to work and what we realized is that there's a lot of ways that you can have a control plane that is totally broken, but also works correctly in envoy.

B

So, for example, we found a bug where, in every single response for eds, we were telling onward to remove every endpoint. Obviously that seems horrible, but it turns out that envoy actually ignores the removed resources for endpoints, which is because it has a parent resource. The cluster another example is that when we we occasionally fail to remove a resource resource at all and that's generally fine we're not really bothered by extra clusters in general, uh but it does leak a lot of memory and wastes a ton of resources in both envoy and the control plane.

B

And so at this point you know, obviously we fixed these bugs, but we were worried about the general problem. How do we ensure that our delta xds implementation is as robust as our old one, and so what we came up with was sort of this dry run mode.

B

So what we would do is that for each request we would run both the old state of the world code, which we trusted was fairly robust and the new delta xgs code, and so what we wanted to do was compare those two to see whether the delta xgs response was accurate.

B

In order to do that, we started storing the previous state, and this is when we're in like debug mode only so normally this would be far too expensive, but with the previous state of the state of the world response and the new state, we can then kind of compute the optimal delta xjs response, and then we can see how we did with our actual delta x jazz response.

B

So this helps us detect bugs such as failing to delete the resource that we should have if we deleted a resource that we shouldn't have or maybe, if we fail to update, uh failed to send an update that was required. It can also help us optimize our solution, because we can see if we sent an update over delta xds that wasn't really required, because the configuration didn't change at all.

B

Another issue we saw was with big resources, so we had kind of naively hoped that delta xds would be the silver bullet which wasn't quite the case out of the box. So with delta, you get the granularity of a single resource and if your resources are giant, it doesn't really help that you can only send one at a time.

B

You know, for example, at a ingress gateway. It's very common to have many many different virtual hosts. You know over the same port, and so if we still have to send that entire set of virtual hosts every time one changes we haven't got much benefit out of delta xds.

B

So one of the solutions that have been has been kind of actively pursued in envoy and it's still ongoing is kind of decomposing. The resource types so far, there's been three types that have kind of undergone this one is the virtual host discovery service, which is splitting up the routes by virtual host. That's kind of the example I showed before another one. Is the locality endpoint discovery which is splitting at the endpoints, so you can send smaller chunks of endpoints.

B

This is really helpful if a service has a ton of endpoints that you don't want to send on every update, there's also the extension config discovery service for sending individual filters, and so what this allows us to do is to break up our components more and more so that we can send smaller increments. uh You know rather than the full thing.

B

Finally, we have an issue where you know: if we have a giant configuration, then it's it's slow at startup, and this can be problematic, especially in cases like serverless, where startup speed, directly impacts user facing latency and so with delta xds. When we start up, we still have to send all the configuration it's only on the updates, where things get quite a bit better.

B

So one of the solutions that are being pursued on the envoy side is this concept called on-demand xds and so, rather than send envoy the full configuration we can send it a very small amount of configuration that tells it to just lazy load or, just in time load the configuration it needs.

B

So, for example, when a client requests example.com, if envoy, doesn't know about that virtual host, it can ask the control plane for the route the ro or the control plane can send it back and then envoy can serve the request. So that allows us to have this rapid startup speed and still get the benefits of. You know the vast configuration that we want to send to envoy.

A

Okay, yeah, so now I'm going to talk about the implementation with an istio, so our on so the way we use envoy is the envoy. Configuration is dependent on a bunch of kubernetes objects and so based on. Whenever these kubernetes objects change, uh we can determine what envoy resources change and then determine what should be pushed basically to every proxy.

A

So, for instance, if just a service changes with a certain hostname, we have to find all the clusters with that host name and then, once we find those clusters, we have to push those clusters to the proxies that care. uh So this is essentially the same as sending the deltas or like diffing like go control plane does because we're getting updates so yeah. This is the example.

A

So if you like service, a and that'll correlate to a few clusters um uh with the same host, stamps on proxy.I o will have same those same hostnames as long as they match. We'll push those clusters we'll generate and push those clusters, as opposed to generating every single cluster ever um so it's much much better. um So yeah now I'll go over like the push flow.

A

um So, for we've only implemented delta xds with the cluster discovery, service or cds, um and this is because we saw the biggest impact in cds without like the most implementation. um So here's the push flow for cds, so essentially um the kubernetes in former api.

A

We use that to figure out when these objects change, and so when a certain service changes uh we'll get notified of it through the informal api, and so this will lead our control plane to generate configuration data and that configuration data is just the deltas because we got the update as like a service was updated.

A

Then we know that whatever comes from that service, whatever clusters are associated, those are only the changes of clusters as well. So we then gather those clusters, and so we know those are the deltas and then we'll push those selected clusters um to those selected envoys um that care about those clusters.

A

um The same works for deletions if a service is deleted, like basically we'll just compute like what clusters are still associated with that service and then we'll just tell envoy, hey delete these clusters um so yeah, that's the push flow for the cluster discovery service.

A

It's pretty simple but, as we add more and more like services, we're working on destination rules, and so, as we add more of these objects, it should get more complicated.

A

So the fun part performance results um so john created this benchmark where we scaled a mesh from zero to a thousand pods, and then we updated a service every second changing something arbitrary and essentially uh this is a graph of network throughput with the time and the area under the curve. um From like any given point, is the data being sent at those times so which state of the world, which is the first one like we can see just kind of like almost constant huge wall of configuration being sent? That's a lot of configuration.

A

I think if you found the area under the curve, it'd be a non-trivial amount, and uh so that's the state of the world, and we can see that it remains relatively constant. um So it's a lot of data being sent and then we can see after like round 16, maybe 13 we can see the delta benchmark so essentially the area under the curve. There is also the amount of data being sent and it's significantly less there's that initial bump that we see john talked about it. It's a slow bootstrap.

A

We have to send that um because onboard needs to know about itself and other things as well. um uh The solution to this is just uh on demand xds. So if we do that, then there'll be no bump there and just be like flat, um but essentially um after that initial slow bootstrap, you see it kind of trails off to almost nothing, but that almost nothing that, like a sliver of data, is actually the amount of data being changed and that needs to be sent to the envoys.

A

So the difference between delta and state of the world is the amount of redundant data, that's being sent across your network from the control plane to envoy. um So this is the benchmark and it shows that delta is significantly better performance-wise.

A

um So once this is implemented like mesh wide for a bunch of resources, there should be quite a bit of performance improvements like a noticeable performance improvement um in your mesh, so use delta, xds yeah. Thank you uh any questions or no okay. Yeah sure.

C

C

uh So you talked about enabling this through the go control plane. But could you explain how exactly how to enable this in istio, you said it's not enabled in 110 should we expect it to be fully enabled in 111? Oh definitely.

A

Not uh it's nowhere near where it needs to be, uh but the way you enable it is just by setting an environment variable. I think it's istio delta xds, true on the agent and the control plane and then that'll work. um But it's nowhere near stable, like uh don't, don't get excited about it being inflated by default.

A

uh The issue with deltex or not the issue, but the inevitability with delta xds is that um you need to get it right, because if you don't get it right, you're going to send incorrect updates, and if you do that. Well, your message out of sync and that's bad.

A

So we have to get it right for it to be on by default, but yeah any other questions.

C

C

B

um Yeah in the protocol itself, I don't think there's any sort of reconciliation. um Oh sorry, I'm supposed to repeat the question.

B

The question was: is there any sort of reconciliation loop in between envoy and the control plane to make sure that it doesn't fall out of sync at some point if we miss an update so like I was saying, there's no specific, like reconcile aspect of the protocol itself, but the control plane does have the ability to push whatever configuration at once, and so you could reasonably say every hour we're going to compute all the configuration uh you know like the old state of the world code, send it over to envoy just in case we, you know, made some mistakes in the past.

B

We're not currently doing that, but yeah. You know it's definitely possible.

B

I do think that if we, you know it's a good fallback plan, but if we, if you're in that state where you need that, then you may have some serious issues, because uh you know if you miss a security update or you know some other critical update. Even if you have the loop an hour later, it may be too late.

B

D

Numbers about cpus and memory usage: do you have performance number about cpu and memory usage? Not the bandwidth.

B

um We don't have a graph to show you today, but we have done some some analysis and it is. It is quite a bit better. um You know, generally, the eastio control plane is spending 95 percent of its time on this code. um You know generating the xds configuration especially marshalling it, and so we expect as we're doing far less of that, that it will go down substantially.

B

One thing we're also pretty interested in is the cpu and memory usage at the envoy side in processing xds, which is oftentimes a bit overlooked, because it is relatively small for a single envoy. But if you have 100 000 envoys, then even a small increase makes a big difference. So we haven't measured that yet, but we are excited to see the improvements there as we as we progress.

D

Okay, thank you.