Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2019 (San Diego), 22 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Fine Grained Mesh Metrics for Better Visibility With Native Performance - Mandar Jog & Kuat Yessenov

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Fine Grained Mesh Metrics for Better Visibility With Native Performance - Mandar Jog & Kuat Yessenov, Google

Knowing granular traffic patterns is crucial in understanding the functioning and health of a service mesh. The existing Envoy extensions collecting metrics are either not granular enough or can consume too many resources. In this session, attendees will learn about an efficient way of producing granular metrics. This method introduces a new metadata exchange protocol between peer workloads and uses the new Envoy/WASM metrics API to produce richly dimensioned metrics based on the exchanged metadata. The operator can add arbitrary peer dimensions like availability zones and locality to get additional insights into the traffic. Istio will use this technique to efficiently produce highly dimensioned Istio standard telemetry. Attendees will see a demo of rich telemetry collection to Prometheus at near native performance.

https://sched.co/UaYV

A

So my name is Mandar I work on sto at Google and if you're here for fine-grained mesh telemetry, then you're in the right place. Otherwise that is still what you're gonna hear hi.

B

My name is quad I also work at Google on Easter and.

A

The 3rd one who worked on this she job she couldn't come so so before we start, since this is about service mesh. We're just gonna take care of some preliminaries.

A

Why service mesh, I think that is probably covered a bit by now, but I will still cover it a little bit. Why do we need granular telemetry? That's another thing: we have to cover what is available without the work that we did and then what we have added. So we will kind of go through that.

A

So now consider the graph on the right that and then consider that it represents just a monolith.

A

Every node in the monolith is a module and every edge in the graph is a function call now when we decompose the monolith and we go to micro-services the easiest way to just draw a border around a module and say: ok, that's that's my service, and now the picture still looks the same except those all those edges now are going to be our pcs they're going to go over the network and then that's when we move to kubernetes, for example.

A

So essentially, what we have done is we have moved from something that was very fast, very reliable and very secure to something: that's not as fast, not as reliable and not as secure an in-memory function. Call simply does not have that many modes of failure right you make the call and the call goes through not the same with an RPC, so we introduced a whole class of problems that did not exist before and and of course, we know why.

A

Micro servers are good, I'm, excluding that part from here, but as far as the call is concerned, we actually did take a few steps back, so what service mesh tries to do is it tries to restore some of those qualities that we lost when we moved away?

A

Well, when we move towards micro services, there are many service meshes, but my favorite is sto and of course these two is based on envoy and we inject a sidecar and a pilot which is the control plane, your configures Sto, so now we'll just see how service mesh is realized. So, first of all, we start with this simple picture. Just three services: there is product service in the middle shopping service and a shipping service.

A

The direction of the arrows indicate the calls, and these boxes could be parts or workloads, but for this discussion consider that consider that these are parts and now they're just calling just normal, not nothing special here now magic and we inject envoy sidecar proxies in each and every part. We have to make sure that any traffic that that previously flowed directly between the applications is now mediated through the proxy, and there are many ways to inject this proxy there's iptables magic.

A

You can use HTTP proxy variable and there are other ways to as long as you use one of those ways and get the proxy in the middle. We have what we have a place where we can implement functions of a service mesh, so the three pillars are routing security and observability. We're going to talk about observability for for the rest of the talk, a lot about upstream, downstream nomenclature. That I mean that is consistent with envoy, because that's what we use the arrows are pointing upstream.

A

So product/service is upstream of the shopping service and, conversely, shopping services downstream from the product service. This does not so now we need to continuously and dynamically configure the data plane. The proxy is, is the data plane and then we add the actual control plane on top, which is going to dynamic, as you can figure it, it uses the xdist protocol to configure onboard that protocol is defined by envoy. There are, there are many control planes that are available. Sto pilot is, of course, the OSS sto control plane account vendors have their own.

A

So, for example, google has traffic director as our control plane. We are going to talk about sto so for the rest of the talk so kind of the purpose of this talk right. What questions are we trying to answer when we look at a service mesh? So there are certain questions that are at kind of a whole mesh level that sre monitoring the entire mesh may ask, and these questions are of the form who's talking to whom are there any links or are there any services that are more loaded than others?

A

Do I see like many errors somewhere, and it really looks like the service graph that is actually in the picture. So that's the view that you're interested interested in when you go down to a service owners perspective now you're interested in specific activity that happens around your own service. In that case, you do.

A

A

On that cluster, so all those would be client-side cluster sets. On the other side, the server-side stats are captured by listener stats, and these are called downstream stats. In both these cases, we have used the Prometheus format to express the statistics with its Buddhist dimensions.

A

So, let's first consider client side stats so upstream requests total is like one of the standard stats that you can get. It is dimensioned by two quite useful dimensions, or already one of them is cluster name and the other one is response coder in this case, it's the response code class.

A

If you apply a rate function on it and track it over time, you can get rates and then you can know what to expect what what is the norm and because there is the response code class dimension. You also know the error rates and success rates and like for xx. So if the client is doing something wrong versus server is doing something wrong.

A

The other part is quite interesting because that dimension is the cluster name. If the control plane has used a reasonable name for the cluster, then you will actually get very useful stat here right. The cluster name is actually pointing to the destination of this traffic. In this case, it's well in this example is going to product SVC. So, okay, well, it should have said.

A

No, no, that's right. That's right! It is. It is going to progress we see now. If is there like still what is missing? Maybe you want to know if a request ended up in a different availability zone right? How would we add that dimension to the to the client side stand, and it is not a very uncommon situation right in in one cluster, you can have some endpoints from one availability zone, 7 points from another availability it is on and you would still want to know where the traffic ended up.

A

So how would you put that information there? So in this example, we want to have zone equals the u.s. West one a like that that sort of information. If a dimension is a property of the cluster, then it is possible to add it in in the status tip. You can either supply this alternate stack name and you can encode whatever you want in that, or we can just use this cluster name itself to encode.

A

Whatever else you want in order to do that, though, since it is done by the control plane, it has to be a property of the cluster so that the control plane knows what to put there and since Zone in our example, is not a property of the cluster, because one cluster includes n points from multiple zones that this dimension cannot be added by what is available here. So you really need some support from the data plane in in order to do this now. Consider the other question from the server side.

A

Who is calling me so there is a there is an analogous stat on the on the listener site called downstream request total and it is also dimension by response code. So that's just like before, and the other dimension is listener now listener dimension can be useful sometimes, but in this case it's not really adding much value. It uses an IP address. Most services will just have one listener, and it is worth this. So it's not really adding a whole lot of information and then the crucial point.

A

The question that we asked earlier, who is calling me that information is completely missing so, as you can see that the downstream stats are are a bit sparse here so envoy, Anita stats. Are they have some coverage good coverage on the client side with still some missing parts as we saw and less coverage on the server side?

A

So it's very fast. It's purpose-built, but only turns left. So if there are any NASCAR fans here, they would probably get the job, but that's fine. So we want something that can turn left right and still move fast. So just a unrelated to technology question what can top or NASCAR right.

A

All right, yes, form, oh okay, so the wish list that that that we want is we want to have peer, aware metrics, so we can dimension our metrics by pure properties. Like we said, like we saw before the availability zone, it would be a property of the peer, so we do want to be able to dimension our metrics. That way. In this case we have de mentioned it with color equals product, and then we also want to be able to dimension metrics by traffic properties.

A

So headers or like anything else else like that, maybe use open API to classify something put the request here now. Some of these things are actually possible, so you can have like virtual cluster and then, but it is quite tedious. It results in very large configurations, so we need. We need to need something more. So how do we impart peer awareness to this flow right? What are our options?

A

There is actually a very simple kind of obvious option: just send it in a header right, so the simplest option is you when the client makes a call to the server it puts its own metadata in a header. It also puts an ID of itself in the header. So now, when the call goes to the other side, so we will know who called it and whatever metadata you packed and then the server responds with it's its own information.

A

So at the end of the call, both parties know who they each other is, and now you can use this information to dimension metrics, which are whichever way you like. This is a very, very simple protocol to implement like few lines of code. In most cases it is fully stateless, but as with many stateless things, it is not really efficient.

A

One of the things that that you'll see in this protocol. The reason why it's not efficient is because you'll end up exchanging the same information over and over by two parts are communicating they're going to tell each other the same thing over and over, like thousands of times, that's going to add to CPU and latency.

B

So what manner described is an instance of a basic protocol. How to exchange metadata between peers proxies general scheme. Is that every we start every proxy of a small metadata manifest that says which pot it belongs to which namespace what labels on the pot may be availability zone? We never to practice, communicate they send this metadata in addition to the normal traffic, and that way each portion knows what added pew is. So this protocol works for HTTP headers, but it has performance issues, and but we have some ideas: how to improve upon that.

B

First of all, we split the metadata into key in the payload so that any computation that depends the payload, can be cached using the key, including the wire transfer, and second idea is that the cause will control both sides of the communication. We can upgrade the protocol to HTTP 2 and HP 2. Has these nifty features like metadata frames? Is that this these ascend, in addition to normal traffic? But they don't count so that way, you can associate metadata frames with either individual request streams or you can put it into connections.

B

Unfortunately, not every box implements. This is a very recent feature, so it has some downsides. So what about other protocols right? So if you, if you don't just use HTTP what about TCP, so if you're willing to tunnel everything for HTTP using things like HTTP, connect, method or jpc bi-directional streaming, then the technique we just described applies and but what about just? What ICP right?

B

Maybe you don't want to send everything for HTTP, so you need, in this case we resort to sending just raw bytes and the way that this would work is that we put an extension into every proxy that, with every connection over TCP it before we send all the standard payload. We just prefix the metadata as a first bytes on the string.

B

The prefix is length we fix so first thing is a header, and that way works for both sides unfortunately has an issue of course, because both proxy have to be expecting this incoming so that they can disambiguate.

B

So that has a. How do we make sure that proxies agree on this? So one idea is that we can make the control plane. Just tell the tell the proxies that the other side is an invoice ID card. So that way you know each box, you would know which commit which connections are to invoice, which ones are not.

B

Unfortunately, that is it's complicate, because you need to coordinate now between the magic injection and the control plane, and also you have to implement this for every single control plane out there, and we also would like to keep all the choices of control planes here, and the second idea is that in a service mesh like East you'll, all the connections already wrapped in two jealous, because we have angeles automatically enabled so we can piggyback on that and the way it would work is that as part of els.

B

The first message you see on chalice is called client hello. It has a field called LPM, which is an extension to TLS and it's basically application layer protocol negotiation. If you it's a list of strings and we can insert a special string, they're called vco2, which is the designated for protocol, then the server replies of its own hello. It also has a list of protocols, and the two proxies must agree on one of them, and that has to be the first one.

B

So if both proxies see that they support this protocol, they will choose it, and then handshakes second succeeds. So in this case we can rely on the Chile's successful as handshake to basically establish is common to them.

B

So we just proceed normal TCP prefix. So let me just step back a little and remind you why we're doing all this so we're doing this work in the context of evolving each to a telemetry. Former architecture where we had a central service called is to a mixer that we telemetry, collector and processor. Every proxies would send to this service, and then we run extensions inside mixer.

B

So there is a stat extension that from which Prometheus scrapes all the telemetry in the cluster Eastern mixer also runs a cache where it pulls metadata from kubernetes, and it joins this metadata with all the leader coming from the proxies so that we can attach all these fine grained, metrics so, and the new architecture that we're trying to evolve is the where we don't have any more issue mixer. So every extension runs directly in the proxy.

B

So that means we have no longer cache, and the way we rely on the metadata exchange is that we can spread this metadata on the cluster, with on without maintaining a central cash in each proxy.

B

We also would like to leverage webassembly so as part of ongoing work is to develop what some support and envoy. So, if you don't know what what that means, it's a it's, not one standard that allows you to run a spread in portable binary format, for running extensions in a safe environment in sandbox.

B

You can compile to us one from many languages, including C++ Tanjung, go rust, whatnot, javascript of course, and so our team has been investing lot of work in building this support in Watson. So we are working in a repo under and we were called an awesome. That's what we're building our sandbox extensions and, as part of this work, was the defining a new field of sandbox API. So what that means is that we would like extensions to be written against this API so that you can run them inside anyway.

B

So the basic architecture now looks like this. You have any way we run an extensions inside any way in process, and we still does all the proxy related work like protocol parsing and all the new tender confuse. But it also calls back to the extensions that are loaded into envoi. The extensions must be written against these C bindings that we supply it's basically a sandbox API. What that allows us is that we can either compile them directly into envoi, which we it's a method which we call non vm.

B

So these are just statically linked libraries in em boy or you can compile to Muslim and then load them dynamically using the ami with the vast dimensions with ample v8. So that allows us to make a choice between whether to compile them in order random dynamically. The the API is very basic, so what it gives you abilities to subscribe to the normal filter? Like events whenever Quest is processed, you can listen on headers Commission on data. You can listen on PCP connections. We also provide a shared attribute context.

B

So it's it's a context for anything requests related that way. Extensions can exchange data between each other. We also allow you to make cycles, so you can call out using your pc or HTTP if you want to, and finally, we expose them native stats and login subsystem for manually. So that'll give you the whole native performance power.

B

So this the way it would work in this model is that we have this metadata exchange extension that implements the protocol it. It reads some stuff from the node metadata in and away that includes peer metadata. It also writes stuff to the context using the data it seasoned on the wire. Then the next extension kicks in and it's a starts extension. It reads the data it was written before and it also reads all the other requests data and it I mean it omits statistics.

B

So the way it would so it's a bit of pseudocode, but it's you get the idea here. It's standard things you're normally doing in matrix way the code you have to create a metric. We have to compute labels for dimensions. You can pull data from the peer. You can find what the pull name is for the appear. You can also pull data about the request and, finally, you call out to the start subsystem to omit the metric and.

B

I feel God just have to be aware of, because this is still work in progress, because we headed the stat subsystem from anyway. We get some of the performance problems that comes of it. For example, it was built from studies, so it means all the labels and linearize linearized the single long string. So whenever you have to scrape it with meteors, it has to be gauge export, and the second thing is stat subsystem.

B

Has this assumption that all the stats are associated with XDS resources, things like listeners and routes, so that extensions must be careful when they try to access things that might potentially be coverage collected, so I'm gonna pass back to? Finally, this is how it actually looks a nice jaw. So the way we do this is by we align and east your config resources called my filter. It's a it's a basically a static patch, so you can push it in between any control, plane and em boy.

B

It just inserts a few filters, and these filters do the actual work and, what's notable here, is this string at the bottom for unban time. This is how you select, which one time you wanna use with either statically compiled in module or or dynamically loadable watson module, depending which one you use.

A

A

It's a lot of this work has actually landed in sto 1.4, which released about two weeks ago. What one or two weeks ago- and these are some of the numbers that that we have so the chart way on the right- is, if you use the mixer, the existing telemetry model, mixer v1 and the the second one the Prometheus was HTTP meta data exchange is what is already in 1.4. As you can see, it's already a significant drop in terms of CPU used.

A

This does not include the CPU that mixer itself used, which was which is actually almost equal to the to the proxy, so that that's a significant CPU reduction, we're still in the process of like some peers are in some. Some are not in the process of fully delivering the TCP protocol that that court described earlier, but we expect to get even better performance with that, because now you're no longer exchanging headers all the time there are still more optimizations that are lined up.

A

So again, we expect to get this rich telemetry, but still be as close to just native based on Y baseline as possible.

A

So the next thing that we usually track and then track often and actually even other people track- is p90 and p99 for first EO. So, as you can see on the right, it's the p99 chart, we always run baseline at the bottom, which means there is no sidecar at all, and then the next line over is just envoy with with nothing else. The line at the top is what was with with mixer and, as you can see, there is a significant drop in this particular experiment. It's ten millisecond drop from what it used to be.

A

So again we have made significant progress in p99, latencies and also p90 latencies, oh and actually I also have a demo. So.

A

Just just going by what happened in the beginning, when I was not Des, Moines I I have some okay.

A

Okay, I have to stand like this now.

A

Well, I think I'm just going to show the image that I've captured before, because I don't think this is this way of navigating is working out.

A

Okay, mirror on display; okay, great, alright. So oh okay! This is the live thing that that's running right now and now I am going to.

A

Remove mixer so for HTTP.

A

And I am going to create the the telemetry v2 filters. Okay, so now that's the that's the steady-state usage it. This includes both the the CPU that's consumed by the proxy and an telemetry, and very shortly we should see both of them starting to go down, while it does that I'm gonna go back to the to the presentation, okay, so the the future direction is, of course, the work to like all this work like quad mentioned before is in another repo in the Envoy organization and the OP streaming. Work has already started so finishing.

A

That is, it is a big part of what the overall work. We also want to add full support for network filters. This is only HTTP. So far. Also we want to add TCP support. We also want to improve language bindings. Everything is converging around open telemetry, so we want to make sure that the api's are aligned with open, telemetry and then kind of align it further. When open telemetry makes its way into envoy so that we have a nice path to export metrics, and then we this will become the default extension model Forestier.

A

You know in a few releases, let me see if okay, so as you can see, oh so the the little bump there was the transients when both filters were kind of on, at the same time, plus the churn that was caused by the switch.

A

This, which is not a not a usual thing. I just wanted to bring out the fact that we are saving significant CPU, and now you see that we on the right hand, side, the CPU is going down, so the blue CPU on the right is the proxy and mixer CPU just went down to nothing. So if you yeah so like in all these measurements, it is a significant win.

A

So summary efficient, metadata exchange in parts peer awareness and we can build a whole lot of rich telemetry. On top of that, Stewart elementary v2 has made great performance gains in this release, and we will continue to do so, and the Google SEO team is adding webassembly support in on bore, and that should be done in next quarter or to thereabouts and.

A

Yay, this actually worked.

C

So we have to have a few questions we can take, but before you guys leave the survey form that you can fill for this talk from the schedule app. So we can take few questions.

D

Hi nice talk I, we follow Google's best practices on like passing a context object with all of our G RPC calls, and it seems like open. Telemetry covers part of the metadata in those context, objects, and this would cover the rest of the stuff we pass. Is this a replacement for a context, object.

A

So no I mean not really so Oh, most of the open telemetry kind of that the focus is instrumenting application code. This slightly different because here you're using open telemetry to instrument infrastructure code. So here it is going to be bound by like the over line, filter, API and the alignment that we want to get with open telemetry. So it's it's slightly different I, don't think it! It fully applies there.

A

There is a context that is passed in the filter API, but it's not quite the same as the G RPC context. Okay, thanks.

E

Hi guys great talk so I'd, maybe you guys answered it blind quite follow. Why did you need to do of webassembly it like kind of code path or that binding? Why not actually include it into the main source code itself and make it like the default right like I mean, if you guys find it useful like those ten metrics everybody around, you will find it useful. So.

A

The short answer is: thank you if, if they won't, so we primarily want to deliver it first to these geo community. If the rest of the community finds it useful and absolutely we mean we would love to upstream this as well. There is a separate benefit for all this web assembly work, because what you can do now is you can run stock on board and you can deliver these web assembly extensions to the one without recompiling it.

A

What it does is it it enables vendors and even bespoke filters that you may need in your organization to be compiled separately and pushed into Envoy. So this was a convergence of two streams of work. We could have done that separately, but it made a lot of sense to do it together. Okay,.

C

Thank you guys, right.

A

C