Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2019 (Barcelona), 24 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Dealing with the Pesky Path Parameter Problem: Service Profiles - Alex Leong, Buoyant

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Dealing with the Pesky Path Parameter Problem: Service Profiles - Alex Leong, Buoyant

For platform owners and developers to truly grok their service behaviors in Kubernetes, they must understand their services in terms of actual request and responses by HTTP path. Unfortunately, a nearly unlimited number of unique potential paths tends to overwhelm time series capture, making it difficult to get a clear picture of service behaviors. In this talk I’ll introduce the concept of the Service Profile, a custom Kubernetes resource used by Linkerd. The service profile lets operators define permitted routes for the service with regular expressions, which allows Prometheus to scrape and aggregate service behaviors in a manageable way. It also allows operators to set detailed service behavior rules such as retries and timeouts that can be easily reproduced and monitored.

https://sched.co/MPfP

A

Okay, I think we're getting started, hi everyone, so it's it's Thursday afternoon. You've probably heard the term service mesh about a million times by now and and are getting really really sick of it. So I feel you, you probably see the link Rd logo up here. Linker D is a service mesh. This is not a service mesh talk and I'm gonna stop saying the word service mesh now.

A

This is actually talked about prometheus so who here has used or is using prometheus yeah a lot of people? That's what I thought and then of you. How many of you have heard the words unbounded, cardinality, a few okay and probably more than you realize, once once I, once I kind of gave the definition.

A

So my name is Alex I'm, a software engineer at buoyant I work on linker, D I'm, one of the maintainer and, like I, said this, isn't actually a link ready talk, but I'm gonna be using one kriti as an example and I'm gonna show you how linker D would use it for me, thea is very heavily ran into a lot of problems and how we worked around those and how you can kind of use those same ideas in your usage of Prometheus.

A

So I'm gonna talk about this idea of unbounded, cardinality and I'll. Explain what that is and explain how how we ran into that with linker, D and and kind of what we did from there.

A

So prometheus is a open-source project. It is a CNC F graduated project, it's very popular, it's very powerful, it's very, very cool and I would describe it as a time series database. So this means that you can store metrics in it, but you can store them at different times so, rather than just having one value for your metric you'll be able to see a different value for different time points, and rather than just ask you know what is the value of this metric right now.

B

A

Can also ask historical questions, so you can say what is the value or what was the value at 2:00 o'clock this morning or what has the change bin since last week or does the current value deviate from the historical norm, and so this is very useful for for serving as metrics for your services or your applications, because you can monitor it, you can feed it into your dashboards. You can see graphs, you can set up alerts on these metrics and you can kind of see it in historical contract context and see the time series.

A

So it's really really valuable for that kind of thing.

A

So the way this works is that all your application has to do to participate in this is it has to serve this slash, metrics endpoint, so there's an HTTP endpoint that your application has to provide, and then all it does is gives a list. All you have to do is give a list of metrics and their values.

A

So in this example, we have one metric, it's called requests total and it might represent how many requests your application has received in total, and this says that the current value is 17 and what Prometheus will do is it will make requests to this endpoint on a regular basis and it's called scraping, so it will scrape at every minute or every 10 seconds or whatever interval you configure and it'll. Read that value and say: ok now the value is 17 next time it reads it.

A

It might be 18 next time it reads it, it might be 19 and it keeps those data in a time series.

A

And it does this to all the pods in in in your cluster, and this is really interesting, because now, suddenly you can, when you're querying Prometheus, you can query those pods individually.

A

You can say: hey Prometheus, tell me about the time series data for this pod, but you can also ask questions in aggregate, so you could say actually tell me about the total across all pods or tell me about the total across these specific pods and you can kind of roll up that data in all kinds of different ways and aggregate it together, which is really really powerful.

A

Things get even more interesting when you add labels, so Prometheus supports this idea of labels that, on top of these metrics, you can add key value pairs.

A

So in this case we have a label called method which represents which HTTP method these requests are and then, rather than having one total for all the requests we break it down, and we can say here in this case we have 10 requests which will gets and we have 7 requests which were posts and Prometheus actually will read both of those values that you serve and keep them a separate time series. So there's actually two time series here being kept independently rather than one, and now you have the flexibility.

A

When you query this to say, please tell me only about the gets and you can get that data or please tell me about the posts and you'll get that data or you can leave off the label specifier and then Prometheus will aggregate those together and you can get that total of 17 kind of reconstituted. Out of these two time series.

A

And, of course, you can have more than one label, but maybe what you can see happening here is that there starts to be this explosion of combinations. So when we add a second label here, which is status, status, has two values in this case 200 and 400. For now we have two different methods: two different statuses, which means there's four different time series. The two are multiplied together, so you have all the different combinations, so you've get with 200.

A

You've got post with 200, get with 404 and post with 404, and so now we're exposing four different time series for Prometheus to track and, as you might imagine, as you add, more labels and more labels and more labels and more values and more values, more values. This can really really explode.

A

So the total number of time series you have is the multiplication, the product of all the different numbers of values for each different labels. So if your first label has 10 different values in your second label has 10 different values, and your third label has 10 different values. That's a thousand different time series right there. So this can go up very, very, very quickly.

A

But thankfully Prometheus is awesome and a single Prometheus instance can handle millions of time-series millionth. You know, depending on how it's tuned and configured, but it can handle an incredibly large number. So you know for quite a while. You know until you, the number of labels you have or the number of metrics or the number of values that your labels can have really gets too.

A

Large Prometheus has no no problem and as long as you can kind of keep these values under control and and make sure they don't get too large you're not going to run any problems and Prometheus can easily handle it. It's really really powerful mm-hmm, but just to get you a sense of how quickly you can get to a million. If you have six different labels and they have ten values, each that's a million right there.

A

So you really need to be careful and and keep kind of a close eye on how many different values your labels can take and how many different labels you had so yeah. So you want to avoid both adding too many labels, and you also want to avoid labels with high cardinality and cardinality means the number of different values that a label can take.

A

So I warned you about high cardinality, but there's something even worse than high cardinality, and that is unbounded cardinality. These are the two scariest words to anyone: who's kind of setting up for many theists like this.

A

So here's a question: what happens if we add a label for path? This seems like a really natural thing. You would want to do right. You want to be able to slice your metrics down by HTTP request path, because you want to know how much traffic is going to this path. How much traffic is going to this path? You know which different you know, routes of my service are being accessed and and and what's the relative traffic. This is really really valuable and a really really natural thing to do.

A

But the problem here is that that path label value comes from the request. It comes from the outside world, which means effectively. It comes from the user right, it's completely unsanitized user input and as soon as you're doing anything with unsanitized user input. You should be very, very scared, so in this case we're just taking that unsanitized user input and using it as the value of a label which means that that label value can be anything.

A

It can be anything the user provides, it can be slash ABCD, and so it's very easy for this to just completely explode. You know anyone can hit your your application with an arbitrary path and create arbitrary, new time series and Prometheus and and and take the whole thing down. So this is this is not good, but the solution is pretty obvious right. What are you gonna do? If you don't want to have unsanitized user input?

A

Well, you should sanitize it, and so you can do things like have a whitelist of paths that you set, accept and either block everything else or just bucket all that into a you know, an unknown label value, so you are only taking a no number of different values. So if you have ten different routes that you are aware of, you can just specify what those are and anything else should be recorded as as default or unknown. You know unclassified and that way you keep your cardinality under control and you're.

A

Also, not you know just taking arbitrary user data and putting it in your label, values.

A

But there's a another problem that you may run into here, even if you're kind of sanitizing your data and you're, not letting users put in arbitrary things, you can still run into problems with cardinality. So in this example, we have all of these different paths that are coming in so we've got. You know: books, /, 51, JSON, books, /, 52j, Sun, 53 54, and this is really common right. You often have things like IDs or user names or goods, or something like that in the path- and this is not bad.

A

This is not malicious user input. This is like people legitimately using your application correctly. You have you, have these things in your paths for reason, that's how they're structured but number one. This is again gonna blow up your Prometheus, because you have an unbounded, potentially a number of time, series that you're tracking here and number two. It doesn't even make sense to track to split the data out like this. Like you, don't care that you have one request to books, 52, you wanna, you want that all aggregated together, you want to know.

A

This is really logically one route, and you want to keep track of that count for that logical route, rather than keeping these individual accounts on these particular values that are not necessarily meaningful on their own.

A

So we want some way to kind of roll that up in a way, that's logical and can help us deal with first of all, the explosion of cardinality, but also help us do this in a way that is useful to us.

A

So this brings us to linker D linker D is a open source sandwich mash and it's a CNC F project. It is a really cool plot project, I hope, you've all heard of it or checked out the booth.

A

It runs as a sidecar proxy alongside your application and the long of short of how it interacts with Prometheus is that it exposes that metrics endpoint for you. So because it's a proxy and it takes all of your traffic through the proxy. It's able to record all of this data about your request. You know count how many went through how many are successes, what are the latencies of those requests and exposed as those as Prometheus metrics? So your application doesn't have to do that. That manually.

A

And so you get a situation like this right, rather than having your application expose that flash metrics endpoint you have this sidecar proxy that sits alongside and exposes it for you.

A

But you know we have this problem right. We want to be able to expose path based metrics that are very valuable, they're very useful. They help you understand your application, but we have the pesky path barometer problem right when you have a path parameter like the ID or the gooood or or whatever it is that's in that path. You have this explosion of cardinality that you can't keep track of.

A

So what we want to do is we want to consolidate those somehow. We want to group them together in a logical way so that you get stats on the things you care about, and you have a reasonable number of them, and so we kind of thought a lot about different ways. We could do this right. We thought about maybe using some kind of heuristic in order to consolidate those paths.

A

Maybe if we saw that certain path segments had high cardinality but the rest didn't, then we would know to group those together and wildcard out the segment that was, you know, had many different values, or maybe we can use machine learning somehow, but the more we kind of went down this route, the more we realized that there's all these different edge cases that we're never going to handle properly and really every application is different and doesn't necessarily to aggregate paths the same way.

A

So, ultimately, we think the best thing to do here is to put control back in the operator's hand, back in the users hand and say only the person who is developing this application or operating this application really knows how the best way to consolidate these passes.

A

And so that brought us, the idea of service profiles and service profiles are all they are, is a custom resource definition in kubernetes that lets you define metadata about services and link or D will read that metadata and behave accordingly, and so there's lots of different metadata that you can store in a service profile. Just that's really useful, but I think the key one in this case is the list of routes and a route is, is just what we've been talking about. It's a logical grouping of paths and it's usually defined by a regular expression.

A

So this is what a route might look like. It's got a name, so in this case it's the name is get books, ID JSON- and this is just any identifier that lets you logically name that route. In this case we define an HTTP method, so this is forgets and then just a regular expression.

A

So we say that this route should match anything that looks like Pat looks like books and then a number JSON and any request that has a path that satisfies those conditions is going to be part of this route and it will get that route name as its label value so rather than having this explosion of different label values.

A

Now the label values are constrained to just being these path, names and all the requests that satisfy the conditions for that path will get groups together, and this really puts control back kind of in your hands and lets you roll up your your routes in any way that makes sense to you.

A

And, of course, you can define a whole list of these right, so here is a list of four different routes, and, what's really important here and what's really key, is that this is a fixed, finite static, known enumerable list right. It's not something. That's dynamically generated at runtime, based on the traffic coming in. This is four four four routes, which means you're Prometheus label.

A

Four route will have four different values and only four you're never going to have that unbounded explosion, no matter what traffic comes in, it's always going to associate those requests to one of these four or maybe five. If you include the default different values, so.

A

The routes are innumerable, that's how many there are, however, many you define their user-defined and they define these kind of collections of paths.

A

Service profiles, like I said, can define a lot of other things too interesting metadata about your services. That link really will act on so, for example, you can specify that certain routes are reliable, in which case link- or you will know it's safe- to do automatic retries on those. If they don't have side effects.

A

For example, you can give timeouts, which will tell link ready to timeout those requests after they've been ongoing for a certain amount of first Sergent duration and very soon you'll be able to specify things like traffic splits to say well, when you send to this service, actually sends 60% of the traffic over here and 40% of the traffic over. There is really useful for doing things like Canaries are Bluegreen deploys or experiments.

A

So I think you know. Hopefully this is all really unsurprising right when I tell you well, how do you deal with this huge unbounded number of different values? The answer is, you know, I, hope, kind of obvious right.

A

Well, you have to buck up those down to a finite number or to a fixed, known, reasonable number, but I think it's worth kind of calling it out as a pattern, because this is something that can happen not just for paths like we explored here, but for any kind of metric where there's either an unbounded or very large cardinality of values. You really need to be thinking of that.

A

Be aware of that and kind of recognize that that's an opportunity to use this pattern to define a small known set of buckets to collect those data down to so that you don't have this huge cardinality explosion of of time series.

A

So Prometheus is awesome. It can handle millions of time series which is amazing and incredibly powerful, but you have to be very careful because you can get to a million very very quickly, especially when you have these values with high high cardinalities. So so just be aware of your label: cardinality, don't accept unsanitized user input in labels or in anything and if you've got any labels that have high cardinality values, just group them down into a finite number of buckets.

A

Like I said, I work on linker D I'm, one of the maintainer 'z I, didn't talk too much about liquidy here, but I am very happy to talk about linger D at great length. So if you want to talk about liquor, D or about sandwich, mashes in general, come to the link to the booth or find me after and I'm always happy to talk about that stuff.

A

A

We've time for questions yeah.

A

Yeah, that's real question is: if you change the metadata in a CR D in a service profile, is that dynamically reloaded or do you have to restart the service mesh and it's dynamically loaded? So the way this works is that link Rd keeps a watch on one of the control plane, services called the destination service and that service keeps a watch on the CR D. Those are G, RPC watches.

A

So as soon as the metadata and the CRA changes that propagates immediately without restart- and you know, link OD updates its behavior according to whatever is in the service profile.

B

A

So the question is what happens, to paths that don't match any of the defined routes, so there's always a default route, so that I said you know, there's four routes here: actually there's five V, not the default route. So if a request doesn't match any of the specified configured route, it'll fall back to a route, that's just called default and has the default behavior.

B

A

Yeah, so the question is: can you have a route that captures both gets and puts and yeah? So the definitions we saw for four routes were very simple: they were just a method and a regular expression, but these can actually be more complex too. You can have ands and ORS and group these together. You, you can, you know, have multiple different routes, that you are sort of multiple different methods. You accept on a single route, there's actually quite a bit of flexibility. There yeah.

B

B

A

Yeah, that's a great question, okay, so the question is: if your service already exposes a swagger, spec or an open API spec, can you have link already read that directly rather than using a service profile? That's not how it works. Today, this link, rity, will always read the CRD the service profile CRD. It's certainly possible to have kind of tooling.

A

On top that will watch the the swagger specs and convert those to the CRD and kind of update those as they change, but link to D keeps the CR D as kind of the original source of truth, and you would need some other layer on top of that to update that. If the swagger file changes.

B

B

A

I really like that question so question is: is there any anything that can help? You generate these service profiles or these lists of paths when you don't necessarily know what they might be ahead of time, and there are a few different things you can do here. So one was just brought up. If you have a open, API or Swaggart definition for your service, the link already has a tool that will convert that into a service profile.

A

We have the same thing for protobuf, so if you're using G RPC there's a tool that will convert your protobuf file into a into a service profile, the other thing that we think is really cool is that this integrates with linker DS tap feature so long agree. Has this feature called tap where you can watch a request stream and see the request metadata as it's flowing through that stream? You can kind of tap in like a wire, tap and and and see what's going on there.

A

You can actually feed that in to the service profile generation, so you can telling to watch this request stream watch all the paths that are coming through and then build a service profile based on those paths which is really cool and then, if you've got a path parameters here, you may see a large number of different routes and you would need to kind of edit that manually to collapse those down, but it should be pretty easy to do once you can kind of see. What's what's there.

A

Any other questions all right well, thank you very much and hashtag sandwich mash.