Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: From Alert Notification to Comparison of Good and Bad Requests in One Click - Shreyas Srivatsan

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

From Alert Notification to Comparison of Good and Bad Requests in One Click - Shreyas Srivatsan, Chronosphere

Metrics are a great tool for notifying when something goes wrong. Distributed tracing provides the ability to drill down deeper into an issue when triaging an alert with a non-obvious root cause. It’s already difficult to jump from metrics raising an alert to a representative problematic trace, but even once there, users often want to compare a problematic trace with a non-problematic one to help root cause the issue. This talk demonstrates how to jump straight from an alert notification to displaying a problematic trace along with a comparison to a non-problematic trace. This is accomplished with a combination of open source tools such as Prometheus, Jaeger, Grafana and M3. The audience will learn how recent advances in the community can enable them to reduce their time-to-mitigation by providing the relevant context of a bad request vs a good request directly from a graph.

https://sched.co/Zeoq

A

Hello everybody good morning, good afternoon, good evening, depending on where you're located uh my name is shreyas.

A

Today we're going to talk about how we can make use of the current state-of-the-art integrations between metrics and traces to speed up and improve root cause of issues, particularly we're targeting the use case of comparison of good and bad requests to root cause issues.

A

uh A quick background about myself, I'm a technical leader chronosphere, where we're building a holstered, metrics and monitoring platform for large scale, high throughput use cases, I primarily focus on the applications of metrics and traces to solve different observability problems. Previously before chronosphere, I was in the observer team at uber working on the alerting platform there.

A

The agenda for the talk today is kind of split up into three sections. uh We'll first go over kind of like the experience today in root, causing issues with initial intro about different observability signals.

A

We'll follow that up on basically on a journey talking about deep, linking metrics and traces and how that can actually get us to tomorrow, which is kind of jumping from an alert uh to a request. Comparison uh to start off with. uh Let's talk about some observability signals of interest.

A

The three primary signals we're interested in for today are kind of like metrics, logs and traces, uh so jumping through each of these metrics have basically been around for a long time, they're one of the most commonly used tools for observability they're, very structured, they're, fast and efficient aggregate, both at query and collection time, they're efficient from a storage perspective, as they store aggregate information, which makes them a really good tool to kind of identify, high level issues.

A

uh But this you stop short of being able to debug issues in detail, uh so they're not very detailed on a per event basis, uh and you cannot see like request response information in detail.

A

That's where logs and traces come in we're going to focus mainly on the use of tracing in this talk. But I'm going to give you a quick overview on logs. Just for completeness logs are basically the other standard. Observability signal, they're, widely used and they've been around for a long time. Fast. Log search has made it useful for basically search and root cause.

A

They are structured, sometimes, but sometimes they're. Not uh the structured log is improved, like structured logging is improving with better libraries, uh but in general, like they have the full information about requests. Even if they're not structured and like consistent across a full application, they can potentially be linked to other logs based on common tags, like request ids.

A

Finally, we're going to talk about traces traces are useful for understanding and debugging microservices or different code paths. We're going to model it they're, very structured. They store context. Information about the whole like api calls being made. The instrumentation is normally implemented in libraries like rpc libraries or wherever there's a transfer of execution context.

A

So this means the initial setup for the instrumentation requires a lot of planning and work, but once an application is instrumented using tracing users only need to ensure to pass around appropriate request contacts.

A

Unfortunately, uh traces become very verbose uh as they contain the full request response information uh describing every network cop, leading to storage costs, leading to high storage costs so like we need some form of sampling to limit the costs of tracing sampling is a tough problem, uh as you need to pick like what you want to store for the tracing information to be useful.

A

There are many ways uh people do strangling like head-based sampling and like tail base like adapter sampling, uh but many of these end up like losing critical like events, especially for the ones which actually occur uh infrequently um yeah. So moving on, uh I want to talk about like an on-call issue. An engineer gets woken up at the middle of the night. He gets an alert uh uh saying a particular endpoint is seeing higher latencies.

A

So how does the engineer actually like walk through like debugging this issue, so the alert basically has some context: information on it, whether it's an email or page that context information has pointers to grafana graphs or some other visualization system.

A

The first action may actually be to go visit the grifana dashboard and try to see what's going wrong.

A

A dashboard can have multiple graphs. The offending graph in this case is basically the request. Latency histogram graph, we're just showing that some requests are taking much longer than others. This is what we actually need to investigate.

A

There could be many reasons for this. It could be any of the dependencies being slow or just an issue in the application itself.

A

Tracing uh could be useful here to resolve the issue, uh so the user probably wants to use traces or logs to debug further now traces or logs need to be searched based off some tags, but the graphs themselves need to be easily readable.

A

So to get to the actual like tags on the metrics, you may need to go to a different view. Where now you get some tags on the metrics, which here kind of they're, showing you the endpoint uh the instance, the job or some other information now kind of armed. With this tag, information we can use logs or traces for for the root cause for the current scenario. Let's talk about using traces.

A

This slide shows a jaeger search view. Jager is a widely used cncf project for distributed tracing. So once we have the tags, we can basically search for the appropriate traces using using jager.

A

In this particular case, we probably also want to use the latency numbers from the metric to actually search for a target range for target trace like within the latency range that we're interested in so first, the user probably wants to find the trace with the higher latency the trace kind of details, the number of steps it's doing, there's an api call to dispatch a customer which has to look up the customer and that lookup has to go to redis or my sequel in the slow case, it seems, like my sequel, actually has taken quite a bit uh now seeing this, you probably want to see like what the lower latency trace uh actually did.

A

So you basically kind of make another search to look up the look up, lower latency trace. When you look at this, you see that the my sequel call doesn't exist, so it was probably my sequel having the problem.

A

If you actually see a trace difference view with jager supports like comparing both of these traces, especially useful and larger traces, then it's kind of clear that the culprit here is actually my sequel and yeah that then you can kind of use that information to go further and root, cause things further.

A

So trace differences are really powerful tools to help root cause issues, but actually making use of trace. Differencing is not very easy. You need to be able to search for two traces with kind of specific characteristics which in most cases is not a trivial task. uh Moreover, like sometimes like traces for like rarer circumstances, may not even be present, um so the question is like: can we actually jump directly from this email that we got directly to our trace difference in view in the current like with the current technology and jaeger search?

A

This is kind of difficult, because metrics do not have like a trace associated with them, but some work on deep, linking metrics and traces that my colleague rob skillington, presented at kubecon san diego last november, actually makes this uh very feasible, uh we'll first kind of go over a summary of that work here before we talk about how we can leverage that work uh to basically make getting like trace differences, uh much easier.

A

So moving on like talking about kind of deep linking metrics and traces, uh with this kind of with this, it's actually possible to associate like metrics a metric data point with a representative trace.

A

So normally as a recap like the way we did it earlier like. How do we actually use metrics and traces together? The standard integration is to use kind of tags to map between metrics and traces. uh These tags need to be similar like or same like so that when you get alerted on something, you can actually like.

A

Take those tags and search for appropriate traces, and the other challenge is that, even if you try to search for the traces, the traces may not be present due to sampling yeah but kind of with these some of the latest oss technologies, it's actually possible to jump to traces uh directly from metrics.

A

A lot of this work is currently like work in progress, uh but it's possible to store the trace and span ids alongside the metric data points, giving a way for us to go directly from a metric to a trace, we'll walk through kind of the different steps needed to make this possible.

A

The first thing we want to talk about is kind of open, metrics and exemplars, so open metrics is a standard for transmitting metrics at scale. One of the things it specifies is an extended prometheus exposition format, which is essentially the prometheus like regular, prometheus exposition format.

A

But in the comments section of the exposition format, you can like add different foreign keys and context information on a data point for the purpose of kind of this exercise we're going to make the trace id the foreign key. So it allows us to kind of link to traces now, given this exposition format, we need a way to actually emit metrics uh with trace ids and that's where kind of open telemetry comes into play. So open telemetry is a new standard that provides a single set of apis to emit metrics and tracing information.

A

It allows essentially us to emit metrics, along with additional context, information, this solution context. Information can come from the traces it could be. The trace id and open telemetry supports the open, metrics, extended premiums, format.

A

Some of the below things we're going to talk about, and especially in the prometheus and in three phase, are still a work in progress, and we have we're having continuing discussions with stakeholders.

A

So we've been working on like adding prometheus support to script metrics with exemplars and making them available to remote right customers. uh M3 now has the ability to actually store exemplars alongside metric data points, uh the exemplars or essentially the trace ids, are durable and stored for the lifetime of the data point. um The one challenge here is that it does end up using a lot of extra memory uh depending on the length of this foreign key, the trace id.

A

It could be a few tens of bytes per data point, so we need to kind of limit emission and storage of exemplars to just the specific metrics, where the information is most useful.

A

In this scenario, like especially when we kind of combine this with tracing rpc metrics is one of the cases we're interested in the query. Engine of m3 supports returning exemplars, along with data points, and it also ensures representative exemplar to be present even after applying aggregation functions. Like sum and max and others um so one of the challenges with using traces for debugging these like corner case like corner cases, is actually sampling.

A

Traditional trace sampling techniques like head-based sampling or different adaptive, sampling techniques or, in many cases insufficient, because these one-off traces don't get sampled.

A

So we need to use some form of tail-based sampling to pick traces that were emitted as exemplars with metrics. It's assuming. We have a trace holding tier which can cache all the traces for a short duration with the m3 aggregation layer, then kind of indicating to it like which additional traces to persist.

A

So traces, basically like the regular head-based sampling, still applies, but we additionally go and like store, persist. Another like additional set of traces which the metrics are directly associated with.

A

So what does this complete ingestion pipeline looks like look like you? Have an application instrumented using open telemetry.

A

It's using open, metrics, extended prometheus format to emit those metrics. The metrics get scraped by prometheus, go through them through aggregation tier to like long-term storage for the metric data. Point and trace id traces go to the error collector, but instead of going uh 100 sampled and go to the error collector. But instead of going directly into storage, they sit in a holding cache uh and the holding cache basically gets information uh from the m3 aggregation tier to actually store these traces.

A

Their associative metrics, any regular sampling like head-based sampling, can also can still happen, and those traces can also like end up in the tray storage. uh Finally, like different consumers like uis, grafana or jaeger, and the alert engine can basically speak to the tracing in the metric system uh to actually like create, like nicer user experiences.

A

uh So you can basically end up with uh a grafana graph, for example, where, like because every metric data point is associated with a trace, you can make do it make extensions or plugins, where you can, like click on a data point and kind of jump to a corresponding trace.

A

So now, initially, we showed kind of how trace differences can be valuable tools for debugging.

A

The difficulty is actually in searching for these traces now deep, linking between metrics and traces actually makes the job of actually searching for these traces really trivial.

A

So we plan to leverage that to build integrations that allow us to jump from a graph or an alert directly to a request. Comparison, uh I'm going to jump off to a demo and we're basically going to talk about like what on-call experience can actually look like for the purpose of the demo. We're going to use a jager example, application called heart rod which simulates different types of traces request responses. So we can create like a few interesting traces, so jumping uh to hot rod.

A

So we can kind of see hot rod running here. We we basically put some data into it and hot rods like sending requests.

A

uh We had some alert emails come in uh previously now these emails are like regular word emails, but additionally, they have some additional context with like a dashboard which is common, but they also actually point to a trace difference.

A

So if you actually click to the trace difference, we basically get back to the initial example we had, but we basically were able to get here in like one one step, uh we'll talk about how that's possible a little later, but essentially this like automatically, is showing you the difference between like what is a good and a bad trace, uh and you can actually see here that my sequel being present is the actual problem.

A

uh We've also made some changes kind of through, like a grafana graph, has a number of different graphs uh here like we can actually look at something like a 5xx where now I want to see like why this error actually happened and you're able to jump directly to another tracedif and it kind of shows you the trace, difference information.

A

Let's look at another one like yeah, so this trace difference is actually showing you that both of these, both these traces kind of took the same path, but the fade interest like they missed the radius cache uh ends up kind of missing, like failing in my sequel and returns, but the successful case actually succeeded my sequel and made kind of further forward progress.

A

So these are kind of kind of some of the integrations that we can actually set up. So I'm going to move back and kind of like jump back to the demo. Now, let's jump back to the presentation now so uh yeah, so how like so? How is it actually possible to actually like be to like to build these integrations?

A

So we leveraged open, telemetry and open metrics uh so that we can kind of emit the trace ids as exemplars.

A

We leverage prometheus and m3s like support to scrape exemplars, scrape and kind of store exemplars alongside the metric data points, but the critical part is uh we finally kind of like worked on building contextual links uh into the systems which are actually like, consuming the metrics uh consuming the metrics and the trace information.

A

uh So these are essentially your alert engines, your profana and other visualization tools. This is really possible because it's not really difficult to build. These contextual links, as the underlying data is actually readily available. M3 cray returns exemplars alongside metrics.

A

It ensures a single representative exemplar like if present like, across aggregation, functions and kind of using this information, it's possible to build contextual links into different query consumers, as can be seen here like when the value comes back. There is a timestamp data point and there is a span and trace id.

A

So if you have something like this, this kind of shows another sample workflow. We could build into graphing solutions uh where you kind of click on a like one metric and you mark that as your control trace and which in general kind of can be, your two axes can be your control trace and then, if you have like 4x5xx and other like error information here, when you click on like the 5xx and you show a trace comparison, it'll like diff it against the control trace.

A

So that's one idea of like how you could build contextual links um for a graphing engine. You could, in general, just have the graphing engine say that there's this is a control metric for alerting integrations. You could provide the ability to configure metric that can act as a good source of good exemplars sorts of like control exemplars.

A

Another thing which is actually possible is to basically like for standard uh well-named metrics like rpc metrics. uh We can build plug-ins uh that can automatically detect and provide comparisons kind of based on knowledge of the types of metrics emitted uh that is kind of what we did with for our demo, where we augmented jaeger's rpc metrics to actually have the trace id information on it. So, given a failure or a 5x metrics, we know kind of what the success metric uh looks like and yeah so yeah.

A

For summary, uh we kind like we basically kind of showed you that trace differences can be really powerful to debug issues. If you're able to find the right traces and kind of using deep linking support between metric data points and traces, we can build a number of integrations that can speed up root cause.

A

Finally, we kind of showed a sample of how this can look for an alerting engine or a visualization tool like grafana. uh So where can you actually get this like? Where are we on like on this journey? uh The current end-to-end demo is actually is present at this link. uh There are a few changes which you've already made towards this, but there are many other kind of work in progress. Changes uh we've also like had help from the open source, like other from like grafana like bjorn from grafana.

A

I hope this exemplar support and prometheus client and actually change to kind of store, exemplars in memory and then forwarding them on to remote right is still kind of a work in progress.

A

Some other resources that are of interest is like, if you've not checked it out, uh please do go check out rob's talk on deep, linking metrics and traces uh that he gave at kubecon san diego.

A

uh It kind of talks like in more detail about like how this deep linking is actually possible and kind of what are the? What are the technical challenges uh with that, uh and there are a few other resources here on open, metrics, open, telemetry, prometheus m3 and grafana, and finally, thank you so much thanks a lot. Everybody thanks for your time do come stop by our virtual booth to sign up for demos, uh to check this demo out and the other interesting features uh we're actually building uh yeah. Thank you.