Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Graduated Project Lightning Talk: Using Trace Data for Monitoring & Alerting of Appl... Albert Teoh

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Graduated Project Lightning Talk: Using Trace Data for Monitoring & Alerting of Application Health, not Just Debugging - Albert Teoh

Trace data is ideal for debugging slow or erroneous requests, but how can we find (and be alerted about) these problem “needles” in our “haystack” of requests within our constellation of services and endpoints? A proposal of aggregating trace data into metrics is presented; identifying statistically significant problems such as a consistently slow endpoint, then visualizing these results within Jaeger UI or Grafana.

A

Hi everyone, my name is albert and today, I'd like to talk to you about using your trace data for monitoring and alerting of application, health, and not just for debugging, if you're already using a distributed tracing solution like jager. Hopefully this will give you an appreciation for the additional value that you could gain from your span data without adding more metrics instrumentation and, if you're new, to distributor tracing.

A

Hopefully this will give you some motivation to start instrumenting your services and take advantage of the application health monitoring you could you could get for free, along with the trace data.

A

So, just a little bit about myself, I'm a jager maintainer working at logs io on our distributed tracing product and funnily enough. It's based on jager when I do have some spare time, I'd like to walk around the garden and just get my hands dirty literally growing vegetables and fruit.

A

So traces are a goldmine of of observability data. It's rich in context and detail, but actually not the majority of those traces are actually not that interesting. um Once in a while. You do find some nuggets of interesting traces like those with errors or slow requests.

A

But then the question is: what's the best way to find these little nuggets of gold um amongst our our mountain of spans.

A

And so to help explain how we do this. I want to draw an analogy to how pulsars are detected, so parcels are essentially dead stars that are spinning very rapidly emitting radio waves at a clock-like period like much like a lighthouse.

A

um So when they're those signals arrive to earth they're mixed in with other noise from either space or earth, earth-bound sources like mobile phones or tv or radio radio stations.

A

So, as you can see from that first row, it's not really clear that there is a pulse there amongst all that noise.

A

But if we know the period of these pulses, then we could cut this signal up into that period and start adding those signals together, and so the idea is that the pulse signals would accumulate up while the noise cancels cancels each other out, leaving a distinct pulse. As we see at the bottom here.

A

So the gold and the pulse in our analogy is like the single example error or slow span from a servicer operation and the mountain or soil or the noise.

A

It's much like the the millions of spans from hundreds or even thousands of service operations to search through and much like, adding the pulsar signal together so that to to find that distinct, sig that distinct pulse.

A

We aggregate the spans to highlight the statistic, statistically significant issues from the request, errors or duration metrics, that we gather these are also known as red metrics.

A

So now that we have our aggregated span data, what are some real? Some real world use cases that we could apply it to so here. I list out a few of these use cases that come to mind the first one being a high level view of application, health in our organization and a good application of this.

A

A good use case for this is when we deploy a new service, for instance, a new version of our service with a new feature, and we want to make sure that this doesn't impact the organization in terms of increasing error rates or latencies for other services and so having these um these metrics here with like say, sorting by error rates or latency, we can make sure that there are no additional spikes added in a similar vein.

A

We could use these metrics to set up slas, to monitor on, to make sure that we don't exceed, say our one percent sla on error rates and be alerted on them.

A

And finally, an interesting use case that was brought up that was brought to my attention recently in the jaeger slack channel was the ability to detect and identify spans that exceed uh what's what what's expected. uh What's the expected latency and we can measure this expected latency by computing, the the average latency across many of these spans for that service or operation.

A

So here we see a sneak peak at a proposed at a proposal that we're working on of a high level view of application, health per service, with the ability to drill down from that service into the operations and then into its subsequent traces.

A

So you can see here a list of these services with their average latencies, their request rates and their error rates, along with the impact which is the latency multiplied by the requests.

A

um Here's an example of the open, telemetry collector configuration, that's required. The configuration that's required to enable span aggregation, and the idea here is to actually emphasize how easy it is to enable span aggregation in open, telemetry collector, which is literally at least three lines of config, and these config lines are the span metrics line here and telling it to export to the prometheus exporter and adding the span matrix export processor into our pipeline and on the right hand, side here.

A

I've illustrated a simplified architecture diagram uh just to give some context on where the span matrix processor resides in the bigger picture, and so we can see here from instrumentation emitting spans to open telemetry collector.

A

We have a receiver to receive these spans that sends them to the processor, to our span, matrix processor and this processor then forks the data, so firstly, it spans down to the jager exporter so that we can view these spans in jager and secondly, from the metrics that were aggregated into the prometheus exporter down to the metrics collector, to persist those metrics and then grafana to query that data. We also have a work in progress where jaeger's able to query this data.

A

This metric starter.

A

And here's a snapshot of a grafana dashboard that I put together just to illustrate the possible visualizations that you could create from the span metric starter. So, for example, in this first panel here, I've drawn up a histogram of the of the span latencies, where each column refers to the the latency in milliseconds, along with their counts.

A

So the current status of span, matrix processor, is that it's currently available for use in open, telemetry and I'd, encourage you to please try it out and welcome any feedback and coming soon to jager. Is the ability to read, span metrics data and using this data to enhance the ui um in a similar. Similarly, with the mock in a similar uh with a similar idea to the mockup that I showed.

A

Earlier, uh I know that it's a lot of information to take in with little background context, so I've added this resources, page uh resources slide for you to refer to later on um so yeah, so resources on how to view the source code and documentation on the span, metrics processor, along with jager documentation and a link to our slack slack forum, stack channel and also documentation for the open, telemetry uh information.

A

So if you want to learn more about how to how to configure it and how it's designed and architected, definitely there's some good documentation there. That's for now, thanks very much for listening and happy tracing.