Cloud Native Computing Foundation OpenTelemetry Community Day North America 2020, 4 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lightning Talk: Controlling Your Data with the OpenTelemetry Collector

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Lightning Talk: Controlling Your Data with the OpenTelemetry Collector

How do people change telemetry data to support use-cases including PII redaction, downsampling, custom tags and fixing instrumentation flaws? The OpenTelemetry Collector! Telemetry data contains a lot of valuable information, but sometimes that data may or may not contain what people want. Modifying telemetry data by updating instrumentation often proves challenging due in part to application impact. The OpenTelemetry project solves this problem by providing a pluggable architecture in the OpenTelemetry Collector. Learn how others have configured it to support a variety of different use-cases. PII redaction Downsampling Custom tags Renaming spans Fixing instrumentation flaws"

A

Hi, my name is steve flanders and I'm a director of engineering at splunk responsible for observability, getting data in my team, and I work extensively on the open, telemetry project and I'm excited to join you today to talk about the open telemetry collector for those of you not familiar with the collector I'll start with a quick introduction before showing how you can use it to really control all your data.

A

So what is the open? Telemetry collector?

A

The collector is a vendor agnostic implementation that really allows you to receive process and export any telemetry data in a seamless way. It provides a single binary that can be deployed in a variety of different ways, including as an agent on your hosts or as a stand-alone gateway and it's the default destination for the open, telemetry, client. Libraries.

A

Let me walk you through how it looks from a reference architecture perspective. Let's assume you have one or more hosts each host containing one or more applications, and you want to send your telemetry data to one or more different, backends.

A

Typically, how you get started is you would deploy the open symmetry collector as an agent on your hosts, for example, in kubernetes this could be a daemon set.

A

This would allow the collector to go ahead and gather metrics from the host itself and it provides a destination where, if you deploy the open, telemetry client library, it will be able to send metrics and traces from your application to the collector itself. Note that the collector also supports popular open source client libraries, including prometheus zipkin and jaeger.

A

In addition, you may also want to deploy the collector as a standalone gateway service. The open, telemetry collector is made up of variety of components. First, we have receivers which are used to get data into the collector. These can be either push or pull based.

A

On the other end, we have exporters how you get data out of the system. Again, these can be push or pull based in between. You have the notion of processors, which are used to massage the data as it moves through the collector. Now, for each of these different components, you can build what are called pipelines, for example. Maybe I have a pipeline where I have an otp receiver. Otlp is the default protocol used by open telemetry that passes through a batch processor and also a kubernetes tagger processor before being exported out through a jager destination?

A

In this case, you can see that the converter is translating from one format to another.

A

You may also have a second pipeline where, again, you are leveraging the otp receiver, this time with a separate batch processor in kubernetes tagger and this time you're, sending it to two different destinations, otlp and prometheus.

A

As you can see, pipelines give you flexibility and choice when it comes to telemetry data. At the end of the day. It's your data, so you need to make sure that the components that are available meet your specific requirements. The collector provides flexibility and choice through configuration, and there are a variety of different ways. You can leverage it to control your data.

A

First, let's talk about pii reduction. There are a few different types of scenarios where this may be applicable most commonly. We have things like database queries which can contain sensitive information, but this could also apply to application specific metadata.

A

In the example on the left, we're doing two things. First, we're saying that if we notice that a span has an attribute with a key of pii and a value of false, then we don't actually want to go any further with this attributes processor.

A

Otherwise, we want to look for the key of database dot statement and if we find it, we want to hash that value. On the right hand, side. We have a different example here, we're looking for spans where the service is called, sell on amazon and we're looking for an attribute of job underscore args.

A

If we find it, we will delete the job underscore args attribute. Next, we have sampling, which is typically used to control costs.

A

In this example, we're doing two things: first, we're looking for all spans where the service is data, hyphen platform and we're adding an attribute of sampling priority, setting it to a hundred to ensure that these spans are always sampled, secondarily we're enabling the probabilistic sampler and setting the sampling percentage to 15 percent.

A

This applies to all data processing through the system, but note that the sampling priority takes precedence.

A

Data enrichment is used to add metadata or to add resource information. For example, we can add an attribute called environment with a value of production for all spans that pass through this collector.

A

In addition, we've enabled the kubernetes tagger, which allows us to add information such as the pod deployment namespace information directly from kubernetes.

A

This is possible when running the collector as an agent on hosts and is something that an agent is in the best position to go ahead and add, given that the application may or may not be aware of this metadata.

A

Next, let's talk about reducing cardinality. You may come across services that are instrumented in such a way that the span name values return a large number of unique values resulting in high cardinality.

A

This can be overcome using the span processor in the collector. For example, we can include certain service names and we can write rules to actually normalize the span names, thus removing the cardinality. Here you can see an example with core dns, where the service instance and request name are being normalized.

A

Finally, let's talk about fixing instrumentation flaws, it's much harder to actually change and modify code instrumentation than it is to update collector configuration, so the span processor also allows us to override the span name given other attributes on the span itself. For example, here you can see that we're looking for the rails.controller and rails.action attributes we're separating them by a period and we're using them to rename the spans, thus providing value where value may not already exist.

A

As I mentioned for each of the different processors receivers and exporters that you have, you want to define one or more different pipelines. Pipelines are actually defined per data source today. Traces and metrics are fully supported and initial log support does exist. The order in which you define the processors actually matters.

A

The great thing about the collector is that it handles translation for you. This means you can receive in one format, while exporting in a different format, thus providing a vendor agnostic solution.

A

Now that you know different ways in which you can control your data with the collector, I want to actually show you live with a demo for this demo, I'm going to start with a base yaml file for the collector.

A

This will allow me to receive span data in either otop or zip code format and export that data via a logging exporter. So we can see it locally on the screen here. Let's go ahead and run this docker container.

A

Then I've actually constructed a json payload that consists of a single span trace, which I can then send via curl to the collector, and we can go ahead and check the output.

A

Now above here, you can see that we did in fact receive a span. I span with a service name of api. It looks like it is a server span kind and it has a variety of different metadata. For example, you'll notice the http underscore response code, which was 201 in this case.

A

It also has some interesting additional metadata. This looks like it might be a social security number which would be pii and definitely not something I'd want to attach to my spans.

A

I can also see an email address here and well that might be sensitive, and maybe I'm not that comfortable sending that data either. So what can I do about this? Let's go ahead and go back to the configuration here, you'll notice that I'm using an attributes, processor and it's currently configured to go ahead and add some metadata. In fact, it adds an environment tag where the value is set to panic clinic well.

A

I can actually use this attributes processor to do a variety of different operations like, for example, that social security number really needs to be removed. So let's go ahead and use the delete action and I saw the email address too and that's a little bit sensitive, so maybe I'll go ahead and hash that value so we'll just make those two changes. We will fire the collector back up and go ahead and send the exact same data again and let's see what we get this time.

A

So I still see I get an environment pet clinic, that's been added, so that's great.

A

I can see that the email address is now hashed, so I don't really know what the value is, but I have a unique identifier that I could then query in the back end that I send this data to and I no longer see a social security number. So that's that's great.

A

As you can see, the collector makes it really easy to control your data. We hope you'll come take. A look at the project join the conversation on getter or prs are definitely welcome. So take a look at the good first issue and help wanted labels thanks so much for joining, and I hope you enjoyed this presentation.