Cloud Native Computing Foundation OTel Community Day NA 2022, 29 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lightning Talk: Metrics Make Meaning: Why the UpDownCounter? - Joshua MacDonald, Lightstep

Description

SAVE THE DATES!
ArgoCon – Register Now | September 19-21, 2022 | Mountain View, California - https://events.linuxfoundation.org/argocon/

KubeCon + CloudNativeCon North America | October 24-28 | Detroit, MI - https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/

KubeCon + CloudNativeCon Europe 2023 | April 17-21 | Amsterdam, The Netherlands - https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/

OpenTelemetry is a collection of tools, APIs, and SDKs. Use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help you analyze your software’s performance and behavior. Go here to learn more: https://opentelemetry.io/

Lightning Talk: Metrics Make Meaning: Why the UpDownCounter? - Joshua MacDonald, Lightstep
JW Marriott Austin

Speakers: Joshua MacDonald
Link to join the virtual event: https://www.accelevents.com/e/otel-community-day-na-2022
The virtual event concludes at 1:50PM CDT; In-person event concludes at 5:00PM CDT

The OpenTelemetry metrics data model has richer semantics than its predecessors. Events captured through the OpenTelemetry metrics API are more than a just numbers in a line, they carry meaning. With the use of semantics-preserving data transformations, the production of meaning from metrics is decoupled from consumption and interpretation. Dimensional attributes applied to metrics in source-level instrumentation can be safely erased during collection, without loss of meaning, giving OpenTelemetry users new ways to control costs in their metric infrastructure.

In this lightning talk, Josh MacDonald will explain how, to accomplish this goal, OpenTelemetry metrics added a new kind of instrument, known as "UpDownCounter", to give the instruments individual semantics. To understand when UpDownCounter instruments should be used instead of Gauge instruments, first we have to understand the semantic distinction between counting and measuring. After sorting out instrument semantics, systems that graph OpenTelemetry metrics data can guide the user toward meaningful displays and even construct meaningful default dashboards from any metrics source.

A

Hi, I'm josh mcdonald, member of the open, telemetry technical committee and an engineer at lightstep I'll, be talking about metrics instruments and the requirements that gave us the up down counter a new kind of instrument.

A

Let's review how we got here. Metric systems have been around a long time, but I'm thinking about what happened around 10 years ago, looking at open source systems prior to then we had just one kind of metric instrument, I'll call it the number instrument. We would use it to report numbers with timestamps and then we would plot those numbers as a function of time. The result is numbers in a line. The nice thing about numbers. Is they support? Math?

A

You can do a lot with numbers, but first you should know what they mean and 10 years ago, the interfaces we used to report metrics began to change. Why? Because it helps the metric system, help the user when numbers have meaning back then around 10 years ago there were a couple of changes in metrics interfaces. The counter instrument was introduced with an interface dedicated to counting. This is a convenience for the programmer, since they no longer have to track a running total. This also made it easier for the metric system to correctly compute rates.

A

The semantics of a counter instrument are that resets are not meaningful.

A

Another thing that happened with metrics apis around 10 years ago is they changed to support attributes. Now you could decorate metrics events with a list of attribute values and every distinct combination of attributes used would produce separate lines of numbers in the diagram pictured. There are three dimensions: a b and c that categorize the data we can picture. The metric instrument in this case producing a cube of number lines with coordinates equal to the category values for attributes a b and c now still talking about progress that happened around 10 years ago.

A

These two innovations, the counter and the attributes work really well together. When we add a new attribute to a counter, the result is a new way to subdivide the count which generates new ways to group and aggregate the same data counter data for specific attribute combinations can be compared with other attribute combinations or divided by the total. To form ratios and fractions, there are two diagrams shown in two snippets of instrumentation code. These two examples are both cpu usage counters, but one tracks usage on a per cpu basis and the other tracks total usage.

A

This shows how we can remove an attribute from a counter either at the source or when displaying data, without a change of meaning to remove a cpu label or any label from a counter make sure. To sum, the individual counts.

A

When the counter was introduced and defined to support resets, it meant that they couldn't be used for counting up and down for counts, to go up and down a gauge instrument was recommended. Instead, here's the prometheus documentation for gauge a gauge is a metric instrument that represents a single numerical value that can go arbitrarily up and down. Gauges are typically used for measured values like temperatures or current memory usage, but also counts that can go up and down like the number of concurrent requests.

A

As a consequence, we have one instrument with two interfaces: use set to record the current value and use add subtract, increment and decrement to change the current value, and here we are open. Telemetry has separated these two interfaces.

A

The requirement that led to separate interfaces for updown counter for things you count and for gauges for things you measure is best described as follows: open telemetry, metrics processors must be able to remove attributes from metrics data without a change of meaning. We want to be sure that when we remove an attribute, the result is just the same as if the attribute had never been recorded in the first place, when an instrument is being used for counting attribute removal means computing.

A

The sum which is the same count that we would have achieved had the attribute never been used. The point of all this is that a metric system can automatically transform metrics data as a form of cost control and to produce meaningful visualizations without knowing how the instrument is defined. As long as the system knows, the kind of instrument that was used to view any metric at the cluster level, a job level or a rack level, simply remove the irrelevant attributes.

A

However, this would not be possible with just two kinds of number instrument we needed a third instrument. The up down counter an up down counter is something like a counter and something like a gauge. Upto encounter acts like a counter in the sense that attributes give new ways to subdivide the metric, defining meaningful ratios and fractions, and in the sense that the natural aggregation is a sum up down.

A

Counter acts like a gauge, in the sense that there is not a reset operation and in the sense that it defines a current value and is not primarily used to define a rate. These instruments exist in the real world. They generally report the difference between two counter values and where we find these being used is because the two counter values would not be considered useful on their own. One. Real world example is a parking lot counter a device that counts how many cars enter and exit a parking lot.

A

This instrument can be used to display how many spaces are available on each floor of a parking lot structure, for example. Another real world example is the energy meter on a home with a solar array attached to the electricity grid when the solar array is feeding energy to the grid. The count goes down when the home draws power from the grid. The count goes up.

A

Let's review, open telemetry offers three instruments for reporting numbers the counter the up down counter and the gauge these three instruments have distinct use cases. Counters are useful for monitoring rates, up down counters, for monitoring, totals and gauges for monitoring measurements and other derived quantities.

A

When we aggregate metrics data, particularly when we remove attributes to save cost or reduce cardinality, there is a natural aggregation that applies for counters and up down counters. It is natural to group by the sum for gauges. It is natural to group by the mean.

A

Now, you may think that if your instrumentation instrumentation is doing the right thing, you won't need to remove attributes in your metrics pipeline. While that may be true when it comes to inspecting and visualizing metrics data, it is easy to create metrics, where there are too many number lines to effectively view them at once, say a per host metrics in a large cluster. When we have too much too much cardinality to effectively view metrics data, we need to remove attributes, and that's all there is to it up down.

A

Counter is distinct from counter-engage instruments in ways that help a metric system to control costs, because, whether writing metrics data or reading it is useful to be able to remove attributes without a change of meaning. Thank you.