Cloud Native Computing Foundation OTel Community Day NA 2022, 29 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lightning Talk: Beyond NTP: High-accuracy clocks for distributed observab... Yilong Geng & Dan Zheng

Description

SAVE THE DATES!
ArgoCon – Register Now | September 19-21, 2022 | Mountain View, California - https://events.linuxfoundation.org/argocon/

KubeCon + CloudNativeCon North America | October 24-28 | Detroit, MI - https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/

KubeCon + CloudNativeCon Europe 2023 | April 17-21 | Amsterdam, The Netherlands - https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/

OpenTelemetry is a collection of tools, APIs, and SDKs. Use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help you analyze your software’s performance and behavior. Go here to learn more: https://opentelemetry.io/

Lightning Talk: Beyond NTP: High-accuracy clocks for distributed observability - Yilong Geng & Dan Zheng, Clockwork Systems

Debugging issues on a microservices application on a distributed system is challenging when clocks on different nodes don’t agree. Which event really happened first? Is a log message related to another event on another machine? Wouldn’t it be nice if the clocks were all synchronized so you didn’t have to worry about these issues?

From our research work on Huygens (NSDI’18), Clockwork Clock Sync has been developed into an enterprise-grade solution, validated and deployed by many financial and high-tech companies. It’s 3 orders of magnitude more accurate than standard NTP. This talk will discuss the details of the clock sync algorithm and the Clockwork API. It will also include demos of how observability tools such as OpenTelemetry, Jaeger and Pixie can leverage synchronized clocks for observability, such as providing time-aligned distributed traces and surfacing network insights.

A

uh Hi, everyone- and you know thanks for the introduction, I'm elon from clockwork uh today. I'm gonna talk about uh clocks and you know observability.

A

um Time is a fundamental concept in distributed systems right, so you open up every distributed system textbook. There's a section about clocks. We have physical clocks, we have, you know logical clocks. We have hybrid clocks, so clocks are really everywhere right. They are useful for databases, consensus protocols, snapshotting and observability.

A

However, the you know, general consensus in the industry has been to never rely on clocks when you build their own system right, because clocks are very hard to synchronize and they are very fragile, even if they are synchronized, they can go out of sync. Just you know very quickly right so clocks are just uh create. You know, coarse crystals, fundamentally right and all the crystals they have. You know inherently have different frequencies and the frequencies also change with time right.

A

So you know the clocks are ticking at different rates and also in order to measure the offsets between clocks. You have to you know, send packets between the you know, servers and to measure latencies and thus clock. Synchronization is also subject to um you know: network congestion right. So if we had, you know, super stable clocks that all have the same frequencies and we had a you- know versatile network with a constant network delays. Clocks, clock sync is not a problem.

A

However, that's not the right reality right, so nobody actually uses clocks or depend on clocks when they build their systems so at clockwork. We believe that a highly accurate, scalable and stable you know caution system. It's gonna, you know change the dark. You know the um paradigm right. It's gonna, remove the clocklist assumption when building distributed systems and it's going to enable new system designs and it's going to improve the performance of existing systems.

A

So ntp and observability right um so first of all, your timestamps are very important for observability, because you know we have timestamps for spans. We have timestamps for logs they're, just everywhere right and they're, critical for analyzing the delays and analyzing the you know ordering our events and everything, um however ntp is, is not you know serving as well, so even the best ntp solutions uh today, for example, in the cloud in the same region, they can do. You know tens to hundreds of microseconds.

A

However, the one-way delays in the cloud are on the order of 50 microseconds right. So, if you, uh you know, send the packet from c608 to server b and you take a transmit timestamp and receive timestamp on two different servers. You can often measure like that, not t1 with delays right, for example, this figure on the right shows that we are measuring active multi delays just using ntp in the cloud.

A

So these are very bad in terms of measuring latencies and also, if you actually use these timestamps, to determine the order of these. The two events you you're getting the wrong results. Right, you are say seeing okay, I receive the packet before I even send the packet and outside of the cloud ntp is doing even much worse right. You generally got mini seconds and also people sometimes see um you know tens of milliseconds, or even you know seconds right so actually, michael from aspecto just had a session.

A

You know such an instant in his own demo right where all the spans are out of place. Not all like some of these men are out of place.

A

So thus we think you know we have already spent so much effort in um you know, instrumenting the code you know auto generating you know the instrumentation and collecting the data we we really deserve some. You know a pure, accurate time stand for, uh for us to. You know understand how the system is working right, so, okay, so next, I'm gonna talk about three use cases of clock, sync in observability.

A

The first use case is aligning traces right, and this has been actually a long-standing problem um right. The problem is easy to understand. When the clocks are not well synced, the spans are not, you know, well, aligned right and thus the you know the timing diagram is wrong and then we we can attribute delays incorrectly.

A

um The community has also tried hard to cope with this problem, for example, when uh when people know that their child span should be, you know within the parent span. But, however, the time stamp is not saying so. People would just you know, center justify the child span right.

A

Just just to make the you know trace, um you know just make. The trace more mix makes more sense right. However, this you know creates further confusions.

A

um So this is an example where uh bad clocks can confuse us, and can you know um you know, create expand? You know traces that make don't make no sense right. So in this example, initially we were synchronizing clocks using ntp, and we can see that expand from the currency service actually started before its. You know. Parent span started right and another span from the shipping service started after its parent span ended right, which obvious obviously makes no sense.

A

However, if you simply switch to uh actual timestamps, you know the tree suddenly is making sense right and all the timings are correct, and you know everything and there's also an additional benefit is now we can measure one-way delays, for example.

A

Now we can, you know, in addition to run trip times right now we can measure the time between rpc request is sent to the rpt. Rpc request is received right and- and similarly you can also measure the time when the rpc response is sent to rpg responses you know received, so this has never been possible before without accurately synchronized clocks.

A

A second use case of you know, clocks and observability, is to you know, put distributed logs on a single timeline. You know, as we all know, the open temperature community have been working hard on logs right, and this is only one place, particularly within you know. Accurate times, accurate synchronized clocks will help um so in tracing we, you know we can propagate the context right and we can kind of you know infer the ordering of some of the spans from the context.

A

However, this will not be available in logs because logs are you know by nature, unstructured, and we can only rely on the timestamps to determine the ordering of the events um you know in. In this example, uh we have two processes: logging, yeah pro process alex and process, bob right and the logs say alex added one to to the inventory at t1 and bob took five from the inventory at t2 and alex added. You know, 10 to inventory at t3 right if I were a developer.

A

Looking at the you know these logs, I would be wondering: okay, is there a bug in my code, because you know the the logs are showing the inventory? Will not you know, when's an active at t2 right or is it like timestamp problem? Maybe you know the clocks are not synchronized and that's that that's why the you know logs are now making sense right. I would be wondering whether I should be you know, spending time looking to this problem and I won't have any of such doubts. If I had accurate timestamps in the first place.

A

A third use case of accurate time. Sorry about this is uh in instrumenting message based microservices right. So generally, there are two classes of microservices rpc3 based and message based right and nowadays, rpc based systems are more popular, but they're. Still there are many. You know systems are message based right, so in rpc-based systems.

A

So for every request you always get a response right, so you can kind of get away with measuring launching times right. You got a sense of the latency in the system by measuring launcher times. However, in a message based systems, there are no requests and responses right. The messages simply flow through the system and it's actually very hard to pin down where the latencies are in the system. Right and that's why?

A

If we had a synchronized clocks in the first place, we can simply measure timestamps at different stages in the system and get a sense of the delays or the delays happening in the network in the service or in the man in in a message: bus, okay. So these are the um three use cases um that we we, you know uh we thought of, and you know uh for clocks in observability right. I believe there are way more than what we talked about.

A

So a few quick words about clockwork um so at clockwork. We built a very uh you- know: accurate scalable and stable clock sync system right and it you know: sinks clocks to nanoseconds, with neat cover timestamps and in the cloud we have cpu timestamps and we can synchronize clocks to microseconds.

A

um You know the coxing system has got onto the front page of new york times. It has been adopted by many rapid reputable companies and is available on all three clouds.

A

So the um so clockworks clockworks clutching system is very different from ntp all right and we actually actually published the paper about about the system and it's published in an sdi 18.. So for the technical details um you know you can find them in this paper. We also there's also you know a video of a talk in this link.

A

um If I just talk about one difference between clockwork and tp, I would say: ntp synchronizes clocks through a tree right and then there are, you know, a few problems that can come with it. For example, if a certain node fails, uh you know it's possible that the entire subject could fail right. So by the way, just to be clear, so ntp is like multi-rooted tree is slightly more complicated than uh than these three right, but, however, a clockwork. We went much further in clockwork. We synchronize clocks through a mesh right.

A

In this example, we have three regions uh across the u.s west, central east right and we have 10 clocks- or you know, 10 virtual machines actually inside each region, and I just picked the random clock, the pink one, and you see that this pink clock is talking to four other clocks in the same region and five. You know other clocks in different regions right and and every clock. Does this every clock talks to a number of random neighbors right and this forms a probe mesh. It turned out with this pro mesh.

A

We can do clock sync in a much more, accurate and tighter way right. We can actually discover asymmetries in the network. We can discover that we are doing, but you know badly or we are doing very well like we haven't estimated how well we are doing right and we can also you know, handle different types of failures. You know, thanks to the you know, redundancy in the system.

A

So this is, uh you know how we thought about. You know we can help with open telemetry community. uh First of all, we want to make clock sync available to the developers community. All right so we're gonna, have you know utc synchronized time that are tied to you, know gps clocks and we're going to have globally distributed time, servers right so that accurate time is accessible by you know all developers.

A

The clock sync system is going to do orders of magnitude better than ntp, and we also plan to provide a google true time like api, where there's a bound you know associated with every timestamp. So you know how much you should be trusting the timestamps.

A

Secondly, uh you know. The second idea is that you know this is only an idea and we'd love to get your feedback on. That is that we think you know, in addition to timestamps, it's actually critical to know how much we should be trusting the timestamps right and we think maybe we should be you know, generating and propagating and storing the accuracy levels for the timestamps, along with the timestamps right and that's when you are analyzing the data, and you know you don't make cd mistakes right, you don't just trust wrong clocks.

A

The third idea is that you know we can make a timestamp translation service right. It may be difficult for people to switch out and tp and use a different closing system, but, however, we can do translation after the fact you can just run ntp as is and collect whatever timestamps they provide, and when you want to analyze the traces you can just you know, translate the timestamp and get the correct timestamps on demand.

A

And that's uh concludes the presentation, um a clockwork. We believe you know accurate clocks and globally, consistent timestamps can make open telemetry tools better and debugging much faster and easier and we'd love to hear your use cases, and you know we want to hear about our feedbacks too. Thank you.