Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2020 - Virtual, 4 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Improved TiKV Observability: How We Trace Events under Nanoseconds Latency- Wish Shi & Zhenchi Zhong

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Improved TiKV Observability: How We Trace Events under Nanoseconds Latency - Wish Shi & Zhenchi Zhong, PingCAP

Observability is beneficial but often comes with a price. When adding tracing to low latency services (e.g. < 1ms), engineers might find notable performance degradation. Besides, trade-off solutions have inherent limitations. For example, sampled tracing may leave errors or unusual latency sampled out and missed. In this talk, Wish Shi and Zhenchi Zhong will share their experience in implementing a high-performance OpenTracing-compatible tracing library, which was originally created for TiKV. The library can trace events under nanoseconds latency without sampling on the modern x64 architecture. Decisions, design details and trade-offs will be presented, along with an open-source implementation available in both Rust and Golang.

https://sched.co/fGWU

A

Hi there thanks for joining us today. This is wish from the thai tv community, I'm an infrastructure engineer at pincap and also a co-contributor of taiki projects, I'm very glad to share how we improved thai kv. Observability with my colleague john shoe.

A

Recently, we add tracing events to the tag kv with very little overhead. Let's say only a few nanoseconds per event and we would like to show you about how we did it. Hopefully this is helpful.

A

First, let me introduce you, the techie project we are working with. Tachvi is a key value database. It is distributed, transactional and open source. It has recently became a graduated cncf project.

A

So far there are more than 8 000 stars and more than 200 contributors in github as a key value database. Thai kv, accepts key value, read and write requests, for example, get and put. Sometimes there are details. A writer request may suddenly take long time, while most others are normal.

A

This can be caused by different reasons, and we would like to know why. Now we have logging and matrix logging is not very useful. In this case, it is usually hard to link logs related to a single request together and for matrix. Sometimes it is not useful as well.

A

Measure is only reveal aggregated information, like average latency and when multiple payloads are mixed together, adjusted from a single request is hidden.

A

So as a result, we want to use quiz to know how data happens.

A

Takeaway is written in us and there are multiple tracing libraries available. These libraries are compatible with urban tracing or urban telemetry. This seemed to be nice, but we immediately missed some challenges and the most tough one is performance.

A

Data happens very rarely, for example, only once a week. This means we need to trace all requests in order to not miss it and as a key value database. Each request takes very short time only a few microseconds.

A

Thus, the tracing facility must be super efficient negligible compared to a few micro seconds.

A

The second challenge is that there are multiple batch systems in taiqa.

A

For example, these systems receive multiple incoming requests and process them together like in this picture, multiple write requests are accumulated and then a single disk right is performed.

A

Some tracing libraries are not designed for this case. We would like to review all details to resolve these issues. We had to develop our own trading library. I would like to invite you to share this part with you.

B

I am going to introduce our json library, land, miniatures, meaning this step is valid, concise and focus on performance.

B

It's still a poc prototype. So far we are working on letting it be stable and production ready.

B

We design and develop miniatures with the final goal of high performance. Here's the results of microchip micro benchmarks and integration benchmarks.

B

On generation and collection of west bank with 0.02 microseconds, you can see our json library was 17.5 times faster than was tracing and 100 times faster than tokyo chasing.

B

We traced, 100 events by different json libraries then recorded the qps of point game requests. As you can see, while was tracing half origin computers initiates only down 6 percents.

B

I've explained what optimizations we've done for such a performance.

B

The first key to the performance is to reduce contention. Contention happens when a shear resource is accepted concurrently in multiple space or cross.

B

In most class implementations, spans from multiple stress are similarly pushed to the same span collector, which is globally shared stress access and modify the same resource cost contention.

B

They have to pay the overhead of logs or atomic variables for every span.

B

Miniatures doesn't push when span each time to the global speculator. Instead, it collects spans to thread local buffers first after works in northwest autumn spans in the thread. Local buffer are then connected to the global collector in batch.

B

In this way, the global clutter is accessed much less often the contention is reduced and the performance improved.

B

The second key to the performance is to time faster.

B

Let's see how a basis bend looks like for each span, the chasing library will cause when the span is starting and when span is ended. So timing performance is important.

B

Commentation libraries either use system time to retrieve the time or monotonic time counter. By assessing the monotonic clock in our environments, each monotonic clock assessed, causes 25, nanoseconds latency.

B

If we have 10 spans in the key value game quest, the total latency causes by chasing becomes 500. Nanoseconds.

B

Remember that 10kv get requests may take about one thousand nanoseconds to three thousand nanoseconds, so this have a 16 latency overhead.

B

Another choice is to use clock monotonic course.

B

It is fast and results in three percent. Latency overhead, however, is its precision is only four microseconds. According to our benchmarks, which limits its usage.

B

Instead of using these clocks, we use the timestamp counter register available in modern intel and amd cpus. Its value can be accessed via the rd tscp instruction. The tsc register is very efficient with high precision. It only causes 80 nanoseconds each excess in our environment.

B

However, dlc is not perfect in some cpus tlc is not synchronized in different groups, however, synchronized dlc can be discovered by checking some cpu flex.

B

Even with this flex, we discovered that tlc may be non-synchronized due to an unstable environment or some cpu force.

B

Miniatures carefully conduct these situations to ensure that tsc value is reliable and can be used to mature the time when many trades detects that tsc is not available or not reliable, it will forget to use cloud monotonic course.

B

The final key to the performance is to reduce synchronization serialization happens when spent or created in the memory and need to send back to some chasing storage like yoga. There are very frequent key value requests in the hkv.

B

The chasing result, reporting is also very frequent. Serialization may take long time to reduce the seo realization cost in thai kv. We collect all spends related to a request, but only selected report them to the chasing storage.

B

The selection is based on the request. Latency only requests that take long time will be recorded.

B

This is different to the sampled collection that we will not miss any jitters.

B

To chase badge systems, we need to chase the bus to merge contests related to different requests into a single contest. Then these requests share the following child expense. Finally, share spends are created separately to collectors of these requests.

B

We are glad to see some related words in the community. A subset of chain open, chasing, is implemented for performance. Minitrace is supposed to refer to yoga. The amazing echo ui greatly eases our verification. Words.

B

Here is github repository link of miniatures. You can use many traces in your own projects. Now some organization will be contributed to open, telemetry rust. We hope one day the official was the current can adopt all optimizations.

B

The upcoming thai kv 5.0 will support the chasing feature provided by the miniatures hope you enjoyed. This talk. Welcome to contact us through the following channels.