Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022, 1 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Adapting TiKV for Cloud Storage - Xinye Tao & Jinpeng Zhang, PingCAP

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Adapting TiKV for Cloud Storage - Xinye Tao & Jinpeng Zhang, PingCAP

TiKV is a cloud-native key-value database built in Rust. As a distributed storage layer, the underlying storage hardware plays a key role in how it performs. This session will start with an introduction to the modern cloud storage stack, highlighting the challenges and opportunities that come with it. After that, we will deep dive into several new features aimed at improving TiKV's service quality in the cloud, including Raft Engine, Prioritized I/O Rate Limiting and [...]. Finally, we'd love to get feedback as we lay out the future plan on bringing TiKV even closer to the cloud.

A

Hi everyone- I am tau xin yue, I'm a senior engineer from pincap, which is a company behind thai tv.

A

In the past few months, my team and I has putting a lot of efforts to make tech tv work better on cloud storage and today, I'm going to share with you some of those new improvements that we are very excited about.

A

Just before we dive in, I think we should get a basic understanding of the situation. First, let's take a look at techyv long story short: it is a distributed storage engine, unlike traditional storage engines, that serve on a single machine.

A

It have the capability to scale out normally to hundreds of nodes, and we need to replicate both wall and data files to provide high availability, which is actually one of the reason that we are willing to tolerate the complexity of a distributed system.

A

Now, let's move on to the hardware design nowadays, most public cloud vendors provide virtualized disks and those disks can be mounted to a local file system and they appear like a local disk as well, but internally they're forwarding the ios to multiple remote disks that are potentially shelled by multiple users.

A

Ebs, for instance, will replicate any right ios to three different locations, and this internal complexity can be a real problem for our systems, because the latency for stutter is obviously higher than a local disk.

A

Also, since you are sharing hardware with other users, anything that you use will be charged that includes disk bandwidth and iops.

A

Finally, we should all know that cloud infrastructure are not always as reliable as they claim it to be. Service degradation will be relatively frequent and it should be considered in our system, design.

A

Ideally, we want to make a large type v cluster behave similarly to a traditional rdbms.

A

Unfortunately, it's actually hard to accomplish that on a cloud storage. First, we want to build a scalable service. The scale means more failures. To be more specific, we are worried that as the system, this hardware, the storage hardware performance, is more likely to degrade because on a larger scale, a real event will not be so rare anymore.

A

The cost is a problem too, because every storage operation is charged by the cloud vendors. The users now have more reasons to care about exactly how and why our system is using those resources, and by that I mean read and write amplification, which is the amount of files the system needs to issue. Before finishing one user request.

A

Here, I drew a simple graph to demonstrate our system's runtime usage of io resources over time. The user rights are very stable, as you can see from the yellow bar there, but they are amplified multiple times because of background rights, and that includes compassion, compaction and garbage collection.

A

In addition to that, large events include actual ios that are usually not so predictable for this graph. We can see that the most straightforward way to reduce cost and improve scalability is to keep our usage under the hardware watermark at all time.

A

And for that we introduce two new features which I'm going to talk about in details.

A

First, we have raft engine, it is a new log store for thai tv that is written in rust, just like tech, tv and for those who don't know it. We previously used rocksdb to store all transaction logs.

A

Clearly, it's not a optimal choice, but it is a decent solution at the early stage of development, but right now we want to replace it and improve it. The primary goal here is to write less than the rock cb. Consequently, we can reduce I o cost and reduce the possibility of heating storage performance limits.

A

Of course, we have a secondary goal. Here is to improve the performance as well, but it is not being actively working on at the moment and we are really hoping that more contributors can join us to improve it in the future.

A

Now, let's talk about how exactly we accomplish the primary goal, which is actually very simple graph. The engine maintains an email index of all log entries. The reason we do that is not to improve, read performance.

A

It's actually about reducing the background works in rock cb compaction is needed to keep all data sorted and clean up deleted data, but in rough engine we don't need to sort anything, and garbage collection doesn't need to read out obsolete data, because we already have a map of all active data in the memory and then graph engine further reduces foreground ios by compression all log entries are compressed with lz4 before they are actually written to the log files, mainly with those two techniques.

A

We are able to reduce nearly 30 percent of all server right and in practice. That is a very improvement, very good improvement that we can use.

A

After that, we have another feature called priority: io scheduling, it is not a new thing. Many systems already have it. What we managed to do is to add this functionality without introducing major change of architecture.

A

By that I mean we did not change the internal tasking system of taikvi. No additional io queueing is required, and no extra overhead at all the algorithm is again very simple. We first trace and categorize all system ios into three different priorities.

A

We can call it a b and c here during the execution we periodically assign individual io limits to those priorities. In the beginning, the io limits are all very high. All ios run with no restrictions.

A

Eventually, as you can see here at apple 2, the I o usage exceeds a predetermined global limit, which is what we call an overflow after the overflow airport.

A

We will adjust the io limits for lower priorities to make sure that in the next airport system will not use so much io resource, and here in specific, we will make the I o limits for priority b and c much smaller than apple 2., and after that, after that, in apple 3, the global I o usage of the system will be decreased under the predetermined limit.

A

As you can see, the algorithm is not perfect: it tolerates short period of overflow, but in practice it works exceedingly well.

A

Here internally, we conducted a test to simulate large events during online workload. In this test, a large table is mounted is imported, while a tbcc workload is running after applying the priority io scheduling, as you can see, on the left hand side, the system performance is much more stable than before.

A

Well, that's pretty much all the features that I want to cover today, other than that there are a few more important things that we are experimenting with.

A

The cpu limiting feature, for instance, is a new strategy that we are pushing for low resources environments such as four core machines.

A

Basically, we want to smoothly apply that back pressure to the user before system is overloaded and the raft witness it is an attempt to reduce replication cost by using a write, only node that only replicates transaction log, but no readable data in the future. We want tech tv to further adapt to cloud hardware, and we truly hope that the community community users can benefit from our work here.

A

So thanks again for joining me here today and goodbye.