Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2022, 11 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: How We Make TiKV- a Distributed Storage, More Cost-Effective On the Cloud - Sunny Bains & Yang Zhang

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

How We Make TiKV - a Distributed Storage, More Cost-Effective On the Cloud - Sunny Bains & Yang Zhang, PingCAP

Speakers: Sunny Bains, Yang Zhang
As more and more online applications and data migrate to the cloud, cloud-native, low-latency distributed storage systems have played an important role in the growth of these companies. TiKV is one such storage system that has been used by hundreds of companies for their mission-critical systems. Although the infrastructure on the cloud has the advantages of elasticity and pay-as-you-go, all resources on the cloud are paid for, including computing resources, storage resources, gateway routing, and cross-AZ traffic. So when building a system on the cloud, you need to consider a variety of things that make the system cost-effective. First, I will briefly introduce some cloud infrastructure and its typical characteristics, local disk and cloud disk, layered elastic distributed file system, several types of computing resources, etc. Next, I will introduce how TiKV reduces data storage space, reduces write amplification, and reduces the amount of data transfer fee across availability zones, etc. Finally, we look forward to several potential ways that TiKV can improve cost performance on the cloud. I hope that some practices on cost-effective optimization in TiKV can give you some reference and inspiration.

A

So I work on the storage side of things on Thai, KV and prior to this I used to work for Oracle on MySQL and I was the lead on innodb and recently I joined incap and my experience is limited. But I do work on the storage side of time KV, and this is my colleague Andy he's been working longer on Thai, caving and so he'll. Take it from here and we'll talk about. You know how we make Thai caving, more cost effective and that's.

B

Okay, uh hello, everyone. um My name is Andy uh I'm, a software engineer of pin cap and I work on TI TV prior to pin cap I worked for Google, spanner team, and so here I am. um Our topic today is about making TI TV our distribute storage engine more cost effective on the cloud.

B

So here's the agenda at first I'm, going to give you like a brief overview of GI TV and then I will show you some of our own experience, building a cloud service and yeah. That's it and then last I will give you some examples like how we do like Cloud cost effective Works um and as well as some like Works we're doing and they will be down in the future.

B

uh So first, uh what is Thai KV So Thai KV is an open source uh distributed distributed and transactional K Value Store, so it's written in Rust program programming language from day, one uh around like seven years ago. So we were so glad we make the right decision and it was a tough decision to make and since as cubican and cncf Conference I'm honored to mention that pin cap donate tiqv to cnsf Foundation like three years ago and now as a graduated product of cncf and I, gave the link below that's our GitHub link.

B

You can search it online.

B

So you know tikv, as a storage engine has been um globally adopted by many users in many different scenarios. So here are some examples. um You know we are infrastructure for infrastructure. In some cases like there's, a file system called juice FS they use TI TV as their like metadata storage and the the TD Cloud object. Storage also use us as their metadata stories.

B

um So there's also like a blockchain company called like Harmony Harmony. They use TI TV to store the blocking data as well uh I guess it's probably the super super node or something um I'm, not sure like whether you guys play mobile games or not, but uh I guess you know like Pokemon go uh and the company behind Pokemon go Niantic. They use tikv as a like transactional key Value Story.

B

um So you know we are open source uh project, so we have a large community and, like many community members, did a lot of great things like around techyv, so they build a radius protocol on top of Thai, TV and they're called like Titus and Titan uh and of course, like pincap as a company, we provide a like a a solution like a query: uh SQL uh Mass QR compatible database that is called tidb, that's our main product and it uses GI KV as an underlying storage ending.

B

So yeah this is the like overview of techyv architecture, so this entire TV is partitioned into model shards and each charge is like replicated across a multiple nodes and a rough algorithm is used to maintain the consistency between replicas.

B

um So here, like uh a region, you can consider it like a graphic group. I know it's a terrible name. Some some people like um make misunderstanding that, like misunderstanding, this with the cloud regions- but this is a rough reading- a rough group- um and you know like uh in addition to Raw kiwi uh apis, will also provide uh transactional apis and the traffic transaction implementation. We use uh is called percolator and there's a paper about it and you can search it online.

B

um So through this architecture, tikv like skills to Paradise of data effortlessly- and you know, more importantly, um all those scaling uh rebalancing works and it's all done by techity itself automatically so in case you're, curious and and if we zoom out the kikv architecture a little bit. uh Here's the tidb architecture, the whole setup so as I mentioned before KBB is a query, query ending and sitting on top of the storage.

B

And similarly, we also have like another query: engine called Ti spark, so you can write some spark uh like commands or query uh like using spark um syntax um and on the side we have a component called placement driver, uh that's sort of like a centralized service to store some of the metadata of our system, and you know there's a new thing in our system that is called Ti flash, because tikv is a raw like they're, old, role-based storage engine.

B

So it's I mean for like oltp workload and there are more and more customer um like um one like olap system, and so we built a component for that scenario that is called hot flash and, as uh in column, um based data structure.

B

So tiqv or tidb can be deployed both on print and on cloud. uh You know like actually many of our bank users like big Bank users. They prefer to deploy Tech on their Data, Center and tacky. Beyond that um I mean um it's okay, but I have to say it's hard to to support them because, like when anything happens, we have to go to their place. To do some work. It's not a scalable solution. Well, the good side of of this is that so they're very rich.

B

You know, there's big customers they're very rich, so they don't care about cost very much. They give us the hardware and we use it and like they don't ask any questions about like do you have some room left? Is the resource enough or now yeah? uh Well, um but now our focus is cloud, um so you know we're a cloud native database.

B

So by the way uh we are deployed on cloud using kubernetes uh I'm, not sure how familiar with uh how familiar you uh argue with the kubernetes, where you use kubernetes operator pattern to deploy our product.

B

So the good thing about cloud is is a more scalable business, especially for ourselves, uh but you know when we provide our service as like um software as a service on cloud, we have to care about the uh like cost and the resource and efficiency. All those things.

B

You know, like everything, um is now free on cloud right, um so you have to pay for the computational resource like virtual machine and ec2 instances, like instance, and you also need to pay for the storage, and um you know like, for example, the EBS you're, not only paying for the space.

B

You are also paying for the number of iOS um and like a bandwidth, something like that, and you know um if you, if you have experience building a service on cloud, uh you will know that uh there's like a diff uh price difference between cross region, traffic, cross, Zone traffic and local network traffic. So there's many things like specific on cloud, so what we can do to reduce the cost or make the resource more efficient.

B

So here's are some examples, I can't think of, uh for example like uh and the feelings level. You can do some same plan because you know, if you do your budget planning ahead. You'll get a discount just like like a regular normal, normal life and the technical level. um There are much more work. We can do like operational work, uh so if we can reclaim or shrinking down the unused resource, we save a lot of money. But in this talk in our talk today, I mainly want to focus on like architectural changes and optimizations.

B

We did um like to our main component, which is attacking so on the right here. You can see like a resource analysis of uh KB or tidb on cloud um like 75 75 of the like, but it goes to like ec2 instance, for example.

B

um So how? How can we uh like make the resource more efficient right um so for computational uh resource we can? In general, we can write like more efficient code. That's the most effective one in probability, um but we can also like uh do some. uh Where, like uh reducing the unnecessary process uh and for storage, uh we can um like, we can have a system with a smaller size.

B

We can reduce the number of iOS or have like a storage service storage service with less uh or with a smaller bandwidth and for Network that, like like I mentioned before, they have different like past difference. uh So we want to reduce our long distance traffic as much as we could.

B

So. First, computational efficiency, uh as I mentioned before we used rough algorithm uh for data replication or more specifically, uh Mafia raft. That means each rough group stores a piece of data and the whole data set of Thai TV is stored by many rough groups, so I'm sure that uh you know you can imagine uh inside each rough group uh other peers. They have to communicate with, with each other to make sure that they are in sync right and this process is called uh heartbeat.

B

You don't need no like more details about it, but I'm sure you can understand this, the heartbeats so well. If you go with naive implementation of graft, you will find it not super efficient. Why? Because, when the dates that uh like becoming large and there will be like millions of rough groups in your system uh and well in reality, you'll know like most of the, uh like most of the uh groups store code data right.

B

So what we can do is really to putting those in inactive groups into sleep so and we call them uh hibernated regions internally. So, by doing this, we were able to gratefully save a lot of CPU resources, and you know, uh of course, uh some networks as well.

B

So that's what first thing we did like optimizing: the modular process.

B

um Second, um you know you know I see many of you guys like use MacBook, uh so probably running, Apple, M1 or M2 and I have a laptop laptop which is running on uh Apple M2.

B

um So you, you all know that the power of arm uh the new, like not new, about like the the other uh CPU architecture, um so you know like we have been trying to migrate our service from like a x86 to arm, and our testing shows that by using arm we can save like around 20 of the planet but achieve the same level of performance. uh Of course right. So if the performance is not the same, it's pointless.

B

uh That's about like computational resource and next is about storage efficiency.

B

So here is a graph showing you like uh the life of a right in techery, um so we just talked about the heartbeats uh like just now and that's more like the rough core, and this is more like the rough application.

B

So we can also divide the whole process into three steps. So a request is coming to the leader of the rough group and leader replicate the rough lock to his followers, and all the followers will like uh persist, the rough lock in their local storage engine and we call it like rough storage and then there will be another like process called log applying process and they will Fast log from the local rough storage and do some processing and then put the real user data into the real user.

B

uh Real user data storage, ending so uh I guess like we can tell that there are mainly two types of types of I o here right, one being the rough lock and the other one being the real user data. So we will discuss them separately.

B

So first, uh let's see what we can do here uh to the real user data step.

B

So you know on each node techy we like on single node technique, uses rocksdb as a local storage engine um rock DB is awesome. 3D, structure, implementation um and the our asymmetry structure looks like this roughly um so a right coming to a awesome tree will be put into both the mammo table, which says in memory and as well as the red HELOC, which says in the desk and like when the memory table grows beyond a certain limit. It will be flushed into the desk, uh like uh in sort of string table as as key.

B

uh Well, you know if the engine crashes, there will be some data in the map here, map Table and there will be loss right and that's the time when the red hat LR kicks in. So the recovery process will read the log and rebuild the map table. So it's a this is like a typical uh our asymmetry implementation.

B

Well, in reality, you know in our system, rough log like actually plays the same role as a rat catalog uh like native rap dialogue of Rocks TV. So, in our case, you know uh what we can do is instead of using uh the native halog of rock CB, we can just use uh rust reflog.

B

um So you know this optimization is actually kind of like a no-brainer. uh You may ask me like why didn't we do this? At the first place, uh I will tell you like two reasons: uh one choose and One Lie. The first reason is um using roxdb native redhead log. We can get faster recovery time and the second reason is we don't have them in power. You can guess which one is true, which one is why?

B

um Okay, that's about like Israel, third data, stop and um ask for the ref log. You know in the past. Actually uh we store both user data and graph log uh in the same Rock TB instance. So um you know our asympt3 or rocksdb in general uh they have a like better rad performance.

B

Comparing to B3. Well uh I'll. Add some tree structure. You really needs some background activity to do some work. You know to make sure the grid being performing and that's called compaction and there's like a well-known problem of this process.

B

It's called red amplification because uh you know you flash the data from the member table to a sort of stream Table and there will be like a lot of files set in the desk and, yes, sometimes you have to merge them together, like read them load them into your memory and then merge them together and right back to the disk.

B

So uh like one single, like one single entry will be right into the desk multiple times because of this process that and that's called Riot amplification and you know, but one uh character rough lock has is uh they are coming in strict sequential order. So what does what? That means is that we don't really need uh the merge functionality or the compaction functionality of our asymmetry. In this case, um so we wrote Our Own uh like a new storage engine. We call it a rough ending.

C

A

B

So through rough ending, we were able to reduce the total number of I o by like 20 to 40, that that was like a huge win for us.

B

Yep, that's about the I o so for a network you know like, as mentioned across region, traffic and cross-long traffic are significantly higher than local network traffic. So you know our works here are mostly about reducing the long-distance traffic, um so Shia, kvn or tidb. As a distributed system. We usually deploy the whole system in different geographical areas and Scatter the replicas like one replica per Zone.

B

You know, if that's the case, if, by like by default, there's no much room. We can improve this process because we have to replicate the data right at least once but again in reality, some of the users they have, what more than one replica in the same Zone a typical use case is, as mentioned, we have new component top flash, so sometimes they deploy the type Flash in addition to Thai KV in the same zone. So in this case you know, if we don't do anything about it.

B

This two replica will like may try to gather own data from the leader which is in another zone right. um So obviously we can optimize that so, instead of like getting data from the leader. At the same time, we can force the second replica in the same Zone to get the data from the first one right, just cop. The data across zones once.

B

And similarly, the same principle can also be applied to the Reflow as well, so like one sentence like read locally as much as possible, uh but this requires uh utilizing the follower read functionality and provided which is provided by mvcc, which is like multiple version. Concurrency control, plus rough algorithm.

B

uh Give you here's like a summary um so um for computational resource, uh we optimize it by optimizing the multirapt, Harvey, flooding problem and well we're also able to reduce the cost by using arm and for storage. We write our own wrap log engine car, it's all also open source and using that gives us like, uh like uh 10 to 40 percent last.

B

I o and some computation computational advantages as well, because we don't need to do the connection um and uh we substitute a roxdb record log um that saves that saves us a lot of I o as well, um then for Network. We, the main principle, is copy data only once across the domes.

B

So here are some Future Works. We are planning to do basically leveraging more cloud services, for example Cloud Lambda, um so it's sort of like on-demand uh Computing framework, so we can use that to do compaction and, secondly, we can use EBS snapshot functionality to do the backup, uh and at last you know. Currently we we mainly use EBS. We don't use S3 S3 has like large throughput, but the performance uh is not good, but in the in future.

B

If we can distinguish code data from the hot data like very accurately, we may use S3 yep. That's it that's our topic today. uh So questions.

A

Thank you, Randy.

B

A

Anyone has any questions. I can then walk around with a mic and not sure.

D

Thank you. Thank you.

C

D

um So for the keeping traffic in the same AZ, if I were say, deploying my own TI and KB, um is there a way that I would need to configure that to do that.

B

uh uh No I guess uh hour work yeah.

D

Well, I'll do it yeah. So it's just aware.

B

I think there's one word uh we may need to cooperate, which is to label the server right, yeah.

D

So you tell it which zone servers.

C

A

You yeah it takes care of that. Okay, that's cool! Thank you. Anybody here.

C

Thank you, I have two questions. First, is um to to reduce the heartbeats of raft, those those heartbeats are for different consensus groups, and they have different, for example, terms. So how.

A

C

It actually work like who are the sender and what information does it combine together and who are the receivers I? Imagine that will be a every leader or basically every node, because it must be a leader or it's very likely to be a leader of some consensus group to send everyone else. Yes,.

B

Exactly so, the picture I was drawing. There uh was like from a per group perspective.

B

Yeah, that's just one group yeah, the leader sends Harvey to it to his followers right, yeah, and uh so what we did was just to stop the heartbeat process.

B

uh If there's no like rights into this group, but if there's requests, then then we have to resume the whole rough process.

C

Oh, it's not too piggyback a bunch of hobbies into one message: it's more like reduce the heartbeat interval for the idle groups right.

B

What do you talk about is another optimization yeah.

C

Yeah, oh gotcha yeah. Thank you. um Oh um I, think that's it. Thank you.

A

Okay sure anybody else.

B

E

Sorry I joined the late join late, but how can you describe the architecture? I mean if I need to kind of use this in my AWS tendency? How would I use it? I might have missed certain things. Certain parts.

B

uh Yeah, so how to use like technology.

E

Explain the architecture when, if I need to use your service.

B

Right, oh okay! In my you.

E

Know the tenancy, how would I use it.

B

uh I think this is what you are looking for. Is this uh and, uh uh like I said before, um so uh we are being deployed on cloud using kubernetes, uh more so more specifically, kubernetes operator pattern.

B

um Yeah, this is like the whole system and by the way we also have uh like our own deployment tool, called tied up tiup.

B

If you want to play with it like in your local environment, you can use that as well right on cloud. We mainly use kubernetes.

A

Right I would encourage you to use, tie up, and so it will actually even download the binary for you once copy it around everywhere and then start them all up and get them talking to each other.

E

A

C

Thank you another run um so to uh to allow the type flash node to fetch the data from other followers.

C

um It's pretty different from the raft protocol, where the leader sends all the messages to the followers in the push manner. So I I guess the type flash notes can only fetch the committed data in the learner mode is.

B

That how it works yeah exactly it's all done here.

C

Gotcha so for followers they have to get the message from the from the leader directly, so they cannot follow.

B

Each other yeah I mean they did implement the patient detail wise, there's like there are like many more works yeah. We us. We have some other like internal component car like a rough proxy right, yeah yeah that but that's lazy work a bit yeah.

A

Yeah good question anybody else.

A

That's it great organic thanks. Okay,.

B

Thank you so much for coming here.