Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2020 - Virtual, 4 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Elastic Scheduling with TiKV - Song Gao & Yutong Liang, PingCAP

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Elastic Scheduling with TiKV - Song Gao & Yutong Liang, PingCAP

In the world of cloud computing, it is well-known that Elastic Scheduling can automatically balance the cost and load of stateless applications, but it is rare to see distributed database systems have the elastic scheduling feature as scaling is always not easy for the stateful system. Yutong Liang, Song Gao and their teammates implemented the Elastic Scheduling feature based on TiKV, a cloud-native, distributed Key-Value database , which helps TiKV automatically handle the burst of load and traffic. In this session, Yutong Liang and Song Gao will share their experience of implementing elastic scheduling in TiKV on Kubernetes and introduce the mechanism behind it.

https://sched.co/ekGd

A

Hi everyone hi, thank you all for being here. We are glad to have this opportunity to share our experience on elastic scheduling, with thai kv during the next 30 minutes.

A

First of all, let us introduce ourselves. I am yutong an infrastructure engineer at pincap, also I'm the tech leader of the scheduling, special interesting group of thai tv.

B

I'm sungo a infrastructure engineer at pingap, I'm the maintainer of mesh and the commuter of thai kvsig scheduling.

A

Yes, we are working on the scheduling system for thaikv.

A

Okay, let us take a look at the agenda of this talk.

A

There are five parts covered for this topic: an introduction to thai kv the elastic scheduling background, the implementation takeaway on kubernetes, some future work. We are working on and there will be time for questions at the end of the presentation.

A

I will talk about part one and part. Two and a song will work you through the product three and part.

A

Four, before what we dive deep into the details about her elastic scheduling, you might need some background information about the tech tv. So what's a tactic. Techyv is a key value database and it is open source distributed and support transactions.

A

Techyv is a simsaf graduated project currently with a great support from community of and the users taking now has more than eight thousand github staff and more than 200 contributors.

A

Now we are going to take a look at the thai tv architecture.

A

Each title node store different data chart named regions which are divided by range for a single region. It has multiple replicas, we call rat group, it is used to guarantee high availability through the right algorithm and each group has its own wrapped leader and other followers.

A

Takeaway client use grpc to communicate with each tycoon node as well as plain smith driver. The cluster manager of thai kiwi and tachyv will send the heartbeat periodically to the placement driver, which is short for pd, to update the information about itself.

A

Pd periodically check replication constraint to balance load and data automatically across node and the regions.

A

And how does pd balance load and the data pd will collect stack every statistic containing type 3 heartbeat to help make a scheduling addition before we come to the next part. It is better to know how taiki will migrate a region through the rafter algorithm.

A

Let's take a look at example, we have three nodes and three region here. Each region has three pairs which are located on node, a b and c.

A

We use the board border to mark the leader.

A

Consider that we add a new node to the cluster pd has a scheduling decision which are going to migrate appear from the high load node a to the newly added node d.

A

We will show the detailed process in next few slides.

A

Assume it selects region 1 as a source region and a node b as the targeted node pd, will first gather the whole scheduling steps and combine them into a command and extend it to thai kb.

A

The first step of this command is to add a new replica in node d.

A

Since region, 1 is a leader we need to transfer the leader before going on, so the next step is to transfer the region's leader.

A

After the leader is transferred, pd will execute the next step to let taiki will remove the replica on node a when the above three steps are down.

A

Now the region has a three replica which are located on node, b, c and d.

A

That's how pd migrate.

A

Region now, let's move on to the elastic scheduling in thai kb. Another question: what is elastic scattering briefly speaking, elastic scheduling is auto scaling depending on the workloads when the type cluster is deployed.

A

Normally, the scale scaling of thai kv cluster is able to handle the average load, but sometimes the workload will become larger, which requires operator to manually scale out the type qva cluster to handle the workloads when heavy workloads get lower.

A

The operator also needs to manually skill in the carcass cluster to save the resource and cost. The elastic scheduling makes the whole process automatic.

A

So why do we need or choose the elastic scheduling? Here are three reasons.

A

First, elastic scheduling could handle the unexpected traffic automatically now today, we're leaving a word with huge amount of information, and the breaking news always appear with no prediction.

A

Here is the statistic about the storage qps for the operator about its business.

A

As you can see, most of the time the qps for the storage layer stays low and suddenly become larger at some point. After a short period of time, the qpis fall back to low level. Again, the changing curve shows that it is difficult for the operator to manually manage the storage layer, because the traffic is unexpected.

A

Second, elastic scheduling could help us give the results when we don't need it, as the traffic is unexpected and the operator needs to need the ability to handle the heavy workload they have to pay the cost for actual for the most of time. These resources are wasted.

A

If the story has ability of elastic scheduling, the results could be released when the workload becomes low.

A

Finally, the cloud infrastructure has already become virtual nowaday and the elastic scanner can benefit from it. As the cloud infrastructure provides on-demand availability of computing resources, the elastic scheduling could apply for resources from the big cloud infra when they need it and release the resource at its own pace.

A

Moreover, as a as could as a kubernetes have has provided the powerful api to manage container and resources on cluster, it is more convenient to manage the stable, stateful application like distributed database. For now, we have introduced the background of thai kp and elastic scheduling.

A

We also discussed why we need or choose the elastic scheduling for the storage layer. Now we will have a song introduce the implementation in thai tv.

B

Okay, thank you. Tong, I'd like to move on to the introduction of how thai kv scheduling builds, is the last its elastic scheduling management mechanism based on these solutions. First, introduction about the elastic scheduling architecture on tai kv.

B

The architecture is composed of four components: first, the kkv cluster expose its metrics of its status according to the workloads. Second, the monitoring system like promises or collects these metrics. The scheduling system, which is called pd here, will fetch the metrics from the monitoring system and calculate the auto scan plan which is exposed by the api.

B

Finally, the operator system will take the auto scaling plan provided by the pd and scale the techyv cluster. Now we will explain how this architecture works by two parts: the operator side and the scheduling side.

B

We use operator pattern, a famous method of packaging deploying and managing the kubernetes application. It is basically composed of two components: the controller and the custom resource definition, which is also called crd.

B

We use crd here as the elastic scheduling configuration for users after the configuration file is deployed into the cluster the schedule. The controller will start to check the configuration fire periodically and carry the auto scaling plant from the pd.

B

As the operator manage the tag v cluster on the kubernetes after the operator gets the scaling plan from the pd. The operator will start to scaling the tachyv cluster. According to the plan.

B

On the scheduling side, the pd meaning do two things. First, fetch the metrics from promises as tacav has exposed many matches in lots of dimensions. The pd could use this information and calculate the proper, auto scaling plan for the operator.

B

The second thing for scheduling is to decide the scheduling strategy in this elastic scheduling situation.

B

Before we introduce this specific strategy for this situation, we can imagine that what if the pd take the newly created elastic thai kv as a normal node, like normally scaling out as pd, would balance the data distribution between each tv, it will transfer lots of regions to the newly created type kv, as the data transferring need called system resources and also could cause latency.

B

The default balancing scheduling strategy couldn't work very well for the scheduling elastic scheduling to solve this problem, the pd would only transfer the regions which is under high frequent visiting or updating, and their located type kv should be under high load. We call these regions as hot regions, so how do pd only migrate? The hot regions to the elastic scattering type kv, node.

B

Before we answer this question, we need to know how we recognize the hot region. Pd will record the read and write flow of each region and stores each store will maintain a top-end cash to save the most hot region in this store.

B

If our region is beyond a predefined threshold and continuously heats the cash, we think this is a hard region. With this information, the hot reading scheduler in pd will decide if migrating the region can make this 10kv more balanced.

B

Since we have known about how pd recognized the whole region once the hot region is selected, the last thing is to find a way to only migrate it to the new lake scaling node tag tv supports using some predefined label to manage the data placement, so we can utilize this mechanism.

B

Actually pd has many kinds of schedulers working at the same time. Besides the hot reading schedule, when these schedulers start to schedule, we usually use some filters to filter the stock which don't want it to be a source, a target according to each scheduler's rule, and thus we can create a new label for newly scaling nodes.

B

The label consists of key value pairs. The key of the label is special use and the value is hot region.

B

When the node is scaled out, the label will be added by the operator. Automatically pd will get this information once the new link scaling node sends its heartbeat for other schedulers, such as balance readings. Before we create them, we add a label filter which use this new label to it.

B

This filter will prevent the store being selected as target. For example, we only allow the hot reading scheduler to select the store 1 as the target. The.

A

Rush scheduler.

B

B

Here is an example of how elastic scheduling configuration looks like, as you can see here, the configuration point out the necessary factor and the patterns the pd needed during the calculation during calculation, the auto scaling plan. We can also define some concentrates here in order to limit the total counts of scaling tac kv nodes as.

A

The configuration.

B

Is managed by custom resources in kubernetes after the configuration is deployed, the elastic scheduling will start working.

B

Now we will shoot experiment results about the elastic scheduling on thai tv. In this experiment we will use tai db an open source distributed database which used thai kv as storage layer and run the suspense, a famous benchmark tool to show how elastic scheduling work during the heavy workloads.

B

This graph shows the relationship between the regions and the type kv. The column indicates the status for the dedicated kv, and the rows shows that the region and its replication in each tai kv. The blue rectangle represents a follower here for the region and the graph rectangle represents the leader for this region.

B

As the thai kv cluster is consuming lots of cpu resources during benchmark. The elastic scheduling starts to deploy two new thai kv automatically at the beginning of the elastic tech. Av is created, there are no regions being scheduled to the elastic scheduling thai kv, so we can see the sysbench result didn't change yet.

B

After the pd transferring some hot regions to elastic tachyv nodes, as you can see, there already exists some red rectangles under the elastic techniques column.

B

Now we can find that both the tps and the qps of suspense have increased. The whole process shows that the elastic scheduling automatically detects the workloads and the scaling the cluster to improve the performance.

B

In addition, we are going to support more features with elastic scheduling in the future. Here are some ideas.

B

The first is to dynamically increase. The read-only replicas according to workloads, as we know, kv, only can read data from the region's leader in the earlier version, but now we support follower read and thus we can read data from followers.

B

Also, when the reap pressure is high, it will increase the read-only replicas automatically, which can greatly help us reduce the weight pressure. When the read pressure becomes low, it will switch back to the original state.

B

The second is in some seniors: there is some data which is barely accessed.

B

It's better to move this kind of data to the cheaper storage in order to save the cost, combined with elastic scheduling, we can make the cluster recognize this kind of data automatically and decide to scale out the cheaper storage and scale in the highly paid.

B

Storage, this is basically everything we want to cover in this talk. Thank you very much for attending. If you are interested in what we are doing, you are welcome to join us. If you have any questions regarding what we have talked, please ask.

B

Thank you very much.