Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2020 - Virtual, 4 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Optimizing Storage Assignment via Pod Scheduling Under Disturbance Factors - Kenji Morimoto, Cybozu

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Optimizing Storage Assignment via Pod Scheduling Under Disturbance Factors - Kenji Morimoto, Cybozu, Inc.

For distributed storage systems like Ceph, it is essential to allocate node-local storage devices evenly among racks or regions. This talk introduces how to automate this allocation by using the "WaitForFirstConsumer" volume binding mode and tuning kube-scheduler. The use of "WaitForFirstConsumer" translates the problem of storage allocation into that of pod scheduling. Kenji and his colleagues in Cybozu utilize the feature of Topology Spread Constraints to distribute storage pods. They found that kube-scheduler needs tuning from the default to spread pods optimally under disturbance such as CPU consuming workloads. Since kube-scheduler is being improved, the tuning method varies according to the Kubernetes version. The talk covers the tuning methods for Kubernetes 1.17, 1.18, and 1.19. By distributing storage pods among racks, they achieved fault tolerance against a full rack failure.

https://sched.co/ekFB

A

Hi everyone, I'm so excited to share our experience of kubernetes management with you, I'm going to talk about the optimization of stretch assignment for distributed systems on kubernetes.

A

We utilize the power scheduling, features of kubernetes and we tune the scheduler under the real-world disturbance.

A

Factors first, let me introduce myself, I'm kenji morimoto, I'm working for cyrus and I have an account morimoto cyborgs at github.com.

A

Cyborgs is a japanese I.t company. We provide groupware communication tools as cloud services and I've been working as an infrastructure engineer. For eight years we have more than 2000 servers for our cloud services.

A

Our management tools were designed as a collection of imperative commands and we are facing a lot of manual operation tasks these days. So we are now renewing our infrastructure using kubernetes and cloud native technologies.

A

This is today's agenda. First, I introduce what we challenged.

A

We tried to run a distributed storage system on kubernetes I'll recap: the basic of volume management in kubernetes and define the problem I'll describe a basic idea for the program and then inspect troubles in implementation.

A

We got an unexpected result under the real world disturbance, so we turned the kubes scheduler configuration.

A

Finally I'll give a demonstration how our tuning improved the placement.

A

First of all, let's take a look at what is a distributed stress system. A distributed system organizes node local stretch devices in a computer cluster and provides a unified strategy resource for users.

A

In the case of ceph, for example, the administrator defines object, storage devices or osds in short, using local disks and safe, gathers osds into a storage resource.

A

The management of the element disks in the distributed stretch is very tedious. Work.

A

Before seeing rook unsafe on kubernetes, let's recap: the stretch. Architecture of kubernetes on kubernetes strategies are selected as persistent volumes. The strategies can be local disks network strategies in the cluster or cloud strategies.

A

Pods can define their volumes by using persistent burn claims as dead sources.

A

Persistent volume claims specify their requirements for underlying strategies through storage class names, resource requests, selectors and so on. Kubernetes finds or creates a matching pv for pvc and bind them together.

A

We use rook to deploy a safe storage. Cluster rook has a mod to configure osds through persistent volume claims, rook configures acquired pvs as osds for safe and then safe constructs, a unified strategic resource.

A

What we wanted to do was to deploy a distributed storage system on on-premise servers.

A

We had experienced that the management of thousands of servers and disks was an awful task if done manually, so we needed kubernetes help for automatic management. However, there was no standard profile to deploy, distributed search system on kubernetes.

A

We can expect that safe is responsible for replicating data across fair domains for robustness. On the other hand, distributing local disks evenly is a task for the administrators, so the challenge here is that we need to distribute persistent volumes for local disks evenly through persistent volume claims.

A

It's easy to achieve even distribution, if all disks all servers and all racks are evenly available.

A

Let's take six disks evenly from this.

A

It's easy, but in reality this is often not true because, for example, a server node may have broken disks.

A

Some disks, some disks of the server may already be assigned for other use or rock may have fewer healthy servers than other racks. So in the real world we need to consider about an even availability of local disks.

A

This makes our challenge a hard one.

A

In order to achieve even distribution automatically, we need help of kubernetes. The kubernetes does not directly schedule. Storage devices, in contrast to storage handling, kubernetes, provides a very rich set of features to schedule. Pods group scheduler is a kubernetes component to schedule pods on nodes.

A

It can schedule posts based on several criteria, including resource requirements, node factors, port affinity and under affinity and taints and traditions.

A

Now jump into the basic idea, our basic idea is to specify wait for first consumer for the volume binding mode in a stretch class. But what is the boring binding mode exactly?

A

It can take up value of immediate or wait for first consumer.

A

Let's see the behavior of kubernetes for each mode in the following slides the brain, binding mode affects the timing when the binding of a pvc and the pv is determined.

A

The default volume. Binding mode is immediate in this mode, when a pvc is created, kubernetes immediately finds or creates a matching pv and bind them together.

A

This binding will work as a topology constraint for kube scheduler when it schedules a port that uses this pvc.

A

In our case, the pvc is bound to a local disk, so the consumer port is bound to the disks node in effect.

A

The problem here is that we cannot control the matching of a pvc and the pv in terms of even distribution.

A

Another volume, binding mode, is wait for first consumer. In this mode. A pvc is not bound to a pv until a port that uses the pvc is scheduled.

A

We can control the port scheduling by several means.

A

When binding a pvc to pv. There is a constraint that the pv must be available from the node, as the pv is not local. In our case scheduling a pod is equal to scheduling a pv, so we can control the location of a local thread through port scheduling.

A

Go back to our basic idea: by specifying wait for first consumer for the volume binding mode, we can translate the problem of stretch allocation into the problem of post scheduling and, as for the port scheduling, kubernetes provides a rich set of features and we can utilize those features.

A

Now our challenge is translated to distributing ports with pvcs evenly.

A

Well, which type of scheduling criteria should we use to distribute pose evenly? One candidate is anti-affinity. We can distribute one port per node by using undefinity, but we want to use multiple pvs on any one. Node undefinity does not distinguish whether there are two ports or three ports or four five bubbles.

A

Much appropriate criterion is port topology, spread constraints. This feature was introduced in kubernetes. 1.16 became better in 1.18 and is now stable in 1.19 a set of participate constraints compute the scheduling score based on the skew. So we can put a cap on the difference of the numbers of pods.

A

This figure shows how topology spread constraints work. This is cited from kubernetes blog. You can specify max queue to describe the degree to which post may be unevenly distributed.

A

Topology key to group posed by node levels, one unsatisfiable to indicate what to do if the constraints are not satisfiable and level selector to find target pods.

A

In this example zone one has two ports and dom two has one pod. The skew between zones is one now here comes a new part. If this new port is scheduled to zone 1, the skew becomes 3.. This breaks the constraint.

A

If the port is scheduled to zone 2, the skew becomes 0.. This satisfies the constraint, so the new post should be scheduled to zone 2.. As a result, we can achieve even port distribution.

A

We can now distribute pose evenly, but when implementing distribution, we need to consider the case where the constraints are not satisfiable.

A

Let's see the details.

A

To keep evenness strictly may not be desirable in the real world, as described before there may be uneven availability of local disks. A cybernode may have broken disks, some disks over server may already be assigned for other use or rack may have fewer healthy servers than others.

A

In this example, there are six servers. Each server has four disks and server 3a has broken disks. It has one healthy disk and three broken disks.

A

Let's start assignment lab1, I can sign six ports with six disks, one per node lab2. I can assign 5 ports with 5 disks. As you can see, server 3a has no available disk.

A

I cannot start lap 3, because assigning a body on the disk will break the constraints, no matter which server I choose.

A

Do I need to stop assignment here. It's not desirable. I want to use as many disks as I can.

A

We can relax the portability spread constraints by using the parameter of when unsatisfiable. This parameter indicates what to do if a port does not satisfy the constraints.

A

According to the optional document, the behavior is described like this. If the constraints are not satisfiable and do not, schedule is specified, group scheduler does not schedule the pod. This is the default behavior.

A

If the constraints are not satisfiable and the schedule anywhere is specified, cube, scheduler still schedules the part while prioritizing nodes that minimize the skew schedule anyway seemed optimal for us if it really worked as advertised.

A

So we tried schedule anyway not to limit resource usage due to uneven local disks.

A

We expected the behavior like this. If the constraints are satisfiable could be scheduler always scheduled, the part within the constraints sounds obvious. Unfortunately, it did not work as expected.

A

Let's start with several ports already running in the cluster red leveled pods are running not on all nodes. Four nodes are used and the other two nodes are completely free.

A

The existing pods cpu resource to some extent say 60, and there is enough room for the new pods we are now distributing. The new port will consume cpu resource five percent.

A

The constraint we use here is a simple one: keep the skew less than or equal to one among all nodes. Please note that this constraint is not applied for the existing red pods. We apply the constraint in order to distribute stretch management pause evenly.

A

The existing ports are just computing. They are not working for storage management, then deploy six poles for stretch management.

A

The expected placement is like this because the constraint is satisfiable, six poles would be distributed evenly. The skew would be zero here, but the actual placement is like this: all new pods are assigned to the unused two nodes. The max skew is three. This breaks the constraint, even though it seems satisfiable.

A

Why are the new strategy management parts scheduled in such a way we inspected the source code of cube scheduler and find the actual behavior?

A

What we expected had, two, if clauses, if satisfiable blah blah blah and if not satisfiable blah blah blah the actual behavior is whether the constraints are satisfiable or not. Google scheduler does not treat the constraining conditions as real constraints. Instead, the conditions are treated as a part of the scoring factors.

A

So, as a result, fluttering cpu resource usage took a higher priority. Even though the spread constraint was satisfiable. The existing computing ports prevented even distribution.

A

This came from the prioritizing algorithm of cube scheduler, so we need to tune it.

A

The most important criteria in pod scheduling, for our case is the topology spread constraints. So we turned clever scheduler to weigh the constraints more heavily, because scheduler is rapidly evolving. It requires different tuning configurations for its versions for kubernetes 1.17.

A

We adjusted the scheduling policy. There is a weight, weight parameter named even participate priority and its default value is one we increased. We increased the weight to 500.

A

The scheduling policy is a global combination, so this modification requires extreme caution in kubernetes 1.18. The feature of scheduling profiles is introduced.

A

Google scheduler can handle multiple profiles. Now we can create a new profile and apply it only to the storage management ports from among the parameters in the profile we set port topology spreads weight to 500 in kubernetes 1.18 in 1.19.

A

The parameters are slightly tuned by default, so disabling no resources branched allocation is suitable for us.

A

I attached two new configurations in this slide. Please check them from the sketch.com link.

A

Skip the configurations.

A

Then now I'll give a demonstration, I prepared two environments, one with default scheduler and then with tuned quest schedule. Let's see the default one.

A

A

And this is not tuned environment. This is a default schedule.

A

A

This is demonstration environment well,.

A

First, let's see the notes, there are four notes here and.

A

Two computing pods are running. They are running on two nodes.

A

They consume cp resource on this cluster. I deploy a distributed, solar system, rook and safe. I already have destroyed them because it takes too long.

A

There are many posts running for rook unsafe, including non-operator manager, monitors and so on. Persistent volumes are assigned for osd ports. I requested five persistent volumes. Three five ports and five claims.

A

You know five oc pods and five pvcs.

A

Well, let's see where the osd posts are running along with where the computing ports are.

A

Five osd ports and two computing ports.

A

As you can see, five osd posts are distributed only to three nodes. Although there are four available nodes.

A

It seems that the reciprocals avoided to coexist with the computing ports. This is because kubo scheduler put a higher priority on flattening cpu resource usage, and this is not desirable for distributed search system.

A

Now, let's move into another environment.

A

The tuned cube scheduler is running here. Let's see where the oc ports are running, along with, where the computing puzzle, five osd ports and two computing ports.

A

Five osd ports are distributed well onto four nodes. Some of them coexist with the computing ports, and this is what I wanted to.

A

Do let's wrap up with key takeaways in order to deploy a distributed search system using rook unsafe on on-premise cluster? We translate the problem of local search distribution into the pawn scheduling problem using word for first consumer volume, binding mode.

A

As for bot scheduling, we use the portable display constraints for better distribution in order to cope with kuberschedular scoring. We too need to prioritize the constraints.

A

Our configuration for rook unsafe is open source in github. Please take a look. That's all. Thank you.