Cloud Native Computing Foundation KubeDay Japan 2022, 9 Jan 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Capacity-aware Dynamic Volume Provisioning For LVM-based Local Storage - Satoru Takeuchi, Cybozu

Description

Don't miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from 18 - 21 April, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Capacity-aware Dynamic Volume Provisioning For LVM-based Local Storage - Satoru Takeuchi, Cybozu

This presentation introduces a CSI plugin named TopoLVM, which supports capacity-aware dynamic provisioning for LVM-based local storage.

Kubernetes applications usually use network storage to run their pods in arbitrary nodes. However, there are some cases where local storage is better than network storage (e.g., users prefer performance and/or cost-efficiency). TopoLVM is created to use local storage as conveniently as possible.

Satoru will show what kind of features TopoLVM supports:

- Dynamic volume provisioning
- Topology-aware pod scheduling
- Volume resizing
- Thin-provisioning

In addition, he will also introduce how to use TopoLVM and how it works.

A

So our research uh today I introduce you to the uh Rock CSI. She is a storage driver or lbm based local storage. On this. This driver has a capacity aware: Dynamic boring provisioning future.

A

So it's agenda, so first I'll introduce you the motivation to develop our storage driver to power lbm and the next I should introduce what is Topo IBN and how to programmingham works and at last I'll show you the next step or next plan of the OBM.

A

So, first motivation, so let me introduce our company's eyeballs. Cybers is a leading scroll, Subs provider in Japan, and we provide software that support teamwork. So this kind of software is also known as a groupware in Japan, so the sideboard has a kubernetes cluster. It's on premise: kubernetes expressed acrossed in our data center, so it's under development, but we will use this cluster to host our service in the after two or three years, two or three years later- and we are in this cluster.

A

We also have uh have a storage system and on top of this kubernetes cluster, and we have two kinds of storage. The first one is distributed, block and object storage, and we also decided to use the rook and save for the the distributed storage. The second one is a local, fast storage. In this case. Our first means the local nbma, SSD storage.

A

So at the beginning of our development, we need to select the best best storage driver who are localhost storage.

A

So let me introduce the requirement for a local stretch, uh Cyborg's local stretch, so we have two requirements. First, users can create arbitrary sized volumes. Typically, a local storage is our disks or partitions, and they are, they are fixed size so, but fixed size, disks or portions are increased about fixed size. Volumes are inconvenient, especially for kubernetes users, and the second requirement is volume should be spread over nodes based on free storage capacity, in no matter what use storage capacity for each node a movie to utilize the whole whole local storage.

A

So we just, we should decide, uh we should decide and to use what was the best storage driver? What is the best demonstration driver to use our kubernetes cluster? So unfortunately, there was no CSI driver that met our all our requirements, so we decided to create a new CSI driver, Topo lbm, a very simple story. She has no driver, we create driver, so next I'll introduce what is Topo lbn about arbitrary volume sites. It's the uh One requirement of it's a first requirement.

A

Volume volume groups prepared on each node and the Topo WM create an IBM logical volume for each passing, the volume resource and uh probably you're a bit confused, because what PV means, but in this case uh PB, is in this case and in this slide people is not means everym's physical volume, but percent volume.

A

So I'll, let's see the top side of this. uh This figure and it means it means the kubernetes resource. There are three are passed in the volume and the bottom size means Hardware configurations. uh There are node one and another two two nodes, and there are uh one one BG for each node and this uh volume groups are prepared by administrators.

A

Doesn't touch volume group and there are. There- are logical Burns for each percent volume.

A

So race is a port scheduling and volume provisioning of uh or um so the rest is, the top side of kubernetes resources, and there are three kubernetes resources: Port, PVC and storage grass. The first airport consumes are PVC that requests the 10 gigabytes and this PVC points to the storage grass for a couple LVN. That means this volume is created through the uh LVN 1003 nodes, no zero node one node two and the uh notes there has 15 gigabytes, uh free, uh BG, webg storage and the node one is 100 and node.

A

Two are five: uh just five gigabytes expect debris so retro as shown in this case, uh two I have two assumptions. So PVC is dynamic volume provisioned in this case, so it means rpb and the people are corresponding to PVC, will be created at Port scheduling. Not uh so it means in this figure. People is not exist yet.

A

So then uh next assumption is that storage cross volume binding mode in this case volume, binding mode, should be wait for first consumer. So when we create the PVC pivot is not created, so a paper is bound to a PVC output scheduling, so both PVC PVC and people. No, no PV is created on Project scheduling, so of course, Topo VM also supports the not Dynamic so as static volume, provisioning and volume binding mode immediate. But it's the best choice for to use Dynamic volume, provisioning and wait for our first consumer.

A

So as a result of our top European volume, management and product scheduling and support is scheduled to the node having the largest free budget space as possible. In this case, Node 1 Node 1 has 100 gigabytes. It's the largest value among these three nodes and the volume is provisioned on the same node as the port volume and the port are scheduled to node one both scheduled to both one and what is on node one and everybody is created on this.

A

This volume group on Norwell and it's bound to this port and the people and PVC and Report resource.

A

And that's it, and that was the main feature of Topo VM and the topper win also has many other features. So first one is the boring type for uh fire system volumes. uh We on top LVN support, ext4, XFX and paraphrase so. It covers the most domain. Three wire systems in Linux so and the Top Value also supports low block volume.

A

uh In our case, um Topo VM is used for getting uh getting their storage for a drug safe, distributed storage, so we also uh topiram also support other features like generic ephemeral volume and the volume expansion and machine volumes. Shin volumes also support the snapshot and the scene snapshot and shim Chrome features.

A

Now, interestingly, uh this shinboarding future is not was not created by cyborgs what was created by RedBot, so they um they need to implement this feature.

A

So how about Community or top lvm? There are many non-cybers users and Developers I guess there are over 100 of 200 people, the top of user, and there are uh over 10 people. uh 10 non-sibles people are developing the developing 2.0 VM and some companies use in their products. So uh it can be said that the top lbm is used by used in the production environment, not in the not only in the cyborgs, but also in other companies, especially uh one good example is about red, hot red, hot openshift, embed top lvm in their openshift.

A

So the one very interesting story that the first patch from red to heart is about to support their mainframe system in topper, BM I was I was under my album. All my teams. Members are very surprised because I don't I, we don't have memory system how to test it. This much so after discussion after discussion with uh topography, maintainer and our record developers, we decided to merge this patch uh two thousand terms. We don't test it. We can't test it, but uh probably uh representative test uh just dispatch in their environment. No problem.

A

Yes, so it means that the openshift and the kubernetes cost can be run on on top of Mainframe very surprising, so it also means the top, and there is a Topper lbm container working for Mainframe I didn't touch it.

A

So next topic is how to buy William works. So there are two challenges to implement the art of Arabian airport scheduling and uh boring, boring provisioning. The one is the schedule report to the node having a large free busy space as possible. To accomplish this purpose, we decided to use the scheduler extender of kubernetes and it it's a optional kubernetes, which and second challenge is provision and provision the volume on the same node as the port.

A

So we we use our CSI topology. It's it's also a official feature of kubernetes. So, but these future are optional so and the user, or in this case topiro, VM or CSI driver, can Implement their logic and to use a schedule, extender and CSI topology.

A

So I will introduce the scheduler extender. So let's see the bottom side of this slide. It means the step and the step to schedule are put into some into a node. It's a group Cube schedulers work, the first just start and start Port scheduling and in the Second Step uh filter out node that doesn't match condition and some conditions defined by kubisk azure and in the third step, has a topic actually, no, no, no equipment, scheduler scoring the remaining node.

A

It's not filtered out by the cube schedule, and because, in the step two and the first and the last step is, the report is scheduled to a node that has the highest score and in this case uh users can extend the cubic scheduler by scheduler extender, it's implemented as a web hook. So webhook and this web hook can implement the two logic and the one is in the filtering step, and the second is in the scoring steps.

A

The first one is filtered nodes which don't have some conditions, it's defined by a scheduler extender and the second one is auto factor of squaring node. It's also defined by this uh schedule, extender.

A

So uh how about topwell VM schedule extender the first step is quick about filtering filtered nodes which don't have enough free space. Another step is add a property scoring to prefer node having a large free bridge space uh to uh to implement these Logics.

A

uh The two parameters of the top lbm's schedule is necessary, so the two parameters: uh first, one is freebridge space for each node and the second one is total request requested top level in volume size for each Port, which, using the top of the top element, volumes, manages annotations for these parameters in node resource and Port resource.

A

So as for for free Pages space for each node now, kubernetes has storage capacity, tracking features, and this feature can also be used for uh for extending the scheduler, but it can also be it can only be used for filtering. It cannot be used for scoring, so uh the top area beam also can use a storage capacity tracking. But uh for now the top European original annotation is the best choice so about CSI. Topology CSI topology is a feature kubernetes, and this feature schedule report to one of the nodes where it forms available.

A

Where are where volumes are available means the zone or node or something so this feature is used for a Zone, local storage or another local storage, and so on.

A

Create a volume on the same node as the corresponding port.

A

So, let's see the example of uh scheduling a port using the top available in volumes, so top side is kubernetes resources and the bottom size is Hardware configuration. There are three nodes: are not zero one: two and the node zero has 15 giga 50 gigabyte, free budget space and another one has 100 gigabytes and the node 2 has 5 gigabytes and there are three uh three node resources uh exist exist for each uh critical node and this node has the oh. No, no, no, this is node, has uh annotations uh just named capacity.

A

So this uh this annotation means that are real uh budget volume. They are busy volumes, so this annotation is acted by Topo lbm.

A

So let's create both approach and PBG resource. So then PVC requests, 10 gigabyte volumes from Topo LVN and our Port is created to consume this PVC and then our friend creating this port resource.

A

Annotation named capacity this capacity this capacity, this annotation means the total amount of volume size consumed by this board. In this case, just 10 gigabytes.

A

So next step could be scheduler is uh schedule, started to run and it's a filtering process. The first filtering process uh throw us the node resources and are raid capacity annotation and found that the node 2 doesn't have enough free space of enough free bridge space.

A

Filtering process in order to watch filtered out, node 2 is filtered out. So next step is scoring. uh There are two remaining nodes: node one and node two and uh the top airbn schedule at robots two nodes and find that the the largest largest capacity is 100 gigabytes and another one is the best choice and schedule this this board to node well and the next step uh thanks to CSI and CSI topology, the volume is create a provisioned to this node, this node and node one. What is scheduled.

A

Our first topic, Port, is scheduled to node well and the corresponding corresponding volume. uh Logical volume is created on top of Node 1, and this is bound to port and PV and PVC.

A

So the last topic is about the next problem of topology.

A

So next plan we have two plants. The first one is implement. The kubernetes of share capacity. I will put scheduling. So it's because setting up a schedule like standard is a bit difficult. I said a bit difficult, but uh to to record very difficult yeah, so uh we'd like to prepare a cap to implement the implement the capacity area features to to mitigate in this situation, because there are many, many uh issues to claim. uh I fail to schedule, extend the configuration, configure schedule, extend or something that's the was the case.

A

We cannot use casual extender at all due to the orientation of cloud provider or something so. The restaurant is donate Toby William project to CNC, because now uh topoam is great uh created by many companies or many individual peoples, uh I and all topology. Member and disciples consider it's the best choice Library.

A

So the conclusion is that a couple VM is an airbn based Shield site driver volumes and the corresponding part, even with spread for each node. It's accomplished by the scheduler extender and the CSI topology kubernetes features. So we welcome new users and contributions so.

A

Do you have any questions.

B

Thank you for your good presentation. I have one question: okay, why? Why did your company decided to develop it as our OSS.

A

ah Not only yeah.

B

A

This this uh this software.

B

Yes, this software.

A

um It's just because uh there's no um no suitable driver and we are able to create any software. Yes, we have many uh skillful engineers.

B

Is there any benefit to create to develop each other OSS? So so because so you can develop it as a closed source code.

A

ah Okay, I understood uh what you said so he say uh why we created a topo, lvm and publish it as open source right. Yes, like okay, it's because my team, uh my team, firstly decided to uh publish all our products our products about the infrastructure in public as OSS, because uh it's it's all uh it's for many purpose. The first purpose is: if someone actually our product, very easy, uh very bad, uh very good.

A

uh It's uh it's very happy for us, and probably you, as you know, the software about the infrastructure, it's not the source of our benefit and our benefit. So it's not a difficult to publish our products and the last reason is if we publish a our software to as OSS someone read our source code and.

B

We should write should.

A

Write this source code to uh to public and we we can write very good code and yes, in-house software yeah. It's very good and I recommend you to uh write. U.S, publish your source code as oscsc. It's real, okay,.

B

Thanks I understand yeah.

A

Any other questions.

A

Let's see okay, thank you very much.