Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Capacity-aware Dynamic Volume Provisioning For LVM Local Storage- Kazuhito Matsuda & Satoru Takeuchi

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Capacity-aware Dynamic Volume Provisioning For LVM Local Storage - Kazuhito Matsuda & Satoru Takeuchi, Cybozu

This session presents a summary of the existing Kubernetes features to use local storage devices (e.g. hostPath and local persistent volumes) in first, then introduces a novel CSI plugin named TopoLVM. Applications that can replicate data themselves (e.g. Elasticsearch, MySQL, Ceph) are promising users of local storage because of their needs for lower prices and higher I/O performance storage. Although existing ways enable to use of local storage, the following desired features are not provided yet: - Raw block volume - Capacity-aware dynamic provision - Online volume resizing To satisfy these features, the authors created a novel CSI plugin named TopoLVM. It dynamically creates logical volumes of LVM and extends the standard Pod scheduler to consider volume group capacity of each node. Besides, it intends to support more features including online resizing.

https://sched.co/ZerD

A

Hello: everyone, I'm kazush tomat and I work at cyborgs today, I'm going to talk about a csi driver for kubernetes, a capacity air, dynamic volume, provisioning for lvm, local storage and my co-presenter is sato takeuchi and also an engineer at cyborgs japanese cloud service provider of google play.

A

We are now reconstructing our infrastructure into a kubernetes based cluster on the on-prep smith data center, including today's content about utilizing local storage.

A

Here is the agenda, and first I'm going to talk about existing ways to use local storage in kubernetes.

A

Then I talk about the motivation and challenges of dynamic provisioning after that I'll introduce our nobel csi program named toppo lbn and give a demonstration.

A

So why would we use local storage in general? Purchasing storage in kubernetes relies on remote storage systems such as safe. It also includes storage services provided by cloud service providers such as amazon ebs.

A

However, the benefits of using local storage still exist, which io performance and reasonable cost. Of course, using local storage in kubernetes has some disadvantages, such as topology limitation and redundancy problems, but it is a reasonable choice for io bound applications.

A

So if you have, I o bound workloads such as mysql, aerostickset, look and so on also have high performance, work of storage, and maybe our presentation might be helpful content.

A

First I'll show existing ways to use local storage in kubernetes. The most popular and simple way is host bus, maybe which you, which you may already know.

A

In host bus, the kubernetes mount the host file system path to a path in the pod. However, host path has was not designed to be used for stateful applications; they do not retain essential attributes such as capacity and bound status and so on.

A

Of course, it also does not does not support dynamic provisioning.

A

It's mainly used for applications that have control privileges for the host os and or collect information about the host anyway. Hostess should not be used for maintenance application, state or data.

A

A practical method was presented at kubernetes 1.14, and that is local persistent volume. This feature can have topology area pod scheduling with participant once possible stand bones are created manually; in other words, users must prepare partisan buoyance beforehand.

A

Of course, we can utilize some automation tools. However, that is outside of the persistent volume life cycle managed by kubernetes. Therefore, it cannot be dynamic, provisioning.

A

Without dynamic provisioning supported by kubernetes, a crossed admin is likely to meet a challenge like this one developer says hey. I want to run an elastic search instance. So please give me persistent volume, but I cannot say how it will consume disk space, so 100 gigabytes for now. Please another developer says: please give me, give me a persistent volume for my circle, but service growth rate cannot be predicted, but lovely 500 gigabyte, another one says and the service has gone over.

A

So I release the volume and in kubernetes and in the pot life cycle is very rapid, so life cycle of partisan volume is also quick.

A

Therefore, we can easily imagine that the operation and management of persistent volumes must be collapsed in terms of scalability, so we need automated management.

A

In other words, we need a dynamic provision for local storage.

A

Dynamic provisioning is defined in the kubernetes document, and that is the dynamic provisioning feature eliminates the need for cluster administrators to provision storage. Instead, it automatically automatically provisions storage. When it is requested by users.

A

It provides authority, scalability and accuracy for the operation and management of storage.

A

Essential features are mechanisms for dynamic volume, allocation and resizing, because we cannot determine physical volume sizes for services beforehand to realize dynamic allocation.

A

We have to determine where all required persistent environments can be located and considering the future expansion of boring, a node that has more free disk space is preferable to others.

A

Both of them require capacity, available, scheduling, so capacity awareness is key to dynamic provisioning.

A

To achieve capacity aware volume scissoring, we need to follow two steps: the first cons the first consists of gathering capacity metrics from nose. Then the scheduler filters and scores the nose using capacity metrics and the required volume size.

A

The process is depicted on this right and first volume provisioner gathers capacity metrics when spot controller requests port scheduling. The controller should pass the requested bearing size to the scheduler to determine where to locate the port, the scheduler filters and scores all nodes based on the capacity metrics and the requested bond size.

A

After that, partitioned bond controller will request a buoyant from the provisioner.

A

However, the current kubernetes does not support capacity awareness, although the clip which proposed how to expose storage capacity to kubernetes uh it was pro posted and merged, but important scattering with dual capacity is ongoing.

A

So ahead of it, we've implemented a capacity available in provisioner for local storage, namely toppo lvm simply started. We made it it as a csr program.

A

Toppo lvm, of course, has a capacity aware, dynamic, provisioning feature, and it also supports low block volume as purchasing volume which improves the performance of applications they can use block devices directory online resizing is also supported.

A

Let me show you the details of topple lvm. Here is a diagram of top for avm. In this figure, the blue square box is a topo lvn components and the round cut boxes indicates api resources, skate and api means, cube, etc and kube, apa server respectively.

A

The external provisioner is the sidecar container and that water is purchased and boring claims and calls the shea site drivers apa to create and bind persistent volumes.

A

Let's check out the dynamic provisioning sequence by top rvm: first, the top rvm node landing at each server each server and it annotates its own server's storage capacity to the corresponding node resources.

A

When a pawn with pbc is created, the curb etc will receive a scheduling request then pass it to the topple airbnc and the topple rvm scheduler filters and scores or knows, based on the annotation of capacity metrics.

A

Once the location of the port is determined. The external provisioning calls the topple avm controllers api with the topology key. That is a node name.

A

The topo avm controller creates a custom resources resource named top logical volume, with the node name and the requested volume size.

A

Each topo avm nodes, each top of every node watches this custom resource and, if is for its own node and the top of lvm node, will create a value and update the status of the custom resource.

A

The topolvm controller, which is watching the status of custom resource when detecting this update, the tuple vm controller, returns the volume information as the return value of the api call.

A

Therefore, the external provisioner can create persistent volume with the volume information.

A

Okay, I'll give an additional explanation for the capacity area of volumes measuring, as I mentioned, and briefly each of every node, just as capacity of its own nodes, no storage and this metric is written to the node annotation and this way with a specific annotation key topo, everyone cyborgs.com capacity.

A

And toppo avm has the admission mutating hook. It's for modifying port spec according to its requesting boring size like this. It ensures and that the topper lvm scheduler can get the requested bone size via the cuba surgery.

A

The cube scheduler is implemented as an extension of curb scissors, which means the curb scheduler calls the web hook of topple avm serger with pod, manifest, according to the previous mentioned information and the topple, even scheduler, changes which node must be filtered and can calculate scores of nodes to determine where to locate the port.

A

The scoring expression is here that is the logarithm to base two is very simple.

A

I'm also going to touch on the limitations of toppo lvm, although you can get some redundancies using other techniques such as the white one, but it does not provide any no level redundancies because the volume uh just located on the local disks.

A

So the volume is sufficed by topo lvm and should be used by applications. They can be redundant themselves, for example, dbms databases and distributed storage systems such as massive and safe.

A

Okay, next we'll give a demonstration of topo lvm, please subtle: okay,.

B

Okay, I'll have some demos of tuple lvm I'll, introduce following three features that kazushito mentioned in this presentation here is the software and hardware configuration in this demo.

B

There are two worker nodes, kind, worker and kind of worker, too kind worker has one volume group named my bg1 that has a 18 gigabyte free space and kind worker 2 also have a 18 gigabyte free space volume group. My bg2, both mypg-1 and mybj2, are managed by toppo lvm.

B

I'll use three terminals in this presentation: the first one is operation terminal to issue cubic control command, for example, kobe control, gate, node shows three nodes, the first one is control plane and the second one and third one is worker nodes, kind, worker and kind worker. Two and the second terminal is connected to kind worker node.

B

So we can get the information on myvg1 from here. Mybj1's respace is under 18 gigabytes and the last terminal is connected to kind worker 2 and we can get my budgie tools. Information.

B

Its free space is under 18 gigabytes.

B

So uh since both mybg-1 and maybe g2 are managed by toppo rbm, we can get the free space of micro, g1 and mybg2 from the cheese nodes, node resource for the control, gate, node kind, worker or yaml. Okay, please see this annotation double lbm.cybos.com capacity. This annotation means free space of my bg1.

B

So it's about 18, gigabytes, okay, the next one is google kind worker, 2 kind. Worker 2 also has this annotation and the value is same as the micro g1, it's corresponding to the maybe g2's free space, and it's also under 18 gigabytes.

B

B

So first demo is about dynamic volume provisioning. So, let's schedule a port named engine x1. This port use uses a one gigabyte volume, yes, but one: okay, more persistent volume claim turbo bbc1 requests, one gigabyte from turbo lvm and the ngx one port consumes tuple pvc one and is mounted in power www.html.

B

Please note that there is no assistant volume resource. There are only positive volume claim and port results and, of course, cover control. Get people show nothing there. There is no persistent volume in the default name. Space.

B

Okay, so cubic control apply f, one, okay right so bright, okay, so the expected expected result here is that uh processing volume is created dynamically. Okay. So let's confirm the copy control get port. Okay, nginx one is running and uh controller get ppg. Okay, toppo pvc one is created and bound to bbc blah blah blah. It's a persistent volume created dynamically by toppo, lvm, okay, past and boiling.

B

So we found that the persistent volume is created dynamically. So we found a dynamic volume. Provisioning in turbo lbn worked fine and as a side note we can get.

B

This volume is correspond to the logical volume in my bg1, so first budget display shows the free space is decreased to change from 18 gigabytes to 70 gigabytes, and here we display show okay, this volume right. This logical volume is corresponding to the position volume on the positive volume claim new passive memory. Okay, so of course the kobe control get know the kind worker its capacity is changed.

B

It's it was about 18 gigabytes. Now it's it's about 17, gigabytes, okay,.

B

So the next uh next demo is about capacity, our scheduling as a present preparation. Let's exhaust a kind worker tools, volume group now, uh can you wake up to the volume group? My budgie 2 has 18 gigabyte free space to exhaust this volume. Let's schedule a port named nginx 2 to kind worker 2..

B

It uses a 17 gigabytes volume.

B

Press bottom, two: okay wbg2, it has 17 gigabytes consumed by engine x2 port and this engine, x2 port, is bound to kind of work too. Okay, let's upright y f part two okay, common control get part it's under creating. Please wait for ry.

B

So the uh the top of pbc2 and the corresponding volume is under creating okay engine x2 is now running. Control get people rpv, okay, 70 gigabytes.

B

So the gate, node kind worker 2.

B

Okay, it's the mybg2's capacity is now under 1 gigabytes. Of course, we can find it from a kind worker too. Okay, my g, my budget tools, free space, is under one gigabyte.

B

Okay, so now kind of welcome to the capacity is under one gigabyte, so pre right schedule many ports in this case engine x345675 port, these ports. All these ports use a one gigabyte volume.

B

uh Please note that the kind worker tools uh mybuji2 is has only has only under uh one gigabyte free space. So uh it's it doesn't. It doesn't have enough space to create one gigabyte volume.

B

Sorry there is many parts: okay, double pbg3 is a request, one gigabyte and consumed by ngx3, and it's same as four five, six, seven. Okay, let's apply, we control apply many parts.

B

Okay, so the expected result here is all ports are scheduled to kind work. Okay, it's because engineering should uh know that my gp, my bg2, doesn't have enough space. Okay, so copy control get both.

B

Okay, it's already running and the engine x3 to 7. All all these ports are learning and in learning in kind work.

B

It means that these ports are not evenly spread in the spirit of a kind worker and kind worker too. It's because toppo lvm, no kind worker 2 doesn't have enough space.

B

Okay, so then toppo lvm select a kind worker as a node to be scheduled. Okay,.

B

So we confirm all parts as scheduled to kind work without topple oem.

B

Some ports will be scheduled to schedule to kind of work out too, but it failed to start because the low carbohydrate local volume is cannot be created. Okay, so we confirmed that capacity area scheduling in turbo, lvm worked fine.

B

So the next and the last demo is about online volume resizing.

B

So it's very simple: just just expand engine x1's volume to 2 gigabytes, ngx ones, volume is a tuple pvc. One top which you want is good control, comma controller gate, ppc, topo, pvc, one. Yes, it has one gigabyte and the corresponding file system is mounted in engine x1, df.

B

Okay, it's about www.html html. Its size is about one gigabyte, okay, so to expand uh people. She it's very simple, edit, just edit bbc double phone.

B

The current requested volume size is one gigabyte, so what should I do is just change this value to two. So, okay, so the expanding uh job is kicked. So google control get pvc double bbc one.

B

Yes, not it's not changed. Yet uh it's under expanding, so uh more precise, precisely uh the the boring, a logical volume corresponding to toppo pbc1 is now under expanding and after that, toppo lvm expands the file system inside this biological board.

B

So is it done? Okay, it's changed to two gigabytes, it's expanded. So as a side note, uh it's event is get can be get from. uh Google control describe bbc double phone okay. So when I changed, I edited the bbc resource.

B

This event is: kicked external expanded and this means resizing volume and the resize file system started to resize file system and, while stem resize successful it's done so. Let's see the this one is five stem inside inside engine, x1 df by www.html, it's expanded to two gigabytes. Okay,.

B

So expected result is tuple. Pvg one is result resized, okay, the second one is the corresponding file system is resized. Also, I checked so we can find the online board resizing feature or topo. Lvm worked fine, so the demo is done.

B

So here is the takeaways of the presentation.

B

Lbm is a local storage, dynamic provisioner based on lvm, and it enables capacity area, port scheduling based on local storage, and we cyborgs continue to develop toppo lvm and the tupperware m is not a toy program.

B

Our target toppo rvm's target is to use production use.

B

So here is a community and links of toppo lvm. Please join us. Thank you very much. That's all.