Cloud Native Computing Foundation KubeCon + CloudNativeCon + Open Source Summit China 2021, 18 Dec 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Keep Persistent Volumes Healthy for Stateful Workloads - Xing Yang, VMware & Yuquan Ren, ByteDance

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

为有状态工作负载保持持久卷的健康性 | Keep Persistent Volumes Healthy for Stateful Workloads - Xing Yang, VMware & Yuquan Ren, ByteDance

越来越多的有状态工作负载已被迁移至 Kubernetes 平台。这些工作负载依靠持久卷来储存数据。然而，在有状态工作负载配置卷并予以使用后，底层储存系统可能会发生很多情况。该卷可能会因意外被删除、该卷所在的磁盘可能会发生故障、磁盘可能会持续退化影响其性能等。Kubernetes 如何及早发现这些问题并提醒用户？Kubernetes 引入了卷健康监测功能，以发现这些存储问题，并通过发送事件信息将这些问题公开给用户。虽然这种方式很有用，但是需要用户手动修复这些问题。如果 Kubernetes 侦测到卷异常情况后也有方法进行自动修正呢？在此次讨论中，我们将讨论目前卷健康监测功能有何作用，以及我们正在做哪些努力以将此功能提升至下一层次？

More and more stateful workloads have been migrated to Kubernetes platforms. These workloads rely on persistent volumes to store data. However, many things could happen to the underlying storage system after a volume is provisioned and used by a stateful workload. The volume could be deleted by accident, the disk that the volume resides on could fail, the disk may be degrading which affects its performance, etc. How can Kubernetes detect these problems early and alert users? The volume health monitoring feature has been introduced in Kubernetes to detect these storage issues and expose them to users by sending events. This has been very helpful, however, the problem has to be fixed by users manually. What if Kubernetes also has a way to do automatic correction after detecting abnormal volume conditions? In this session, we will discuss what the volume health monitoring feature can do currently and what we are working on to move this feature to the next level.

A

Hello: everyone, my name, is xin yang.

A

I work at the vmware in the cloud storage team, I'm also a co-chair in cncf tag, storage and a co-chair in kubernetes. Seek storage here is my co-speaker nick.

B

Hello, my name is nikolan.

B

I come from bedance and I am now focusing on kubernetes related work.

A

Thanks nick today, we will talk about how to keep persistent volumes healthy for stay for workloads.

A

Here is today's agenda. We will first talk about what problems we could encounter after voting is provisioned. Then we will introduce the volume health monitor feature. We will give a demo of the existing functionalities of the feature.

A

Then we will talk about what we are working on in kubernetes 1.23 release to improve this feature, and finally, we will talk about what we are planning to do in the future.

A

So, let's see what could happen to your volume after it is provisioned, as shown here we have an application running using a c4 set with three replicas. It has three parts: each is using a pvc.

A

Everything looks perfect for a while, but then one day something happened. An admin was working on something and made a mistake by accident and as a result, the volume was deleted from the study system.

A

In any case, the admin didn't realize that he or she did something wrong that could affect applications running on the platform.

A

So, on the surface, everything looked fine, because kubernetes does not check what happened to the underlying volume on the study system after it is provisioned and used by a pod.

A

Everything looked normal until when the application tried to write to the volume, then of course it failed.

A

Now the user tried to figure out what happened by checking the logs, but there are no clear messages that indicate the volume was deleted, so the user opened the ticket with the infrastructure team. Then the infrastructure team started to troubleshoot the problem.

A

It is not easy for them to root cause the problem either as there isn't enough information in the logs. So it's going to take a while for the real problem to be discovered.

A

There are other possible failures as well.

A

The admin may have removed the disk for maintenance or replacement without knowing that it has an impact to the running applications.

A

The disk that the warning resides on could fail.

A

There may be configuration issues with down the lines for the system that affect the volumes health.

A

The volume could be out of capacity, so app can no longer write the volume until the voting is expanded.

A

The disk may be degrading which affects its performance.

A

There are other possible problems that could happen to the volume after it is provisioned and used by a pod for local volumes. If that node, where the local volumes resides fails, the volume also does not exist anymore.

A

Let me be rewrite io errors.

A

The file system on the volume may be corrupted, the file system may be out of capacity, volume may be outmounted by accident outside of kubernetes.

A

There could be other issues that are not captured here. So how do we solve these problems and improve user experience?.

A

To solve this problem, we introduced an other feature called volume: health monitoring without warning house. Monitoring kubernetes has no knowledge of what happened to the underlying volumes on the study system after a pvc is provisioned and used by a pod.

A

With this feature, csi driver can communicate with solid systems and find out what happens to the volumes and communicate back to kubernetes. So kubernetes can report events on pvcs or pods if volume conditions become abnormal.

A

This feature includes two parts: there is an external boiling house monitoring controller, that is a side car deployed with the csr driver to monitor volume, health from the controller side.

A

It reports events to the pvc when abnormal condition is detected.

A

Cubelet also monitors warning health from the node side. It reports events to the pod when abnormal warning condition is detected.

A

So here we have a csi deployment example that shows various kubernetes components: the csi driver and the storage system that is used to persist. The data on the volumes.

A

Here we have cube controller manager on the master node see that driver controller plugin is deployed together with kubernetes csi, external provisioner, external attacher and external health monitor side cars.

A

Note that the csi driver controller pod does not have to run on the same node as the kubernetes master, but it is recommended to run on dedicated control, plane nodes, the kubernetes ca sunset. Cars are watching kubernetes api objects, such as persistent volume claims, persistent volumes, volume attachments to detect, create volume, attach volume, requests the side, cars call. The csi driver and the csr driver communicates with the storage system to complete those volume operations.

A

The external health monitor controller works differently from other side cars. It periodically causes csi driver to retrieve volume cloud information from online storage systems.

A

On kubernetes worker nodes we have cubelet and the css driver node plugin deployed together with the node driver register, statica container node driver registrar, fetches driver information using nodegap info from a cssi endpoint and registers. The css driver with the cubelet on a node cubelet, directly issues a csi node get info node state volume and node publish volume, cost against csr drivers to get info and marked volumes.

A

Cubelet also periodically calls node that volume stats to you get volume stats information, including voting health information.

A

Now, let's take a closer look of the external house, monitor controller this controller costs either list volumes or controller gate volume, csr, pcs and reports, volume, condition of normal events with messages on pvcs.

A

If abnormal volume conditions are detected, if the csr driver supports, least volumes and volume condition controller capabilities, it must implement controller rpc list volumes and report the volume condition in response.

A

If a css driver supports get volume and volume condition controller capabilities, it must implement the controller, rpc controller data volume and reported volume condition in the response.

A

If a csi driver supports both list volumes and get volumes and volume condition controller capabilities, in that case only list volumes, csi rpc will be invoked by the external house, monitor controller due to performance considerations.

A

There is also a node watcher component in the controller if the enable node watcher flag is set to 2. When deploying the external house monitor controller node failure, events will be watched so when a node failure event is detected, an event will be reported on the pvc to indicate that parts using this pvc are a field node the controller side volume monitoring has been alpha since kubernetes 1.19 release.

A

Now, let's take a look of the warning house monitoring feature from the note side, when volume house was first introduced in 2019, there was an external house monitoring agent that monitors health. On the note side in 1.21 we made a design change to avoid a duplicate, csi rpc course.

A

Cubelet already calls node get voting stats to retrieve voting metrics such as available total and use capacity.

A

In addition to that now it also retrieves volume health condition through node guide volume, stats, csi, rpc, call and reports volume, condition of normal events with messages and pause if abnormal running conditions are detected, only csi drivers with volume condition node capability, support volume, house monitoring in kubelet, we added a new alpha feature: gate called csi volume, health in 1.21 release to introduce this feature in kubelet.

A

Here is how volume condition is defined in the css spec. If there is a problem with the volume or the thread system, the abnormal field in the voting condition should be set to true.

A

There is also a message field that you can use to add additional information to explain what the abnormal condition is note that if a volume is not found on the 3d system, the abnormal field should be set to true, so that an event can be reported on the pvc which is still in the kubernetes cluster.

A

So now I'm going to hand it over to nick he's going to show us a demo.

B

Okay, thank you. Xing yeah, as has seen that that we created the demo to visually show what the volume house market can do right now, so we expected that it can send a number of warning events to the ports location. Could I have clicked the video.

B

Yeah first, we will need to set up a kubernetes class locally, using the local app class script provision in kubernetes repo and the kubernetes version and truth is 1.18.

B

I made some changes to the deployment files in order to set up the testing environment and run the demo correctly. I will summarize it the changes later and we can acknowledge right now, yeah and maybe it will need so. It will take some time to get the cluster ready.

B

Yeah, the local cluster is ready and then let's create some csi containers and the hostpath plugin drivers as well. Using the deployment files from the csi drivers for the repository and and pv data in the host parts in the host is an in list. Folder yeah run the deploy script and all the containers will be created.

B

Okay, so after that we will create a storage class and a pvc and as well and a port as well. So let's create a storage class and the pvc.

B

Okay, we can see that the pvc is created and and one pivot dynamical provision that is better host, paused driver and let's describe the tv yeah we can see it is dynamic decoration, but it helps the path driver and I will create a port using that pvc.

B

Okay, we can see that the call is created and is it running now. Okay, then, let's delete the month pass.

B

Okay, after delete after we're deleting them on password, expecting that the volume house monitor can send abnormal events to the poor okay. So, let's.

B

Describe I can no longer events right now, but it may take several several seconds to send to like the warm house, monitors and normal events. Okay, still know of number events.

B

Yeah, okay, the number event is here, so we can see that the number events and the monitor works as expected.

B

Okay. Next, please.

B

Okay, here I summarize the changes I made to the deployment of fires. So uh since the same, some storage driver containers need privilege security option. So I said a low prevalent flag to true for api server. This is the only change I made for for kubernetes and another one.

B

I made a small change to the host path. Plugin yammerfire2 and I changed the csi provisional uh image version to version 3.0.0 and the previous, the previous version, sm5 yeah, and when I use this, I use this yummy files and deploy the deploy the containers and set up the local environment, and we can run the demo correctly. So if you are interested in this feature, you can test this in your environment too.

B

Okay. Next, please.

B

Okay, uh since I just know that the node side monitor component is true black, but actually we create an external agent which is responsible for nodes and modifying work at the beginning. But finally, we decided to move all asian work to kubrick, so for now, occupied will be responsible for checking volume, health conditions theoretically and sending abnormal events. If advanced problems.

B

Also, we are now trying to light public emit matrix too and unless we'll target over version in 1.23 kubernetes version.

B

Okay, next, please.

B

Okay, apart from the current work, we are now also discussing what we can do in the future for volume health monitor. The most important topic we are talking about is automatic reaction to unhealthy volumes.

B

I listed some cases here so, for example, if one node breaks down and we have suitable pores with local pvas running on that node. So what can we do with waiting for more than five minutes for kubernetes to delete the pause, although we can delete this pause, but since they are using local pvs, the newly created ports will get stuck in pen instead, all the time, if the note cannot recover so what what we can do?

B

Oh another option we can do is delete, delete the pause and the pvcs as well, so that we can have the chance to schedule the new pulse to other nodes.

B

So another case is if the volume is out of capacity. Maybe we want to resize the volume automatically in order to avoid the data loss.

B

Also, if the underlay volume is deleted by mistake, maybe we prefer to deleting the relay delete the related up applications automatically.

B

If we have the ability to do reactions, maybe we can another thing we can. We want to do is support push mechanism too. So customized drivers can report problems by themselves.

B

Maybe you will need to add a new keyway accuracy status, condition to indicate whether the underlying volume is healthy or not, and the reaction part can react to this status condition and some users are asking for this feature too. So maybe there are maybe some other important features we need. We need to add four volume house monitor, so we can discuss list offline. If you have any ideas.

B

Okay- next, please, okay and- and we added some resources here. I want to say that if you are interested, please join us together to make this feature better. You can try this feature. If you need it, you can report your problems when you deploy or use it and it'll, be super great if you can contribute your proposal to solve this problem.

B