Cloud Native Computing Foundation Cloud Native + Open Source Virtual Summit China 2020, 11 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Deep Dive: Kubernetes SIG Storage - Xing Yang, VMware & Michelle Au, Google

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Deep Dive: Kubernetes SIG Storage - Xing Yang, VMware & Michelle Au, Google

Kubernetes SIG Storage is responsible for ensuring that different types of file and block storage are available wherever a container is scheduled, storage capacity management (container ephemeral storage usage, volume resizing, etc.), influencing scheduling of containers based on storage (data gravity, availability, etc.), and generic operations on storage (snapshotting, etc.). In this session, we will deep dive into some projects that SIG Storage is currently working on, provide an update on the current status, and discuss what might be coming in the future.

https://sched.co/cpBN

A

Hello, everyone today michelle and I will be giving a deep dive on kubernetes, seek storage.

A

Here are the six storage leads? My name is xin yang. I work for vmware in the cloud storage team. I'm a co-chair of seek storage along with sadali from.

B

Google, and uh my name is michelle: oh, I am a engineer at google um working on kubernetes storage. um I am one of the tech leads along with uh jan from red hat.

A

Here is today's agenda. First, we will talk about what is six storage. We will talk about what we have done in the 1.80 release and what is coming in 1.19 release, and we will talk about our future plans. What we will be working on after 2019 and how to get involved with the sig.

A

Storage, so what is six storage? Six storage is a special interest group that focuses on how to provide storage to parts in your kubernetes cluster seeker storage. Scope is in the storage control plane. It provides a way for containers in the pods to consume block of file storage.

A

This can be persistent, long-term storage that leaves beyond a paws life cycle, or it can be effemoral temporary storage, which becomes available when the pot is started and goes away when the pot goes down. Seek storage is responsible for the life cycle of volumes used by the pots.

A

This includes provisioning a new volume attaching a volume to the node and mounting it so that the pod can use it our mounting detaching and deleting the volume when it is no longer needed taking snapshots so that it can be used to restore the volume if the original volume is corrupted. For some reason, sig storage also looks at how to influence the scheduling decisions based on topology information, to see whether the storage is accessible to a node and make sure volume is scheduled to a node which can have access to the storage.

A

Also, seek storage is responsible for managing storage capacities, managing quota resources based on capacities or number of resources, and provides ability to expand volume if volume runs out of space.

A

Sig storage owns the persistent volume and persistent volume claim feature. This allows sturdy vendor to create a volume and persist, storage, persistent data in this volume which can be preserved. Even if the pod goes away.

A

We have the study class concept. A story class provides a way for administrators to describe the classes of storage. They offer different study classes, might map to different quality of service levels of the storage system in dynamic provisioning strategy class is used to find out which provisioner should be used and what parameters should be passed to the provisioner when creating the volume kubernetes volume plugins include in-sheet, plugins, auto-tree, flex volume and csi drivers.

A

The kubernetes implementation of the container storage interface csi has been ga since the kubernetes 1.13 release, sig storage has been working on migrating from injury plugins to out of three csr drivers. New features are only added to the csr drivers.

A

Csi is supported by multiple container orchestration systems and storage vendors. It is designed to be vendor, neutral, interoperable and has a focus on specification.

A

It defines a set of storage interfaces so that a storage vendor can write one plugin and have it work across a range of container orchestration systems other than persistent volumes. There are also ephemeral volumes. Effemoral volumes are specified directly in a pod spec.

A

It's mounted on the part as a directory data can be stored in a file under that directory. Fmoral volumes include secrets, config maps, csi, ethmoral volumes and so on. Its life cycle follows the life cycle of a pod.

A

On this slide, you can also see a seek story. Team page visit this page. You will find a lot of information about the sig.

B

All right so now we're going to move on and talk about all the new features that sig storage has been working on in 118, so we landed a lot of features and graduated them to ga status.

B

First is the raw block feature. This allows containers to access a volume as a block device instead of as the formatted file system.

B

This is very useful for workloads such as high performance databases that may operate and have optimizations on with their own file system format and want to deal with a block device, instead of actually having someone else format the file system on it.

B

So that feature is now ga for both in tree and csi drivers.

B

The next feature that is ga is cloning. This is a feature that is only available to csi drivers, and this gives the ability to create a pvc as a copy of another pvc.

B

The next two features um are also csi. Specific one is called the skip attach feature. This allows a csi driver to indicate that they don't support the attach operation and they don't have to bundle the the csi attacher sidecar with the driver. So that's what this feature enables and then the next feature csi pod info on mount.

B

This allows a cs5 driver to receive pod information, such as the name and name space of the pod.

B

That is mounting this volume, and this is useful if, if the csi driver needs things like pod, specific credentials or any other pod specific info, so those are the features that graduated to ga next graduating to beta is open, stack, csi migration, so overall, the csi migration feature allows users to continue using the kubernetes in-tree volume apis, but the main difference is that the back end of kubernetes will call out to the csi driver instead of calling out to the entry volume code.

B

So the the main purpose of this feature is so that we can remove third-party as many third-party dependencies from the core of kubernetes as possible. This will bring a lot of security benefits and also help reduce the uh vendor size of the kubernetes components themselves.

B

So the um openstack implementation of this has gone to beta in 118. and then in 118. We also introduced a number of new features that are alpha. First is csi windows. This is basically the ability to support csi drivers on windows notes.

B

So this is now an alpha and currently a couple. There are a couple of csi drivers from azure and gce that support this and I believe, more are coming on the line for all of these csi features. They all depend on the csi driver to actually support and implement it. So please check with your csi driver whether or not it supports these features.

B

The next alpha feature is to be able to support uh setting uh an fs group change policy in the pod security context, so this is to help solve a scaling problem with the current fs group implementation, where the, where kubernetes has to recursively walk through every file in a volume and change the ownership of those files to match the ownership on the pods.

B

This is very time consuming for volumes that have a lot of files in them, so this alpha feature is providing an option to optimize that so that we don't have to actually recursively walk a volume on every single mount.

B

Instead, what this new change policy offers is the ability to only update the ownership of a volume uh on the first mount when it detects that the ownership of the volume is different from the pod, so that is a great alpha feature out that will help that can potentially help reduce the mount time for any of your volumes that have a lot of files in them.

B

The last alpha feature is also a scalability feature.

B

This is the ability to mark a secret or config map volume as immutable, and this will base this feature will basically help reduce load on the api server from nodes, because currently cubelet needs to watch the secret and config maps for updates and then be able to update the volumes periodically.

B

So if you run a lot of pods on nodes- and you have a lot and those pods are using a lot of secrets and config maps, you may hit some api server bottlenecks with that. So this feature is going to help for those use cases.

B

So those are all the features that we've done in 118 and now I will go through all the features that are planned to come in this upcoming 119 release. So we are currently in the middle of um or we're close to the end of the release cycle for 119..

B

um The 119 release is targeted to be um in a little over a month now on august 25th, so I'll go through all the features that sig storage is planning um for. The 119 release, so graduating uh to beta are the azure disk and vc vsphere csi migration features.

B

So these are just additional um cloud provider, implementations for the cs overall csi migration feature and will allow the core kubernetes apis to be supported by csi, back-ends and the.

B

So this release adds the azure disk and these sphere implementations for that um and then similarly, the uh csi windows feature that we introduced in alpha in 118 is graduating to beta in 119.

B

And also the immutable secrets and config maps feature that was introduced in 118 is now graduating to beta in 118 as well.

B

In addition to those graduating features, we are continuing uh making improvements on existing beta features, so these features have been beta in have graduated to beta in earlier releases.

B

They are remaining beta in 119, but we are making continuous improvements to them. So first up is a volume expansion.

B

um There are improvements being made uh with related to the way that we process um offline expansion for uh volumes that can only support doing in a volume resizing operation if the volume only if the volume is unmounted so detecting that support in the driver and then having kubernetes orchestrate the volume expansion operation to work with those kinds of drivers is a big improvement that we are making in 119 and then the volume snapshot beta feature which was introduced in kubernetes 117.

B

We are continuing to make bug, fixes and other improvements to the implementation.

B

119 also comes with a number of new alpha features.

B

First, up is csi storage capacity tracking, so this feature gives a way for a csi driver to report how much capacity they have for a storage pool on a particular topology.

B

So if you have a local volume, csi driver, this feature lets you node for storage class. If you have a csi driver that deals with storage on a rack basis, you can report how much capacity you have for provisioning volumes on a per rack and per storage class basis.

B

So this is a really important feature that will help enable uh local volume, dynamic provisioning, which has been a pretty long long awaited and requested feature. So this is one of the key features we need to enable that use case. So, if you are interested in this use case, please go ahead and try this out with your csi driver, and you know, help us fix any bugs and make improvements to it.

B

The next new alpha feature is generic ephemeral volumes, so um this is sort of a improvement, it's sort of an additional um alternative method for implementing ephemeral volumes in csi. Although it's not, it's not actually specific to csi, um what this feature lets you do. Is it lets? You take any volume plug-in today that supports persistent volumes.

B

It allows you to specify them directly in a pod spec and have the volume life cycle handled automatically.

B

So what what that basically means is when you specify in a generic ephemeral volume in your pod spec, you will actually specify a volume claim template which is very similar to the pvc spec and underneath the covers um it will go and manage the life cycle of that pvc with uh the life cycle of the pod.

B

So um this feature is really powerful and will and and can enable a lot of use. Cases such as having a external, any csi driver automatically gets supported with like a scratch space use case. So if, for example, you want to use an eds volume as a sort of an empty dir type of volume, where you you're just using it as scratch space and you want it to get deleted when you delete the pod, you can use this feature to specify an ebs.

B

A pvc using a ebs storage class and the provisioning and deletion of the pvc will happen automatically by and will be managed by kubernetes.

B

Another uh important use case that this can enable is something like offering a empty dirt alternative, but using specialized local storage.

B

So if you have say a csi driver that can provide local storage with using local ssds on your node, you can use this feature to basically provide an empty dir like behavior or empty dirt, like volume, but on top of a local ssd, instead of instead of using the boot disk on the node.

B

So this is a very powerful feature again. I would highly encourage everyone to try it out and help give us feedback and help develop the feature further, so that we can graduate these two beta and nga.

B

The next alpha feature that we have added is csi volume health. This basically provides a way for csi drivers to monitor the status of provision volumes and to be able to report that back to kubernetes right now. This will um right now volume health status is reported as events on the pbc objects, so that you can you have any monitoring.

B

um You can build some monitoring on pvc events to determine if your volume is in a healthy state or not, and then the last new alpha feature that we added in 119 is to add a fs group support policy on the csi driver itself.

B

This will allow a csi driver to explicitly state whether or not they support fs group. So currently in kubernetes, we use some heuristics to determine if a csi driver, if fs group should be applied to a volume and right now we, the heuristic, is that if it's a read write once volume- and there is a fs type set on the on the volume specification, then we will then kubernetes will go and apply the fs group changes to that volume.

B

But this what this feature does is make that choice actually more explicit by the driver and the driver will indicate whether or not it supports it, and this will be useful so that drivers that fall outside those heuristics, like nfs drivers or any other multi-rider drivers. They can potentially opt in to having kubernetes, manage the permissions on their volumes.

A

Thanks michelle, I will be talking about our future plans. What we will be working on after the 1.19 release. There are a number of features that we are targeting, ga or beta in 1.20.

A

The first one is csi volume expansion. We have been working on bug fixing in order to bring this feature to ga.

A

We are planning to deprecate the online offline, one extension plugin capability in a future csi spec release targeting 1.4 css spec release online offline. One expansion will still be supported, while csi driver receives a request for voter expansion.

A

It should return failed precondition if it only supports offline order expansion, but the request was issued while the volume is in use, kubernetes will stop calling the csr driver when it receives this return code.

A

The next feature is a csi warning snapshot. We have been doing bug fixing and adding tests trying to bring the gga in 1.20 release. We are going to add a validation webhook to enforce the api. For example, there can be only one source in the world in snapshot.

A

It can either be the persistent volume claim name for dynamically provisioned snapshots or the volume snapshot, content name for pre-provisioned snapshots, but not both. Also. We want to make sure that these fields are immutable.

A

Similarly, in volumes natural content source, it can either be volume handle for dynamically provisioned snapshots or snapshot handle for pre-provision snapshots.

A

We are also planning to add a matrix to the snapshot controller using the matrix utility functions that we added earlier in 1.19 release. The next feature is non-recursive fs group. This feature was introduced as an alpha feature in 1.18.

A

It allows volume, ownership and permission change to be skipped during the month. We are planning to bring it to beta in 1.20, and also we are planning to bring the azure file. Csi migration feature to beta in 1.20 release.

A

We also have some features that we are going to introduce an alpha feature in 1.20 release. The first one is passport service account token to csi. This feature proposes a way to obtain service account token for parts that the csr drivers are mounting volumes for, and the second feature is recovery from volume expansion, failure.

A

This allows recovery from volume, expansion, failure caused by insufficient quota.

A

It relaxes the api validation and allows the user to request a smaller size than the previous request, so that one expansion can be completed successfully within the supported quota limit. However, this does not really support volume shrinking the requested volume size must still be smaller than original volume size.

A

So we are also trying to bring this one to other in 1.20 and the third feature we're trying to bring as an other feature in 1.20, is non-recursive, secure linux. This proposes a way to speed up container startup by mounting volumes with the correct sql linux label. Instead of changing each file and volumes, recursively we're also planning to add a configuration so that csr drivers can opt in to support this feature.

A

There are also a few features that have been in design or been prototyped. The first one is the saf csr migration and that is being designed. The second one is voting groups.

A

This feature proposes to introduce a new volume group crd, that groups multiple volumes together and a new group snapshot, crd that supports taking a snapshot of all volumes in the same group to ensure right order consistency.

A

We are also looking at how to use the volume group concept to support failure domain spreading, for example, how to place volumes in the same group across different disks in the same storage system.

A

The next feature that we're working on is the generic data populator.

A

This allows a pvc to be provisioned for an external data source, not just a volume snapshot or an existing pvc, but it could be an object, store, a github ripple or an image or something else.

A

There is already an any volume data source feature gate which was introduced as an alpha feature gate in 1.80 release.

A

We want to have a generic data population implementation completed so that we can promote this feature to beta in the 1090. In the 1.20 release.

A

The next feature is the container object, storage interface. Cozy.

A

This feature allows an object bucket to be provisioned and used by pod. There have been weekly design meetings.

A

It now becomes a sub project. Under six storage, we have ripples created under kubernetes six for a prototyping purpose. If you are interested, you can join the design meetings and help with the prototyping.

A

We also have a cross seek working group called the data protection working group. This is a collaboration between six storage and sick apps. I co-lead this working group with shenzhen from google in this working group.

A

We have been discussing a number of features that we already talked about earlier, such as volume group and the generic data populator, and so on one feature the data protection working group has been working on is how do you quite ask an application before taking a snapshot and unquies afterwards, we initially proposed an execution hook, crd approach, which requires an external controller to run command through the pod exact sub resource, but during the implementation we got feedback from the api reviewers that this is not secure.

A

So now we are looking at another approach called container notifier, which is to define the command or signal as an inline part definition. External controller can request this container notifier, but cubelet will be the one executing the command or sending the signal to the container.

A

This is currently still under design discussion.

A

We also co-own volume expansion for safer sets with sick apps. Currently, we can only support expanding a volume used by a pod, but not with a staple set, so this is a design in progress.

A

Another design in progress feature is how to clean up pvcs used by a safer set when stiffer set is being deleted or scaled down.

A

If you are interested in getting involved with six storage, take a look of the sig storage community page. It has a lot of information to get you started.

A

We have a bi-weekly meeting that happens on thursdays at 9, 00 a.m. Pacific time it is a zoo meeting near the agenda doc. On that sixth order page, you can feel free to add items to the jaina doc for discussion.

A

We have a lot of bugs that we need to help with so help fix a bug and help write tests, how bright features kubernetes releases, usually every quarter, but this year, due to kobe 19, there are only three releases for every release. We will be doing planning so in the beginning of the release, then come and join our meeting and participate in the planning, and you can see if there are any features you are interested in helping out and you can get assigned to work on that.

A

Every feature must have an enhancement issue and a cap which is a design spec. We definitely need more contributors.

A

So this is the end of the session. We have some additional resources at the end of this slide deck. You can download this to take a look. We hope you find this session helpful. If you have any questions, please don't hesitate to reach out to us. Thank you.