Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022, 2 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Storage Deep Dive - Xing Yang, VMware & Jan Šafránek, Red Hat

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Kubernetes SIG Storage Deep Dive - Xing Yang, VMware & Jan Šafránek, Red Hat

Kubernetes SIG Storage is responsible for ensuring that different types of file and block storage are available wherever a container is scheduled, storage capacity management (container ephemeral storage usage, volume resizing, etc.), influencing scheduling of containers based on storage (data gravity, availability, etc.), and generic operations on storage (snapshotting, etc.). In this session, we will deep dive into some projects that SIG Storage is currently working on, provide an update on the current status, and discuss what might be coming in the future.

A

Thank you. uh My name is janja franek. I work for a hat and I am a tech lead of six storage in kubernetes.

A

There was supposed to be shing together with me, our co-chair, but unfortunately she got covered she's, fine, just she's positive and she can't go here and uh sorry for the technical difficulties. I had like a beautiful presentation. We've worked on a long time for that on that, but you won't see it so. uh For some part, we can go with all the slides. uh The first thing, who is six storage? Six storage, is special interest group. uh We look after storage in kubernetes, we have almost hey.

A

A

So well now I can show you the agenda, so I will briefly cover who we are. uh I will dive deeper into csi migration, which is one of the latest features we do and then we I will cover the past releases, the future releases and how to get engaged so who we are uh officially. We have two co-chairs sad ali from google xingyang from vmware, and uh two tech leads michelle from google android and me from red hat. But it's not only us for people.

A

We have many contributors from many companies, shingles the numbers and we have more than 4 thousand people on our flag channel on the main channel. We have more than 1 000 people on selection, just for csi.

A

Our bi-weekly meeting is attended roughly around by 25 people and throughout the time we got 32 uh different approvers in our packages and github repositories. So we are pretty big group of people, but not everybody is active. Somebody just appeared asking questions like and it's done silent, so we are always looking for new contributors and new coders. You don't need even to code. You can have different views. You can have that with trashing bugs you can fix our documentation, which deserves some work and so on so don't hesitate.

A

uh Just contributes and pull requests ask on slack. We are there for you.

A

What we do actually is.

A

We maintain the storage apis in kubernetes means, like persistent volumes, present very claims, storage classes and everything around that from dynamic provisioning volume, attachment mounting, amounting detaching, deleting volumes and related stuff like snapshots uh resizes and that stuff we maintain kubernetes in three volume plugins, and we have a deal with uh signal that we were maintaining part of the ephemeral volumes like secrets, config maps, and that does the like real storage stuff, the mounting and they do the other stuff like getting the config maps and getting the secrets from ubuntu's api server.

A

We also maintain kubernetes, sorry, maintenance, yeah.

B

A

Components: implementation of csi: we maintain all the side, cars and, quite surprisingly, we don't maintain too many uh csi drivers, most of the css drivers that kubernetes ships are owned by sql provider.

A

These are the cloud css drivers for the clouds. For example, rook ceph has its own csi driver under rook and so on. So we maintain basically nfs and couple of uh similar csv drivers, but yeah that's what we do now. I would like to dive deeper into csi migration. Unfortunately, we started late and I don't even have time so we will barely scratch the surface. So what is csi migration?

A

uh As you probably know, we have some entry code in kubernetes github repository for volume plugins and at the same time we have csi drivers that do basically the same thing. So we would like to move the code from kubernetes completely wipe it out and under the hood silently route. All the storage calls to the css drivers. So we don't need to maintain everything twice.

A

The same idea got a sick cloud provider. They want to move the cloud providers from entry from kubernetes, move it away, so the volume plugins must naturally follow. We are not migrating any data that the data state is or stays where it is, although, like the csi migration term could resemble data migration, but not we are not migrating anything anywhere. We are not even changing the api, so if you have entry volumes in three pvs entry storage classes uh and your stateful set deployments that use that you don't need to change anything in theory.

A

If we did our homework right right, you just upgrade to a version that has csm migration name enabled or you enable it manually while, while it's off by default- and you don't need to change anything, everything should work and, of course the cloud-based volumes will go first and the others will slowly follow. We are moving portworx and seth right now.

A

Who knows who will move the other ones? So brief introduction how currently entry volume plugins handle all the storage requests. So on the left, there is a user that creates entry, storage, class and pvc on the right. There is a pv controller, persistent volume controller in the cube control manager that decides there is it needs to provision a new volume, so this diagram shows dynamic provisioning.

A

So the pv controller sees the claim it sees the storage class, it sees its entry storage class, so it finds the volume plugin calls it. This is a simple function. Call nothing complicated. The volume plugins calls the cloud provider request. Provider uses some cloud, the api and provisions volume right so now the same scenario with csi migration enabled it starts the same. The user have the exactly the same storage class as before no change there. It creates the same storage class. They create the pvc pv controller wants to provision.

A

Pv controller, doesn't have any inter-volume plug-ins, they are gone. So what does it do? It translates the storage class, the interior storage class to csi storage class using csa transactional translation library, and since it has a csi storage class now it knows what to do uh with csi. It just marks the pvc for dynamic provisioning and on the csi driver side in the csi driver port. It looks this way there is external provisioner, which basically translates the kubernetes api objects into csi cores into csi grpc cores, so the external provisioner it sees.

A

I have a pvc, I need to dynamically provision it and it says entry storage class, because the translation in the quick control manager- it happens just in memory, so the external provisioner sees the entry storage class. It uses the same library, translates it to csi and it handles in the csi way and, of course, the driver and the driver across the course the cloud. So all these translations they happen only in memory.

A

We can't update storage classes. We can't update pvs in the api server. Basically, these objects are mostly immutable. Most of you probably already heard that and for good reason like if we change the objects during, I don't know attaching or during dynamic provisioning like things could go very, very racy, very, very wrong, so they are immutable. So everybody who processes, entry, pvs entry storage process now has the library they can translate it to csi and use it after that, and all the other components like external attacher or attached controller resizer.

A

Everything everybody does this in memory: translation.

A

So why is complete so complicated because the api is bad?

A

Well, it's not bad, it's good, but it doesn't allow us to change the objects and even if we could change the objects- and we would uh change the storage class and pv- and you decide that you don't like the new version with csm migration is broken and you want to downgrade you would downgrade to a version where you have the csi objects, so you couldn't really downgrade, but in this case we keep the entry pvs entry storage classes as they are, so you can upgrade downgrade as you want.

A

There is a link to cap and design. It's pretty complicated. Like the picture. I showed it's just barely scratching the surface.

A

We have a schedule, the schedule you can see. We started in kubernetes 14, that's three and a half years ago we had the first alpha, but in kubernetes 124 the first two are ga openstack, cinder and azure disk ga means it is the csi migration is on by default. The inter volume plugins are not reused and you cannot turn it off, so you must use the csi drivers there.

A

The address most of the other entry volume plugins based on cloud will follow very shortly: azure file, amazon, ebs, google, persistent disks, will follow in 175 and vsphere, probably in 126.. All these timelines is kind of work in progress and subject to change. uh We've switched this with the ga a couple of times already now it looks like we are really going for it and save rbd and portworx. We just started moving the entry.

A

Maybe it will go beta in the next release, maybe not we'll see and also when we are doing this csi migration. We are sometimes deprecating features that were in tree and are not supported by the csi drivers, for example for vsphere.

A

If you want to test csi migration, you need vsphere version 7.0.2, which is pretty recent one. So if you have anything based on 6.5 6.7, it's time to upgrade it to 7. something.

A

So if you want to test it, if you use some managed kubernetes in the cloud or use you use some enterprise, kubernetes distro consult your kubernetes vendor. They should tell you what to do and also they will handle the migration during upgrade to gi gi version. So we just follow the documentation.

A

If you are on your own and you use one kubernetes, you of course must install the cc driver, the replacement cc driver and either you test it. While the main by the cs, immigration is still off by default, uh then you need to just enable a couple of feature gates. uh Nothing special here uh there is only one catch. You need to update uh to flip them in a certain order like cube, control, manager, first and then cubelets.

A

If you do it in wrong order, bad things will happen, uh and why are you flipping the gates on notes? You should drain the node flip the gate and uncoordinate. So so, like the migrated and non-migrated volumes, don't mix are not mixed on one node. The whole node must be either migrated as whole or not migrated as whole.

A

For the features.

A

In csi migration, we support only the features that will deliver supported before the migration. So if a volume plug-in supported resize uh the same volume uh migrated to csi will support resize, but it will not get new features that csi driver offers. So you will not get snapshots. You will not get cloning. You know you must use the csi uh pvs right away. If you use entry uh that will not work, and I already told you about the deprecations.

A

Please release read the release, notes very, very carefully carefully and finally, uh that was csm migration now brief overview, what we did in kubernetes 123, if you had any issues with fs group being applied too slowly, because we applied it recruitedly on every file on the volume. There is a way how to skip it. So it applies only once when the pod first starts, but all the other ports will skip it.

A

This is now ga. uh We also allow csv drivers to opt opt out from cs from fs group at all. So if a csv driver provides a volume that doesn't have like p6 or that can't support fs group, for example, nfs shared with root squash, the csm drivers have option how to tell it. Kubernetes and kubernetes will not try to apply fast group and finally, generic ephemeral volumes, rga, which is like uh empty there on steroids, much better version of empty there.

A

A

We allow csi drivers to apply fs group by themselves, but it is only for csi drivers that have like mount option or something similar to apply fs group, all the other csi drivers should depend on kubernetes to apply the fs group and we are moving with system migration in 124 and as alpha users can now cancel volume resize if it failed on the storage backend. So well, you know the kubernetes. It always retries three tries retries, but this storage backend always says no. No, no, so you can now cancel the resize.

A

uh We are honoring reclaimed police in pvs in better way than we did, because if you delete pv objects manually, sometimes the reclaimed policy wasn't honored and we left orphan volumes in the storage backend. Now it's going to be better, we will always honor the reclaim policy. Even if you remove, if you delete the pv manually, uh we are working with sick apps to remove pvcs automatically from sticker sets and again we are moving cc migration and was the time awesome so and for the rest, I should yeah. Well I closed it.

A

Then uh I had a reason.

A

Oops. Sorry, I have a recording from sync and I hope it's going to work. The audio is going to work.

A

Can you pass the audio from station.

C

Yeah, so extension was introduced as an alpha feature in kubernetes 1.8, and it went beta in 1.11 and with kubernetes 1.24. It is finally ga this is a big milestone.

C

This feature allows users to edit their pvc objects and specify new size in pvc. Spec and kubernetes will automatically expand the volume and underlying storage backend and also expand the file system being used by the pod. This can happen either online or offline depending on csi drivers. Support command has done lots of work to bring this feature to gga.

C

There's also this recovering from resize failure feature mentioned earlier. That is also aimed to make this feature more robust.

C

The second feature I want to highlight is csi storage capacity, tracking csi storage capacity tracking was introduced in 1.19 beta in 1.21, and now it is ga in 1.24 storage capacity tracking at an api for csi driver to report storage capacity and uses that information in the kubernetes scheduler when choosing a node for pod.

C

This feature is especially important for local volumes, where capacity is bound to each node, thanks patrick for driving. This feature to ga.

C

We also have a beta feature volume populator in 1.24. This is an important feature for the backup and restore use case.

C

The volume populated feature allows us to provision a pvc for an external data source such as a backup repository, not just for another pvc or for a warning snapshot.

C

In addition, it allows us to dynamically provision a pvc having volume populated from frontbox, backup, repository and owner the wait for first consumer volume binding mode during restore to ensure that volume is placed at the right node, where pod is scheduled, when there is a request to create a pvc with a data source, the volume populated controller makes sure pv is created and populated with data from the data source and binds with the pvc to use this feature, the any volume data source feature gate needs to be enabled it is better in minor 24, so the feature gate is enabled by default.

C

Now we also worked on a few other features in 1.24 warning house. Monitoring feature allows seaside driver to communicate back to kubernetes regarding volume's condition after it is provisioned and used by a pod, so that kubernetes can report an event on a pvc or pod. If the volume becomes unhealthy, it has controller and agent-sized logic.

C

This feature was originally introduced in 1.19 in 1.21, the agency logic was moved to tubular in 1.24 we did an update and added volume health into matrix on the kubelet side in 1.24. We also introduced an new alpha feature: non-graceful, no shutdown.

C

This feature allows stave overloads to fade over to a different node after the original is shut down or in a non-recoverable state, such as hardware failure or broken os.

C

You might have heard about the gracefulness shutdown feature and wonder what is the difference between these two?

C

The graceful note shut down can be graceful only if the no shutdown action can be detected by a cubelet ahead of active shutdown. However, there are cases where no shutdown action may not be detected by cubelet.

C

This could happen either because the shutdown command does not trigger the system d, inhibitor locks mechanism that cubelet relies on or because of a configuration error to use the non-grease for no shadow feature. You must enable the node out of service volume, detach feature gate for cube controller manager manually set the out of service thing on shutter node.

C

The parts on the shutter node will be detected will be deleted. Persistent volumes attached to the shadow node will be detached and for stable sets. New parts will be created successfully on a different running node, and next I want to talk about this new ava feature control volume, mode conversion.

C

Without this feature, it is possible to create a pvc from a volume snapshot with a volume mode that is different from original body mode, thus could lead to a potential security issue if there is a cve in the kernel, on the other hand, converting the volume mode from file system to block when creating a pvc, front-wheel snapshot is being used by backup vendors for more efficient backups.

C

So we introduced this alpha feature that allows kubernetes to check whether the user has the permission to convert the volume mode. If not reject the request. This way, we can support this value use case only for authorized users.

C

There are also some deprecations and remote removals in 1.24 volume snapshot. Even if v1 beta 1 api is removed in 1.24. Please update the volume snapshot. We want api as soon as possible.

C

Csi storage capacity, viva beta 1 api is deprecated in 1.24 and it will be removed in future release least field version. Less than demo u2 is deprecated in 1.24. This is related to the csm migration feature. We recommend users to upgrade to sample u2 and higher as soon as possible.

C

1.24 was just released and we just started with 1.25 planning. I will quickly go over what we're working on in 1.25. There are a few features targeting ga in 1.25. The first one is csi ethmo volume.

C

For this feature we set the volume type to csi in the pod, inline definition and specify the driver name and volume attributes for css driver to support csi ethmoid volumes. It must be modified or implemented specifically for this purpose, a csi driver is suitable for csi ethmoidal inline warnings if it serves a special purpose and needs custom per volume. Parameters like drivers that provide secrets to your pod secret store, csi driver is a good example.

C

A sisa driver is not suitable for csi f, more inline volumes when provisioning is not local to the node or when fm volume creation requires volume, attributes that should be restricted to an administrator, for example, parameters in a storage class.

C

We are also planning to move local ethmoidal storage capacity. Isolation feature to ga in 1.25.

C

This feature provides storage usage, isolation for shared partitions.

C

There is also delegate fs group to css driver instead of kubelet's feature that we are targeting ga in 1.25.

C

We are also targeting a few features to beta in 1.25. This includes season warning health, recovering from resize failures and non-graceful. No shutdown.

C

In 1.25, we are also working on a few alpha features. Cozy is a project we have been working on for several releases now. Cozy proposes object, storage, kubernetes apis to support orchestration of objects or operations for kubernetes workloads. It also introduces grpc interfaces for object. Storage providers to red drivers to provision buckets cozy components include a cozy controller manager that binds cozy created buckets to bucket claims.

C

This is similar to how the pv controller binds tvs to pvcs.

C

Cozy components includes a cozy said car that watches cozy, kubernetes api objects and calls cozy driver cozy components also includes a cozy driver that implements grpc interfaces to provision buckets.

C

We also have the secure linux relabeling with mount options that tries to speed up container startup time and avoid changing each file on the volumes recursively and there's also volume snapshot, namespace transfer and provisioning volumes for cross namespace snapshot. Pvc. Those features are also targeting alpha in 1.25 release.

C

We also have a few features that are in design or being prototyped.

C

We also have a cross-seek working group and projects that are listed here. I want to mention change, block tracking or cbt. This is a feature that the data protection working group is actively working on.

C

This feature identifies blocks of data that have changed. It enables incremental backups to identify changes from the last previous bypass. Writing only change blocks without cpt backup vendors have to do full backups all the time. This is not space efficient, takes longer time to finish and needs more bandwidth, so we have done a poc of cbt and design is in progress.

C

We are trying to target alpha in 1.25.

C

There is also runtime assisted mounting of persistent volumes. That is a project we co-owned with a signaled. We also try to bring to alpha in 1.25.

C

Another feature is in-use protection or links. We co-own that feature with a api machinery. This feature proposes a generic way to protect objects from deletion while it is in use. We are targeting alpha in 1.25 release.

C

So those are all the features that we want to talk about today.

C

Now, let's talk about how to get involved.

C

Here is a community page that shows a lot of information on how to get started. In six storage. We have bi-weekly meetings that happens every second thursday at 9 00 a.m. Pacific time, in that meeting, we will go over the features that we are working on for every release and we will discuss about designs, talk about some prs that need attention so join that meeting, and you can learn what we are doing there. If there is anything you are interested in, you can maybe pick it up and help contributing.

C

We also have a mailing list that is shown here. So during that main list, you will get invites to your meetings. There are also slack channel. You can also ask questions on the slide channels.

C

So that's all we have today. Thank you all for attending bye-bye.

A

Thanks jing, uh I just noticed there is one thing I forget about 123: we are deprecating flex volumes if you are using flat volumes, stop please move to csi so, but that's basically what I wanted to cover here.

A

uh At the end, there are a couple of links for the newbies like uh the landing page storage, some concept uh for about tvs, pvcs and this kind of stuff. So if you are new, uh this is a great place to start, and I am now open for any questions. I'm not sure how much how much time we have. We.

D

Can he said we can go five minutes over, which is very nice of them. Remember wait for the microphone so that your question gets recorded for the recording and who would like to start and thank you jing. I think jing is watching um and there were no online questions at the moment. But if there are, I will flag you questions come on. I know this is very talkative group. I've known you all well for years, especially the front row over here.

D

Okay, here we go hang on.

E

So I was wondering about non-graceful shutdown of the node, based on what information kcm is able to detect whether the node is shut down or not.

A

Right now it is very manual process. Somebody must add special thing to a node that the node is really shut down and also the person must make sure that the node is released and will not come back in unforeseeable feature and we will start predicting stuff. Of course, people are thinking about some automation based on cloud api, because based on ipmi or whatever, this will come in the future. I believe.

B

Hello join me: we are using a very old provider which is the fiber channel. It doesn't have any csi provider.

B

What will happen with this kind of volumes.

A

So if we want to migrate entry volume to csi, we must have replacing uh csi driver, we don't have one. We didn't have a good one for csr for ice castle. We don't have a good one for refurbishment, so we are not migrating them. They will stay in three until there is good replacement.

A

At the same time, I know that all the storage- well, not all but most of the storage vendors have their own csi drivers for iscsi nfs.

A

And fiber channels, so you should start using the vendor driver if you can and uh leave the in three pvs for fiber channel for ice because they just leave them there. We are not removing the support until we have the replacement csi driver.

A

Does it answer your question? Don't worry.

D

Other questions: okay, hang on.

F

Hi, uh are there any plans for supporting look like the real production, one host prof csi driver.

A

uh Who's puff csi driver yeah.

F

E

A

Is a test tool? It's not anything like production, ready, yeah,.

F

I know I wonder if there is any plan or.

A

What's the use case, I would ask.

F

So we are using the local disks for our database and we would like, for example, to set up some quotas.

F

On the local disk right so.

A

Why can't you use the local static volumes like local volumes, api instead of csi or topo lvm, or something like that.

F

So the topol vm, it is something that brings you another layer and we are worried about performance and we require also an xfs partition for our use case. So something like a production ready host buff would be really great.

F

A

Like again, it's a test tool, we stuff like weird stuff that we need to test yeah.

F

A

We use it for testing of csi features, so we there are some hooks and this kind of stuff. If you don't want that in production, but at the local volumes like it was just local volumes, they should just work, but.

F

There is no dynamic provisioner for them.

A

Well so hold on hold on so either you want dynamic, provisioning or and lvm, or something that does dynamic provisioning or you want to disks like raw disks. You can't dynamically provision raw disks.

F

A

Maybe somebody like who brings the disc and connects it to the machine right.

A

You can call it dynamic provisioner, but.

A

A

F

Well, if you create directories dynamically on some partition that on some partition, that is attached to your uh volume and you are able to set quotas on them, this.

A

F

Like it is dynamic, provisioning.

A

Yeah it is but yeah. I guess this is a special use case. Like uh gather some community write, a css driver.

F

I did, I wonder whether I'm duplicating the host puff so.

A

Yeah, we don't have such plans. Okay,.

D

I like his idea, though it's community gather the community and build it right all right here. We go probably have time for like two more questions.

G

No, I think I because we we are we've just built. We just released an open source operator for databases and actually, I think that's a use case that we especially on bare metal, you can you you might want to explore so yeah two years ago, at in san diego, I was actually talking asking about. I mean the same questions to rook people.

G

They took me for crazy. You know that time. I remember that, but I think we can talk more yeah, but I think, as I mean creating logic, I mean.

G

Physical volumes and logical volumes on the fly also with separate permissions. I think that's that's something that I think could have a use case. Maybe it's niche, but for high-end.

A

Like feel free.

G

Because yeah yeah, okay, you have contributed yeah, we already contribute to the database, so I think if we can get some help on the storage, that would be good. Thank you.

D

All right, I think, that's probably all the time we have but jan can probably stick around for a few minutes. If you have any.

E

Extra questions.

D

For him, but I just want to say thank you jing. We hope you feel better soon. Thank you, young can we give him a hand big round of applause. Thank you.