Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2018 (Copenhagen), 14 Nov 2015

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Policy-Based Volume Snapshots Management in Kubernetes - Jing Xu, Google

Description

Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).

Policy-Based Volume Snapshots Management in Kubernetes - Jing Xu, Google (Intermediate Skill Level)

Malware and unexpected file corruption leave your data at risk. Storage snapshots -- either on premises or in the cloud -- can offer protection. In the event of a problem, user can roll back to a snapshot from a specific point in time and restore the storage system to that prior state. Snapshots can also be used to replicate data efficiently. In this talk, we present Snapshot feature in Kubernetes. It allows users to create/delete snapshots and restore/clone volumes from snapshots in the same way of managing volumes. We also show to how build custom controller to manage snapshots based on pre-defined policy. The policy specifies how often to take snapshots, what commands to execute before and after taking snapshots, and what is the maximum number of snapshots are allowed. The custom controller can automatically execute snapshotting workflow and clean up old snapshots automatically.

About Jing
Jing Xu obtained her Ph.D. from Electrical and Computer Engineering Department, University of Florida in May 2011. After graduation, she had been a lecture in School of Computer Science in Florida International University for about 4 years. She moved to Bay area in late 2014 and joined VMware, working on Storage Policy Based Management project. Later in 2016, she joined Kubernetes team in Google and now is working on storage management.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Good afternoon, everyone thank you very much for attending this talk, and so today, I will talk about policy based management of volume snapshot in kubernetes. My name is sting I'm, a Google software engineer, working Inc when at his storage lifecycle group and my co-author Anthony, he is also Google engineer. He is working in communities, local Oh team.

A

Unfortunately, he cannot join us and that this is thing you want introduced. Yeah, hello,.

B

Thank you me. Hi I'm Shinyoung I work on open, STS, I work with a gene on adding snapshot, support into CSI and also add a snapshot support in treating kubernetes.

A

String and I leading the snapshot project in 6rh. So if you have any questions feedback in case, you don't have taught the time to talk to us and leave the contact information at the end of slice. Yeah. Your questions, feedback always welcome.

A

Okay, this is the overview of this talk as gyro some background, the snapshot and the concept of cloud volume snapshots in kubernetes forums and with this background, I introduced the kubernetes volume snapshots, API design and the workflow based on this API, and there is the volume snapshots API support. I'll talk about how to use mana controller, to design a snapshot. Schedule policy in the house used controller to manage the policy.

A

The last section I will talk a little bit about some related work is basically the in-place restore workflow. We are currently working on so first thing why we care about snapshot. I want to use a quote from Tim berners-lee, the inventor of world wide web. So theta is a pressure thing and will last longer than the system themselves.

C

A

Is valuable but the system we used to store data might have problems either in hardware like hard disk failure or in software, so data might be corrupted and user might also delete their data by mistakes. How we can protect our data right is valuable data, so snapshot is one of the commonly used way for this purpose. Basically, it is a read-only copy of the data at certain point in time and a snapshot comparing to the data copy or take a clone.

A

It has some advantages, so it's very fast and it has typically little performance impact on the running application and consume less capacity.

A

So considering the cloud environment, so we have the concept of code volume, so data will be stored on the cloud and crop volume. Basically, it's very similar to just a normal hard disk, but this can be attached detached to different virtual machine instances. So user has the flexibility to run their applications different places, so it's no longer tying to a single node and many cloud providers, support snapshot to their volumes and normally it's fast. It implements as incremental step shots and the snapshot data will be uploaded to the cloud.

A

So user can access the snapshot for some cloud provider like it applies. Their snapshot is available in a single zone, so user can access as a snapshot from that zone. If you want to access snapshot from different zone, you need to copy the snapshot to those domes and for Google Cloud snapshot itself already available. That means you can access your snapshot and you anywhere you want.

A

As we just mentioned, the snapshot is very for protecting your data and in cloud environment. You can see it's also useful for a rep layer data to different places.

A

So before going to the design of autumn snapshots in communities, I will just use a couple slides to give overview of kubernetes volume concepts. If you attend studies talk early today, you probably already familiar those API objects, so in kubernetes normally most and we are represented by REST API s and for volumes we use a pair of API tremens it from user point of view. We have persistent imodium claim. It is a request for the storage and the user can specify how much storage they need.

A

What type of storage it is a namespace API, but API objects, and when usual some may submit such requests. The controller will provision a volume dynamics way or might find a matched volume existing volume, depending the configuration and the volume information will be stored in a another. Api object called persistent volume and the controller will bind these two objects together, which means they have pointers to each other and once those ap objects abound, user can't use their using those models, so they can in their pass back, they can reference.

A

The PVC and the lifecycle of PVC PVC will be independent to their paths using them now, I will show some yam file 2 over the PVC PV as a simple example. So for user, if they want to create and use some storage, they can create PVC m, it's very simple.

A

In this bag you specify specify how much storage you need and the access Mo's and when you skip control, to create such an file and a PVC object will be created with the information we just seen and controller will go ahead to provision a new volume and also a PV API objects represent this volume. You can see the taste information like the name will be stored in the PV, and since this is none namespace API project only system being can access. This information and the controller will bind these two object together.

A

Now the parts back and start using this piece of storage by referencing, the PVC.

A

Now is the concept of kubernetes volumes. We can go ahead, talk about the volume snapshots, API design, so for step, router workflow. First, we want to support, created, snapshot and depending how you prepare your application. Before taking snapshot, we can have different types. The first is the simple one is random, so you don't prepare your application at all and you just take step shot at certain point, but because application mind can use writing data on to the disk and you might have considered the issue.

A

So it's suggest to rephrase the I/os of your file system to achieve some crack consistency in the file system level or for some applications. You might need to pause application flush, the disk and mana disk before taking same shot to achieve application level. Consistency currently in the correct work, volume snapshot design. We only support the random and in the future we plan to support other types and was the snapshot is created.

A

It has might have additional face for cloud providers so for cloud provider it will upload the bottom snapshot. Data to the cloud this might take. Some time varies from minutes to even hours, depending on several data size, but for Chris time shot itself should be very fast. It should be instance.

A

And after taking snapshots, you can use snapshot to first to create new volumes and to create new volume from seychelles. That means the data will be populated into the volume. So, instead of just empty volume, you create as before and at same time many cloud provider. Allow you to change your volume configurations. For example, you can change the type of your volume from standard to SSD and you can also increase your volume size.

A

So you can see. This is also a use case for using snapshot to change your volume configurations and DV snapshots is straightforward and you can delete your snapshot from cloud and now for the API design. Volume snapshot is very similar to PvP VC design, so we also have to AP object to represent it from user point of view, you have a volume snapshot, visitation request for taking a snapshot to a volume, so in the bottom Seshat spec you specify you'll have a reference to the PVC.

A

That means you want to take that route from that volume and once a warrior on a snapshot is taken system will create a volume snapshot, data API object which will store the detail. Information about the snapshot like the ID and controller will find these two object together. Just like PVC pv a pair and the left cycle of volume. Snapshots were also independent to their pv pv c.

A

Now we also use them file to show the workflow of creating snapshots, and you probably see this is a very simple um file and in a spec you only need to specify your PVC name. That means you want to take a snapshot for that particular piece volume and when you use cube control to create such volume, such snapshot, a volume snapshot API, which ID is created and also a lot of structured data.

A

Epitaxy is created in non namespace, so you will see the snapshot, ID and also creation, timestamp accenture, the status information in there and the controller was also buying these two API charts. Api objects together.

A

Now, after that, you have volume, snapshots, API object available, you can use it to create new volumes, so here you can see to create new volume is almost identical to the workflow. We just show in creating volume before the only difference here. Is you specify your snapshots name into the TV cm file and currently because the snapshots implementation is out of tray, so we use a notation to put this information. You may know also notice. We are storage class name here, because also because out which way we use external provisioner.

A

So you need to specify this provisional name.

A

Okay, similarly, so a PVC AP object is created and also a PV object and the controller will bind them together, but this time you will see the data will be populated into the volume instead of creating an empty volume, but for user point of view, the workload operating volume. Almost the same, you just add your snapshot. I information into the gem file.

A

And the in the current state we have a auto snapshot: API support in external storage, so Oh, the old API objects are actually CRT custom resource definitions and we use external controller and external provisioner to manage to create delete, snapshot and provision morning from the snapshot, and we are working on pushing this entry as alpha version and also add CSI support.

C

A

With the support of for team snapshot, API objects. Now we talked about how to use a controller to manage snapshot policy. So just now, when we introduced snapshot, workflow for user right take create a single snapshot whenever they need. However, you managers kisses system in one tool, I, say periodically, take snapshots everyday and also delete snapshot, for example after certain period time.

A

So that means we need a schedule and we call a snapshot policy and also a controller to react based on this policy, and such policy should be also represent by API object. However, we don't have any building AP object to represent such a policy, so CRD would be very useful for this case, so we can create ERD for policy type and create policy, but by just creating policies, CRT itself won't do anything. Just like all the other AP object.

A

You need a controller right to define to manage based on this policy, so that is what we call custom controller.

A

Normally in custom controller. You need to have some implementations components such as sharing forward to keep watching the update from ApS server and the reconciler to create a control loop to manage the state's its ancillary work. But good news is for mana controller. It is reframe work provided by my my co-author Anthony and by using the manner controller framework, all these common components already provided for user point of view. They only need to create their CRT for their.

A

Let's say in this example, this natural policy and also to put their business logic into the sync web hook and the Rican seller will create a control loop on your behalf and call the sync function periodically or when it's needed.

A

So you snap shot policy, as example in the CRT definition lets. You create a new type, which is called snapshot policy and with this new type, I can create my own snapshot of policy policy, one and I put in this bag. I snapshot interval like six hours. That means every six hours I want to create a new part, new snapshot and also what is it spirit that means after 10 days, I want to delete that snapshot.

A

This selector will match labels. Basically, it will find the resources that have the labels that means issued controlled by this policy.

A

And the let's look at the controller side, so magnet controller provides a view, controller patent, why it's got a composite controller.

A

Basically, in such controller, you defy a parent resource and at orange sauce, and the parent resource in this case is, would be your step short policy and children resource will be the PVC and snapshots. You want to managed via policy, and you can also define like config.

A

How often you want to sync in this national policy controller- and this is would be the logic you put in your sync function, since this is a web hook, you can use any language you want and in this thing function is very simple so for it we'll have to control loop for each snapshot like managed by this policy.

A

Your check, the creation time, if it's larger than the validation, whether it period you need to delete this snapshot and also it will check each PVC under this policy. What is the creation time for the last step? Shots taken four days PVC and compared with the temporal interval. If it's longer than the interval, it will go ahead, create a new new snapshot for this PVC.

A

Okay, it's time for just a simple demo to play safe, I.

C

Will use this video to show you I.

A

Hope it's a large enough, so you can see it so. First I'll check the controller yum file, so you have the custom CRT definition and converted controller. You define parents, resource children, resource and also the department to run your controller by your skill control to create your snapshot controller, you see, you have CRD created the controller created and deployed, and now I create a new policy in the policy spike I specify it takes a shot every 10 seconds and the valid period is 20 seconds now.

A

I can apply this policy and now you have policy and you have controller running I create just a number PVC and in this PVC I repressed, some amount of resources, and now we can check for volume, snapshots and right away. You see we have two snapshot taken already so every 10 seconds, and also since we have validates period 20 seconds, so you see the otherwise deleted and new eyes created now.

A

It's time I want to change my policy, so I changed from 10:00 to 5:00 forceps out interval and the periods make it longer to 50 seconds and I don't mean I only need to apply this policy again, miss cube control, and now, when you check this, you can see immediately every five seconds a new snapshot is taken and the old one wants to be here until exceeds 15 seconds.

A

So indeed, is similar demo, but from this demo we see the potential of the possibility that you create complicated, high-level policy controller based on your knees, so with volume, snapshot, API and demand our controller framework, it's very easy to create your own, and you can modify your policy file and apply it anytime. You want and you can customize the logic.

A

For example, in the step shot a sync function, you can also trigger a job to prepare your application before taking step, shot and also resume application after step shot is created and also you can manage multiple policies by using the labels and you can specify which sub resources you want to manage by this policy.

A

Okay, so now it's the last part of this talk to talk about the restore workflow, so this lies. It's basically remind you of our crazy morning from seychelles workflow. So remember, the only thing you need to do is to add step shots information into the PVC young file, and then the controller will provision a taste from the snapshot and the au pair of PV PVC will create and bind together. However, this simple approach have some limitations.

A

First, you need to modify your PVC source file in many in sub use cases that met applicable and also for PvP BC. It's my currently use. That means a part. May reference it and running your application use this storage. You cannot actually change your PVC and restore your data into that PVC. So some manual step you might need to do is, for example, including delete your part first and then you need to delete your old PVC and pay attention. Your original volume might also delete it.

A

If your retention policy is delete, and now you can create by modify a PVC source file, you can create. A new PVC valve is the same name and after the volume is probe agent and pv pv c-- point together, and you can resume your part which reference that PVC. So this whole process. That means you need to tear down your pod right. So you may have a long long time because of this.

A

So to overcome the limitations, we propose a in-place restore volume workflow and we can use a new API project called restore volume in the API. Without you specify what is the snapshot source and what is the persistent, the PVC name and in case this PVC is currently used by some pot and when user created such restore volume and the controller will based on the information first to provision a new volume from the snapshot and use PV Prime to represent it. This step can be done independently.

A

It won't affect existing PV PVC and when the volume is provisioned after is provisioned user can decide to delete a part, and the controller will detect not at this time, PVC no longer in use by any part and quickly switch the point pointer to the new PBS and right after that, your new part can be resumed.

A

So in this way you still have your PVC, it doesn't change, but the pointer bind pointer will point to the new PV and your pods will start using this new volume and the Tumtum time, for a part, is only basally to switch your pointer from the old one to do one. It should be minimized. Your downtime, and we consider this in place restore, would be useful for some use cases. For example, you have set four sets the applicants running, but one of the discs is his problem. You want to restore from a snapshot.

A

You can just create a restore volume. If you object to achieve this, and also it might be useful for, like loading updating scenario,.

C

A

So that's most of our my talk and in summary, so currently we have a out of tree volume snapshots implement in external storage people, so for experimental you can try it. It provides all the basic functions. I just mentioned, like create DD snapshot and create volume from the snapshot, and currently we support a few plugins, including gcpd.

A

It arise of star cluster and host pass and we are actively working on snapshot feature, for example, to add support for CSI in also pushing the water scheduler API entry and work on the in-place, restore workflow and provides the snapshot policy controller. So those are all ongoing. Your work and your feedback and questions are welcome.

A

So, finally, this is a teamwork, besides our Google team, and we have outside contributors from open STS from Red Hat, and this is our company information, yeah and my interest rate Isis know- and you can involve in this. Thank you very much.

A

So we can take some questions.

A

No, so to provision volume from some shots, it actually don't need snapshot. Controller would flow. It is the basically follow the same volume creation workflow. So it will just check your since you have a snapshot leak where protect, so it will get snapshot, API object and find the snapshot ID and then pass this information to the provisioner and provisioner just go ahead to add this snapshot information when crate steps, event create volume. So it's not related to controller snapshot, controller.

A

Scarola is using external provisioner yeah, so, but after we move into a rat, it's basically the same workflow when you create a new volume right, so it goes through the provisioning workflow. It's not depends on the snapshot controller itself. Central controller will just responsible for creating tv8 snapshots yeah.

A

Thank you. Yes,.

D

Is it possible if you suffer catastrophic loss to to reconnect a snapshot? That's been taken previously, in your say, a new cluster in the fact that you've taken an EBS snapshot. If you are Amazon and now you're restoring your entire entire cluster. And now you need to connect that back to the snapshot manager system.

A

So you mean, like you, create a snapshot outside and then you want to make it available in communities. Yeah.

D

A

D

Created it on the cluster, that's that's died and now you're bringing up a new cluster with this, and it's been created not known by it.

A

So I think your question have two aspects: why is you you create you to use a plan to create step shots through the kubernetes api? Okay yeah? So you have snapshot AP objects' available, but you want to use it in another cluster when you bring it up. So as long as your new cluster can access the same API server right, you have your snapshot. Api objects available, so you can retrieve from that API server and you can be able to use it. Yes,.

A

So then depends right because currently order snapshots ID, like information, is stored in the API in the snapshot. Ap objects. So if you don't have other like record of yes yeah, you can still yes.

A

E

If you have a snapshot taken from the storage system, the external storage system, can you import into kubernetes and use.

A

Yes, I think it's possible. We didn't like work on that specifically, but if you have taken a snapshot outside and you can probably manually to create a volume snapshots, ap objects' and puts manually put information into those objects.

E

Yeah: okay, thanks.

A

Is possible, but it's the main hate you more, some extra work to do it. Yeah.

F

Thank you. So my question is about the meta controller. Do you recommend creating you know? Operators or policies are using meta controller. I have been trying better controller on and off, but it's been a little bit difficult for me.

A

Yes, I think it's my have a learning curve, but the good news is like Anthony who is in charge of this project. He is continuing working on that and also I will show you he has new, like here. I think he is still working on that, but it's basically have a lot more information: how to use the man that controller and it has much richer information and help you to understand the workflow. So when I start working on that also have some learning curve, but after a little bit, I think is give us.

A

It was very powerful in kind of framework to work on to design your own. Yes, so I definitely recommend using it. Yeah. Thank.

F

E

G

We have more than one policy for same snapshot, object, yeah,.

A

You have have many.

G

Policies so we'll do the policy validation, then. Is there any quantity in the policy who will do the validation of the policy who.

A

Will do the valuation for.

G

A

So country is just like us small time all working on this really have like validation. So what do you mean by validation? Your policy like whether it is feasible, yeah.

G

A

You define an interval.

C

G

Is conflicting each other to delete a policy so to delete a snapshot or creation of a snapshot? So if there is a conflict in the policy, when you define more than one policy for this names never shot up yet so.

A

Normally I'm thinking your policy should manage a subset of your resources unless you have overlap among different policies right, so each policy should in charge of a like different group of resources, and so your policy will depends how you implement your business logic right. So you can have some validation into your policy into the business logic you put in the sync function: okay,.

B

A

A

Okay, so I think that's the end of my talk and thank you very much, and so your feedback questions yeah welcome. Thank you.