VMware Velero Office Hours, 22 Oct 2020

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Velero Office Hours - Oct 22, 2020

Description

October 22

Is cross cluster migration on the roadmap?
We do have support for it.
You take a backup of a cluster and all the cluster objects are in the configured object storage.
When restoring a new cluster, you will need to make sure that the new cluster has access to the object storage bucket where the backups from the original cluster were written to.
Velero will then read those backups for the original cluster and restore them to the new cluster.
For cross cluster migration of persistent volumes, it is necessary to use restic.

Note: There are some details missing from our documentation on how to provide credentials to Velero.

How does the Velero CLI communicate with the Velero server?
Is there a REST API? Communication happens via the creation of CRDs. The Velero controllers process those CRDs and as part of the reconciliation loop, will start the backup/restore operations.
Velero communicates with the Kubernetes API server, not etcd.
Velero is purpose built for Kubernetes using Kubernetes.

When backups are in progress, we can see the state and logs. Is there a way to view the backup progress?
In Velero 1.4, backup progress was introduced. There is a field on the Backup object which details the number of items to backup and the number of items that have been backed up so far. We don’t maintain a list of the items to be backed up as it difficult to keep this up to date during the backup progress. However, once the backup is complete, the list of all resources that were in the backup can be viewed using velero backup describe "backup-name" and checking the Resource List. It also differentiates between Kubernetes resources and restic backups.
TODO: Details of the progress item are missing from the API documentation. We should add this.

Demo of Velero using MinIO
MinIO running in a docker container outside the cluster. Instructions to run this are available on our FAQ page
Using velero-plugin-for-aws as MinIO is an S3 compatible storage.
Using CSI hostpath driver to provision volumes on a KinD cluster.
Restic demo:
Enabled using either opt-in or opt-out. More on this here

Getting started with contributing to Velero
Please see our docs on
Contributing
Joining the Velero community
Twitter @projectvelero
Kubenetes Slack #velero
Google Group

A

Okay, so we started the recording for this uh office hours uh a little late, so this is the october 22nd office hours. um We have uh samira who's who's, a community member who's just joined, and we were helping her uh with the cluster migration scenario and uh walking through how cluster migration works.

A

So just for the recording purposes, I will start over um or say whatever I said again so the way cluster migration works today is you have an object store where all your backups from your clusters are written to and to migrate the workloads from one cluster to another? The way you would do that using valero is, you would run valero and grant valero access, whether on the new cluster and granted, access to the object store where backups were written to and valero would then read those backups and restore them into this new cluster.

A

So that is how cluster migration would work and we also have a page on our website for cluster migration. I know that this page can use some tender, love and care. So please refer to this. If there are any, if there are things that aren't clear, please open talk, doc issues or even better, yet open pull requests to help us make our documentation better.

A

uh Samira. Have you looked at this page.

B

A

B

Tried a couple of pocs on this part so tried migrating from aws, uh sorry azure to google cloud. So I tried this scenario like uh during this process. I've observed, uh I I'm not sure if these things have been fixed recently, but uh there are a couple of minor things that I've observed. During this I mean the whole process was very good like I liked the entire thing. So what happened is like uh I was just installing in as in a valerian and azure cloud, and then we have this key to be a good account.

B

We have to supply the key right, though that's in the deployment yamala uh valero controller, so I mean that isn't very visible for me on the dock over the documentation.

B

So, okay, probably I mean I found that in one of the github issues and I've contacted the slack team of valero and probably uh these kind of minor issues. If we can include in the documentation, it would be helpful.

A

So this is around documentation of giving your giving valero in your azure cluster.

B

A

B

A

Okay, cool, uh I think that's that's a valid point, and this document does not address that. uh We we can definitely um help help. Take care of that. uh If, if you uh well so in the end, were you able to uh uh yeah.

B

I was able to migrate yeah.

A

B

Taken some help from the slack team of england and someone else said you helped me over this night.

A

uh So if you, uh if you wouldn't mind, would you uh would you be able to like put those instructions of the steps that you followed in into a document and open a pull request for us yeah.

B

Sure definitely I will do that.

A

Yeah that that would be super helpful. Thank you.

A

And were you also just uh just out of curiosity but you're, also migrating uh persistent volumes in your uh migration scenario, yeah.

B

I did I did your system.

A

B

A

You were using restricted, so I believe yeah.

B

A

B

uh Rustic I've used it on a nice rustic. Also, I've tried, but uh I've mostly used the storage, persistent volume using the cloud provider. Okay, storage class. I've tried both the scenarios. Okay.

A

But for migration you would have pro you would have used drastic.

B

A

Yeah cool, thank you for offering to help with the migration our documentation stuff. That's going to be super useful and helpful for everybody in the.

B

Community also, on the controller side, we have the client community communicating to the controller over the rest, api right.

A

So the client, when you say client, are you referring to the developer.

B

Is contacting the valero controller through a rest api so uh since.

C

B

That all the communication is happening I mean, even though I've read the architecture and some I'm not sure if I'm missing or something but I've read some internals of valero, but uh in the internals of valero, I've observed that valero even contacts the uh api server and not the etcd directly for any kind of information.

B

So uh is it possible that to get the valero controller out, uh I mean not all because you see this has to work uh when probably the kind of uh the kind of things where we just contact aps server for minor things for the creation or the crud part, we can keep the controller there and uh can we make it a bit lighter.

A

So they fell uh the valero client there yeah like you're right. There are two components that the velaro server, which runs all the controllers and then the valero client, but the client and the server don't communicate with each other directly.

A

So the way the communication happens is via creation of custom, resource definitions or custom resources, so the valero client is very light. It's it's just going to perform some validation on the client side and create say, let's say, for example, backup objects which the controller running in the cluster cluster will recognize and as part of its reconciliation, loop will run the backup process.

A

Doesn't make sense.

B

Yeah yeah, it makes sense.

A

Okay, so the uh there is no rest api per se, but um the valero client just creates.

B

Objects in the cluster.

A

And then, as far as valero communicates with the api server and not cd, yes, so all the backup as part of backup and restore all the operations that valero runs, is going to go through the api server directly or to the api server and not to hcv directly that that is why this uh valero backup is like purpose built for kubernetes using kubernetes.

B

So we have this valero uh status so where, once the backup has been started, we have the provision to describe the backup and see the logs of the backup. Do we have any kind of watch mechanism where we can see where the.

B

So when we have backup, we have the provision to describe the backup and see the state of where the backup is and see the logs of the backup. Do we have any provision of seeing uh the watch status like where the backup is uh what are like? What are the things that have been backed up.

A

Yeah we re in as part of valero 1.4. We added this new feature of backup progress. um Let me which, which is constantly like updated as the backup is running, uh I'm trying to pull up documentation uh like our in our api types.

A

A

Okay, I don't think we have this website uh this, this documentation updated, but I can bring it up onto our.

A

A

A

If you look at our api types.

A

We have this uh struct, which is in the backup, object itself.

A

If you see in the backup status this item, this object or this field gets constantly updated, as the backup is running and has the backup processes running. So we have a running count of uh the items that have been backed up and the total backed up items, but we we don't maintain, like a list of each item being backed up uh mainly because of it's hard to maintain this, keep this updated in a timely fashion.

A

However, when you like describe the backup with details once the backup is completed, we have the resource list in the backup which which will have everything that's in the backup.

A

So this is what I'm talking about.

A

So this is the resource list uh which will have all the items that were in the backup. So we have events, we have this namespace that was backed up. We have these pvcs or these pvs, these pvcs and pods and etc.

A

So that is what it would have- and this is the this is coming from the progress that I showed you like. We had 24 items to be backed up and we backed up all 24 of them.

A

And this also includes differentiates between like valero native snapshots and rustic snapshots, plastic.

A

Backups did that answer your question. Yes,.

B

B

B

But I really like the cluster migration part of it's really awesome.

A

A

B

I I think from my side, these are the questions.

A

Okay, cool and as part of our regular uh uh agenda we have. We wanted to do like a demo of how backup and recovery in valero works, and uh once we are through with that we can, we will also go over how we can start contributing to valero.

A

uh I think so, uh one of our members in the community added this question. uh I am going to uh I'm going to say: let's wait till that person shows up in their interest so that they can ask that question straight up. uh I think bridget had a demo that was prepared richard do you want to uh run through that.

C

Yeah, I can do that um absolutely um so, just in terms of like what it was I had prepared. um I just want to make sure that like uh for this, is this just.

C

I don't know um so somewhere. It sounds like you've already got like a lot of experience with lara, so I don't know maybe like what I'm about to do might be too basic. So I don't know whether we want to carry on and do this just for the recording, um so other folks can watch it later um or whether I don't know where I'm going with this.

A

Yeah, I think, for the purpose of the recording and for folks who will watch it afterwards. We can do like a basic demo. The demo can start off with as basic as possible and we can build on it if we want or if time permits, sure.

C

uh Let me um I'm just going to need.

A

C

Going to get some things set up here, I've kind of gotten there, but I just need to.

A

C

A

Yeah, oh I'm, I'm sorry, I did not ask if my terminal was visible when I was showing all this, so this is the resource this time is talking about. This is coming from the progress field from here. Make this also a little bigger.

A

So this is coming from the progress field, which is which is part of the backup status, and then these are all the resource resources that are in the backup which is just listed in the backups describe output. So this is the command that I ran. V is my alias for the valero client. That is on my laptop that I built from main. So, oh, that's, that's the command that I'm running. Nothing super.

C

C

Okay, I think I I'm good to go here.

A

Okay, so I'm going to stop sharing and let you share. Okay, your screen.

C

Okay, so I'm sharing, I can to see the size and everything here: okay,.

A

Yep, it's fine for me great.

C

So I'm just gonna move this over to one side. So um so I I've set up a cluster locally, um which has valero um already installed so to install balero. I've used the valero install command so like I'm also using v, as my alias just for a locally built version of valero that I have, and so I've installed it.

C

I'm using aws is the provider in the aws plugin, I'm not actually using aws directly, I'm actually using mineo, um which is an s3 compatible storage which I'm just running locally in in the docker container, and so, whenever I've installed valero I've set it up so that um I've passed it um the credentials. In order to talk to this bucket or sorry.

C

In order to talk to the storage, I've got a bucket um here for where I'm going to store the backups, and then I've also passed um the configuration details to valero as well, so that it knows how to communicate um with this particular storage. So I've got the s3 url, which is just a local um ip, and then I'm also going to enable rustic um for the backups I'm going to create.

C

So whenever um so, after I've done this installation, we can view valero running um in the cluster, so I'm just going to do get pods and by default valero will create all the resources that it needs um inside the valero namespace. This is configurable, um but for my purposes I'm happy to just let install on the blur namespace.

C

So I do get pods. We can see that we have a rustic pod running and then we also have the mean valero pod, and so they so the vlara runs as a deployment, and so this is just the the pod that's been created from that deployment and then wrestling runs as a daemon set on, so that there's a pod running on every node in your cluster, I'm just running a kind cluster locally, so I only have one node, so I only have one instance of rustic okay. So um so right before this, I.

C

I've been uh so I've been working with ashish and he has been pointing out some really useful tools within um the csi space for doing um for being able to use. So the one thing I have installed on my cluster um is the csi host path. uh Drivers. That just means I'm only able to create persistent volumes and claims uh using the hostpass drive host path, driver locally- it's not really relevant, but with that project.

C

um One thing that is there is that they just have some um small example applications which you can deploy into your cluster. So that's what I have here. So I've got this namespace called csi app. So if we take a look at what is running in that namespace and see that we've got this one pod, I'm just gonna describe that to show what is in there. um So this is just a pod.

C

That is um it's not really it's it's running very basic commands, but within it it also has um some data mounted local um some data from a volume that is minted. um So what I'm going to do is I'm going to perform a backup, I'm using valero of this namespace, and so hopefully, with this, um the blur backup is going to back up um all the resources in the same space, and that includes a persistent volume that I have a persistent volume claim in csi app name space.

C

So let's perform a blura backup so to create a backup. You simply just use the valero backup create command and then we're going to give it a name. So I'm going to call it my uh I'm just going to call it app backup and whenever you're creating a backup you can. There are lots of different configuration options that you can pass to valero.

C

So you can backup specific name, spaces specific um you can choose to include or exclude namespaces, and then you can also um instruct valero to um include cluster resources or not in the in the backup. So if you check out the flower documentation, uh there is um backup reference, uh so you can so I'm just gonna make that a little bit bigger. So you can exclude different um specific items and also specify some orderings.

C

I know that there is, I think there are other some other configuration options for backups, but I think for now, for this particular um example: we're just going to create a backup, I'm going to say include name spaces, I'm going to say just back up my csi app namespace.

A

Legit, I pasted a link in the zoom chat which takes you to the resource filtering docs. I think that's what you were looking for.

C

Excellent, yes, um I'm gonna open that up just so that people can see it folks can see it in recording after this is over.

C

Sorry um I.

A

Sent it in zoom chat. Sorry.

C

Yeah I'm trying to find zoom, I don't have the zoom chat open. Where is it there? We go sorry. It was a very big bright screen.

C

Okay, so yes, this is the resource filtering, so resource filtering, um so you can say include particular name: spaces include resources include cluster resources, and then you can also apply um selectors. That was the other thing I was trying to remember, but couldn't um so if you only want to back up particular items that have a particular label applied to them. That is an option that you can use as well.

C

C

So uh whenever I ran the um blur backup command, what it did behind the scenes was that it created a backup custom resource um on the cluster. So if we take a look at um uh get backups, so we have the app backup that was created. um So whenever that custom resource was created, the the backup controller that's running from valero um is running um it's running on the cluster it once that um resource is created, the controller processes that and then starts the backup operation.

C

So we can feed the logs for that back up back up right, yeah.

C

uh So yeah, so here we can see.

C

We've got a whole host of blogs of uh various things that um valera's giving us information about, but we can see that um it's decided uh which items it's going to back up and then we can get um a log line say that then has been processed and that's been backed up. But at the end um backed up six items.

C

I've nested a total of six so backed up all the items that blair thought we need to back up, and so, if we take a look um at our object storage, um so this is where the backup has gone to. It's gone to my mineo um uh object: storage. To take a look in backups, we have the app backup- and here are all the the different um resources representing a backup that valero has captured.

C

So we've got the json which describes the backup, um details and results, and then we've got the logs and then also um various other um backups as well.

C

So um so now we can delete the namespace and then we've got this backup. We can delete the namespace and delete all the resources that are in that namespace and then we can use valero restore in order to restore them back into our cluster.

C

So I'm just going to delete namespace csi.

C

C

C

C

I don't know why that didn't uh terminate um okay, so the namespace is gone, um so we no longer have have the pod that was running um or the the data and the persistent volume that um went along with that. um So now we can do uh blair, restore create and.

C

So the so I'm going to tell the layer to do is perform a restore from the backup that I created so just for a reminder.

C

Backups sophie restore, create.

C

C

Backup and again um this time a restore object has been created and the restore controller from valero is running and is processing that line. So if we do blur restore describe, um these are the details, so all the resources that were in the backup will be um restored, so we so one thing you need to do as well as filtering resources on backup. You can also choose to filter the resources at restore time as well.

C

So if you have um a backup that you've taken, that includes all your cluster resources, but maybe you only want to restore a subset of those. You can apply some filters um as well at that stage, but for this case I'm just backing up everything that was in um or sorry I'm just restoring everything else in that backup. So now, if I do, if I check the namespaces that we have, we can see that the csi app has um namespace has been created again and.

C

Yes, then, we have our csi app up and running.

C

That makes sense so far. So, okay, actually, if I missed anything uh very obvious.

A

uh No all all looks good.

C

Okay, great um okay, so next stage and what we're going to do is um we're going to I'm going to perform another up and restore. But this time I'm going to make use of rustic, and so I'm going to use jurassic to back up the contents of some volumes that I have which are attached to this pod.

C

So if I do yeah, so I'm going to just exact into the csi app um container that um within this pod.

C

C

So I'm just going to do another describe of um this particular pod to show it my.

C

So this pod has a volume attached to it and this volume is mounted into this container as flash data. So in the within this container um I am going to write some some data or a file to um to that to that directory and ultimately to that volume. So if I go to data there's nothing in this directory, but if I do.

C

Excellent, so we have a file on the um in that directory and in that volume, so I'm going to do now is perform another backup, and so this time I'm going to um create the backup um so there. So whenever you want so in order to perform a backup using brassic, there are two ways to signal to valero that you want um breaststick to be used. So the first way is that you can um annotate the pod.

C

um You can find an annotation on the pod which valeria will see and those to look for, and if that annotation is present, it will um use rustic to um to back up the volumes. However, um with 1.5 um we have a there's, a new thigh has been introduced, which is called default volumes to rustic, and what this lag does is that if there are any volumes which are present, which she's correct me, if I'm wrong, if there are volumes which are present, which don't have snapshotting capabilities associated with them, is that correct?

C

So if you can't perform a snapshot, then um we'll default to using rustic to perform those backups. So I saw on the little screen that you know that so I'm gonna assume that was okay.

A

That yeah, if, if your volume provider doesn't have snapchatting capabilities, then you could use that or like in samira's case where they were migrating across providers. That is also a scenario where you might use resting, which is like cloud agnostic or provider. Agnostic was file system level, backup of your volumes.

C

Thanks um yeah, so this time we're going to create a backup, um but we're going to use that at d fault.

C

Volumes to rustic.

C

Okay, so that backup has been created, we can get backups, and now we can see that this backup is in progress.

C

Let's see if we can view the logs for it. Oh, I didn't. I didn't filter on the namespace, that's okay! um So this time, I've just back to blur to back up um everything in the cluster. Essentially, so that's why uh there are so many more items this time, but again I'm going to um so. First of all, actually before we do the.

C

Restore in our um in our object storage, we know we have a rustic directory um and within that this is where all the the rustic um snapshot, information or sorry, all the rustic um backup information goes to so um so all of the the layout of this is all very rustic and specific.

C

So we don't need to go too much into that, but just know that that day has been backed up, and that is where it is. So, let's go back here, so I'm going to delete the namespace again.

A

So while we wait for like the name space to be deleted, I added another link to the to our notes on which which has information about like opt-in and opt-out behavior for rustic um by default. How you can enable the stick for part wall um to backup pod volumes without it without having to annotate each um pod? uh And then you can also do the opposite. Saying you want to say you.

A

You can also say enable restick by default, except for these volumes, which you can supply using an annotation, so uh that is also possible and while, while rustic is enabled by default, uh there are three kinds of volumes that we don't uh back up.

A

They are like the backups or the volumes mounting like the default service, account tokens uh or like the host path volumes um or like secrets uh volumes mounting secrets. Because secrets are a different uh community subjects than objects themselves, secrets and config maps. They are going to be backed up by regular valero backup.

A

We don't want to uh back up volumes that are mounting these secrets, because there may be scenarios where the secrets may change from one cluster to another. If you are doing cluster migration.

C

Awesome thanks to the extra info.

C

Okay, um so the namespace and the associated resources have now been deleted, um but now I'm going to perform um the restore, so I'm going to do restore create.

C

C

C

I've just created another backup from the sorry, just create another store from the backup that we just created so.

C

uh Yeah so our.

C

So when we perform a rustic restore um how we do that is valero, when restoring the pod object will um run an init container, which runs rustic to extract the data from your rustic store and um write the data from from from rustic to uh the volume where it was taken from so in this case. This is why we now have um an ending container running. This is the the rustic container. So I do I describe this.

C

Pod, uh you see that we have this uh rustic container, that's running, so it's just running a restore helper and from the blur team, which runs rustic and restores the volume. I'm sorry resources, data to the associated volume.

C

C

Yep that pod is now running, I'm just going to exact into that again. Hopefully, if I've done this correctly, the the file that we've written is there. So, let's.

C

um Next, there we go so that was the pause been restored, but then also the the data that we wrote um to the volume that was attached to that pod um has also been persisted using rustic and has been restored.

C

Okay, so that was uh my demo of a basic uh backup and restore using valero. um Is there anything else that you think that we should add at this stage or anything else? That's imp, uh interesting for any folks that are watching this to know about.

A

No, this was like uh awesome demo, uh it was. It was more detailed than I uh had in mind of uh preparing for so this. This was. This was good.

C

Excellent thank.

C

You, okay, so um so I know that we also had in the uh office our stock um to potentially chat about contributing to valero do should you think we should do that now or do you want to hold off and do that um during another session? What do you think.

A

I am okay doing this, I'm going to let no sorry jonas uh have have an input as well. uh Maybe this can be like an ongoing section in each of our office hours.

C

I'm just gonna stop sharing my screen guy.

C

Yeah feel free to.

A

Add in demos and discussions and everything to the office hours.

A

Absolutely okay: I think we can uh close off today's office hours with the information about uh how to get started, uh contributing to valero and how to reach us for help getting started contributing to valero.

A

So, as let me start sharing my screen again,.

A

So yeah, I'm not looking uh going back to our documents website. We have these links that will help you start contributing, which has like things that you need. You might want to familiarize yourself to get around or navigating the valero code base and and how to navigate issues that have been opened.

A

We also have instructions on how to like build valero locally. We will also cover this in another office hours uh episode sometime uh either next one or the following one: how to build palero locally and how to run that and like locally without using any cloud provider like the setup that bridget was using today, uh we will do a step-by-step walkthrough of how to get that going for yourself on your laptop uh and then we also have some coding standards.

A

Basically around opening prs like requiring you to sign off, commits and adding a change log just so that you can get credits for the features that you contribute or any changes that you contribute and stuff.

A

So this is a good resource to make use of the other thing that we would really welcome. uh I know samira, you said you are going to help us with some documentation, so we have like a style guide for documentation uh which seems pretty long, but it's a pretty quick read. uh So I would strongly suggest familiarizing yourself with some of these may not be all just to help help get help the review process of the prs go through faster and anything else that we want to add.

A

The other thing that I wanted to mention is like how to recharge you can you can contact us either via slack? uh We, our slack channel, is uh valero in the upstream kubernetes slack, and this is our team.

A

These are all the engineers and tim is the manager on the team for who kind of facilitates stuff for us. So this is a good place to stop. I suppose uh you can so a couple more things on how to reach us. You can reach us via slack on the velaro channel in in upstream kubernetes slack. We also have a google groups that you can join and then you can also reach us via twitter and our handle is here.

A

This is our twitter account and you can follow us for updates with respect to valero. You can follow us for announcements that we want we make so we also uh also post job openings that we have on our team, like this one so uh feel free to follow us or reach us in. If you need help uh with that, I think we can close off today's office hours unless anybody else has other questions that you may have thought of. While you were watching the.

B

A

And on that note, I think we can stop the recording and will do.