VMware Velero Office Hours, 24 Sep 2020

Previous Meeting

⏯

youtube image

►

From YouTube: Velero Office Hours - Sept 24, 2020

Description

The very first ever Velero Office Hours!

---

Sept 24
What is Velero

A tool to safely backup, recover and migrate applications running in kubernetes along with their persistent volumes.
This is a solution purpose built for Kubernetes using Kubernetes Constructs.
How does Velero run backups and restores? What happens under the covers?

There are two components to Velero:
CLI client
Server running as a pod in kubernetes
The client and the server components don’t interact with each other over network connection, but use the Kubernetes Custom Resource Definiton construct to communicate.
The client creates the CRD API objects which the server pod which runns a reconciliation loop for these CRD types watches and reconciles the objects. For example, if a Backup object is created, the server, in the processss of reconciliation will perform the backup operation.
Some of the other CRDs are: Backup, Restore, Schedules, BackupStorageLocation, etc…
Refer to our API types docs for more information
Reference for more details: How Velero Works?
What’s the difference between a PV backup and a restic backup?

Backup of applications using persistent volumes will include volume backups as well.
Velero can perform volume snapshotting operations native to the volume provider using VolumeSnapshotter plugins specific to the volume provider. For cases where, a Velero VolumeSnapshotter plugin doesn’t exist for a volume provider or if the volume provider doesn’t support snapshotting operations Velero’s Restic integration can be used.
Restic Integration:
Restic is a 3rd party tool that Velero integrates with to perform filesystem level backups of persistent volumes.
Velero handles installation and setup of restic as part of Velero installation.
Refer to our restic integration docs for more information.
References and resources on some of the topics that were covered:

Velero website
Velero API typpes
Backup Hooks for quiescing and unquiescing applications before and after backup.
Restore Hooks to customize application restores to perform any necessary setup before the restored application pods can run.

A

On so hello, everyone welcome to our first valero office hours. uh This is a place for folks in the community to join. Ask us questions, uh questions relating to how to use valero or how to contribute to valero or any questions related to valero. That you would like to ask this meeting is going to be.

A

uh This meeting is going to occur on every second and fourth thursday of the month uh and uh excited to get this started. So I see one uh person from the community joining jin chang, so welcome, and if uh do you have any questions that you would like to start us off with.

B

uh Actually, uh not, I saw you post on some kind of general topic uh under today's topic. I think that's I'm still new to this uh community, so uh but I'm part of the data domain. Actually, the backup restore is not a surprise subject to me, but I think uh I definitely have some interest in this, uh but I know how it play around.

B

I think this was used in some of our kind of software products right, but I think my background is mainly coming from data protection appliance perspective, um so I want to see how this technology can be used in an all-in-one solution from appliance perspective.

B

This definitely brings an interesting idea, um so I I want to just kind of join this community learn, learn from your guys and hear more about it. Thanks.

A

That's awesome, uh welcome and we'll be here to help out in any way we can to help you get started and stuff. So with that, let's get started with like the topics that we have lined up today, uh considering that this is our first office hours, we wanted to start off with, uh from the beginning from the scratch answering basic questions around what valero is, how it works and uh all the various capabilities that it has just just like scratching the surface, but not diving too deep today.

A

uh So, let's start with, like the first topic that we have on our agenda like uh what is what is valero um valero is like a uh or weller as a tool to like back up and restore applications running in kubernetes uh by applications. I don't just mean, like your app definitions, like your manifests, but also your persistent storage used by your applications, so this is this can be used for disaster recovery of your application or to migrate your application across clusters.

A

A

And then, like now, coming to ferrero, does backup and restore.

A

Valero has like two components: one is the cli and the other is the server. The cli is what runs on like a cluster admins laptop and the server is like a pod running in your kubernetes cluster.

A

uh The the two, the the client and the server don't communicate with each other directly uh over like the network or something so this is uh so valero is like purpose built for communities using kubernetes constructs, so we make use of the kubernetes constructs of crds and the the client and the server communicate with each other through crds.

A

So if the client creates a crd, then the server notices that and reconciles that crd. So this is like a controller uh reconciliation loop that valero runs.

A

So there are like various different kinds of crds that valero uses uh like the backups backups are a crd. The restores are a crd. You can also create schedules which is like running backups on a schedule.

A

Then there is the this thing called as a backup storage location, which is like a object store with where the backups uh taken by valero is persisted.

A

So that's how valero operates under the covers like when, when uh when, when the cluster admin wishes to back up a particular application, our backup resource is created in the cluster and the belero server part which is running in the cluster notices that backup object being created and performs the necessary operations to backup that application- and there is uh like I mentioned before- bellero not only backs up pr uh application definitions, but also backs up the persistent storage that is used by your applications uh through volume snapshotting, uh based on where, uh where your kubernetes cluster is running, uh you may be using like one of the cloud providers.

A

First, as your volume provider or you can you or you may be using some other storage provider, so valero can leverage or can perform volume, snapshotting native to your storage provider, using a volume snapshot of plugins, which is like a custom binary for built, specifically for that storage provider, which understands that storage provider api apis to perform volume snapshotting operations.

A

uh If you are running in a in an environment where the volume provider either does not support volume, snapshotting operations or plug in a valero plug-in to perform volume. Snapshotting operations in that environment, as uh if that plug-in, does not exist, you can you we have. We also have another integration with a tool called restic, which is like a file system level, backup of a volume uh that you can use um to backup your applications.

A

uh With that, I'm going to stop and let you all ask me, ask questions based on whatever I have spoken on spoken till now,.

B

Yeah, I I mean how about some naive questions. Sorry, if I can ask some obvious questions: oh this is.

A

B

A

For that, so there is there's.

B

No knife questions you just mentioned like uh you, you cannot back up the manifest of the uh combination cluster and you mentioned varela itself is running as a part, so I I would assume that it has its own manifest within the kubernetes as well. So when you do a whole backup iu, also backup the manufactures that belong to the valero itself or those are being kind of unpurposely being kind of excluded, is that some policy driven by the user.

A

So, uh that's all that's a good question, um as as a policy in fellero, we back up more and we uh that way the users can have exercise control over what they can restore. So we back up as much as possible uh during backup and users can uh selectively restore whatever whatever applications that they want to restore.

B

Okay, so, which means user can kind of selectively? Not uh selecting will error itself as a backup.

A

uh Yes, so the user can ignore valero. This is this is uh totally uh in the hands of the user. Under under their control, they can use label selectors or like uh backup, only specific name, spaces, uh etc.

A

We uh in in subsequent office hours, we will be doing demos uh where this will become more clear, but uh the user has complete control over what they can back up and what they can restore uh valero offers, where the various kinds of resource filtering that users can use to select what they want to backup they can choose to backup the whole cluster. They can choose to back up only certain namespaces, etc.

B

Okay, so in the in the traditional account, we have this uh kind of asset right, the asset could be a vm or any of uh kind of file system snapshot. We we want to back up, and also we have a policy which define uh when and how frequent we want to do the backup or backup to wear so all of those probably leading towards the crd. You just mentioned right.

B

So basically, if I understand correctly, is the collection of the crd, whether it could be a backup, restore volume, location for the backup it, the combination or logic, combination of those leading towards a specific customer, um backup, asset and uh and associated policy along with it. So that's kind of the mapping towards the real world is that is that correct kind of.

A

Yes, yes, so the backup crd has all the resource, filtering capabilities or the fields for resource filtering uh and like in terms of when you say assets. Here, the assets are like applications running in kubernetes uh and uh the users have control over what they can back up and when in terms of policy, and when things can be backed up uh you, you can use a valid schedules or that will create backup objects on a schedule which will effectively mean you are backing up on a schedule.

A

Does that make sense.

B

Okay, I get it yeah and also right now the default. The backup object is, can only be the object store right, there's no kind of block storage or the file system can be used.

A

Yeah, so right now for object, storage, we support like any any of the cloud providers: storage like s3 or like the azure blob store or the google google storage.

A

uh We support any of those um and we also have a support if you're not running in an environment like uh in a cloud provider environment, we also have min io, which is like an s3 like storage uh that you can set up and valero can basically talk to any s3, like storage, to store and retrieve backups.

B

For the for the for the application level right, we always worry about like application consistency, so uh each of those like legacy database like like oracle sql server, they have its own hook, which allow you to acquire a scene. So so, when those legacy applications move into the kubernetes running environment are those? Do you expect those legacy? Applications will still maintain those kind of application pricing hook so that whatever can call upon when they try to do some backup.

B

Do you expect that model to be the similar as kind of legacy kind of application level, uh data protection.

A

So this the scenario that you mentioned is not not only applicable to legacy applications, but also any sort of uh store, uh stateful application right. You will want to have support quizzing and unquesting uh before and after a backup. So for that we have this extension point called as backup hooks, which are basically like uh a cust extension point where users can say before running.

A

The backup of this particular part run this partic this command in the container, so that the application so that the database stops writing and we can successfully snapshot the volume that the database is using and once the backup is done. We can also have this first hook, which will unquest the database and letting the application make progress and do meaningful work because we have finished backing up the application.

A

So we have uh the backup and uh backup hooks that you can use for queries and quers during backup.

A

uh We also newly introduced this concept of restore hooks which lets you perform like custom actions or extend or customize your the restore process allowing users to run custom commands or custom programs during or after an application has been restored.

A

Okay. So, for example, if you are backing up uh mysql that is running in your communities cluster, uh you can like I'm not very familiar with my sql uh backup scenario, but you can perform operations to make the application. Stop writing to my sequel.

B

A

And then, when you, when you are restoring you, can you need to do like a pg dump and then another extra step uh to get the to restore the mysql database? So you that's. That is where you can use restore hooks.

A

So you would restore the uh part and then make the part and give the part data and then let it continue.

B

A

B

Yeah all right, yeah thanks.

A

And as a segue, I would like to mention that we have uh what we feel is like a exhaustive documentation on our website. uh It is the little dot io.

A

This is our landing page for everything valero, where you can find all the latest blog posts and all that, and then we also have our documentation linked here.

A

This is, and this search feature in our web page is really cool, so you can uh make use of that- and this is where I, these are the hooks that I was talking about the backup hooks and then also the restore hooks.

A

And as coming to the preview or backing up a little bit to resource filtering, we also have a page on that which lets you choose like gives you information about how to apply resource filtering on a backup basis.

B

A

Yeah other questions.

A

I see somebody else joined uh tilshan, hi, darshan,.

A

Hi, would you would you like to introduce yourself.

C

Yeah I'm dilution and I'm from sri lanka, so you are able to be using own neo as our backup strategies. So I was interested in joining.

A

Awesome, do you have any questions that you would like to ask.

C

Yeah I was uh interested with these uh backup books, so it's a part of that also comes in here. So.

A

uh So do you want me to like give you like an overview of what backup hooks are and uh or do you have anything specific that you want me to answer.

C

No actually, I was listening to the previous stage, where you explained the uh backups and the rest of that. uh Basically.

B

C

Didn't apply that, but for the stateful application we do have some kind of scenarios that we need to consider that but yeah we can look in.

A

Okay, so um considering that there are, there is interest in backup books. Let me go a little bit deeper into backup hooks. uh There are like two backup hooks: one is free backup and a post backup. Three backup hooks are like uh is where you would perform questing of your application um in post. Backup hooks is where you would perform the unquestioning of your applications.

A

um We can. We can uh specify backup hooks as like annotations on pods. So, for example, if I have like a stateful application running- uh and I can annotate that part with the necessary hooks and velero when backing up that pod will notice these annotations and perform the hook, actions and based on you can also control what you want valero to do when there is an error when there is a timeout or any any such scenarios.

A

Similarly, for post hooks, you can also control uh you can specify the hooks via annotation or also you can make them or you can specify them in like the specs of the backup itself, so that let me show you what I mean by that. So when I'm looking at the backup, spec.

A

So this is I'm looking at the api types, and this is I'm looking at the backup api, and this is where we can specify hooks that have to that are to be run in the backup. So.

A

If the annotation or specifying the hooks in the annotation is very convenient, if your commands are like very straightforward and simple, but here, uh but if they are in case, you have very complex logic that are hard to specify in annotations. You can provide them in the backup spec as well.

A

So these are the uh pre uh hooks, and these are the post hooks which follow this, which follow similar.

A

A

And we have similar formats for the restore hooks as well.

A

So this is again uh actually let me put up a little bit and talk a little talk about the new feature that we introduced in the 1.5 release of valero, like the restore hooks so considering we had backup hooks, but users were weren't able to use valero workflow end-to-end for restores.

A

um There was a missing extension point of a restore hook to allow users to run custom commands during or after a restore, so that kind of led to the requirement of supporting restore hooks into valero.

A

So these are like custom actions uh that can be run like for, while performing restore operations, uh we currently support um two kinds of restore hooks one. It one is a init container, store hooks uh or a store hook. What this hook does is it adds in it containers to pods before the application can start.

A

So, if you have a database application, a database, let's say- and you want to make sure that when you restore the database, the back from a backup the data in which the the data that was previously backed up is available before the database is up and running. You can specify this.

A

You can use the init container store books which ensures that the init container can do the necessary setup before the application can start. So the app the init container restore hook can copy data or, like unpack it, unpack data, uh so that the application can use it straight away once it starts the other kind of restore hooks that we support is like exec restore hook, which is similar to the which is basically running commands inside a pod uh during a restore or after a restore.

A

So we have. These hooks can also be specified either via annotation or via the restore restore spec. So these are the annotations that we have, that you can use to specify uh restore hooks and the spec equivalent is also available in the api types here under hooks. So if you scroll down, you can see hooks here and then you can either specify init container restore hooks or you can provide exact or you can specify exact, restore hooks.

B

How does sorry for a question?

B

How does this compare to the um the original uh init container belonged to the staple application before uh for take the take the elastic search as an example right, there's some init container, which may set up some java heap or whatever environmental uh things for the for the particular applications?

B

I would imagine those in new container will be part of the manifest which was back up before so this adding another layer to make sure before the actual kind of application. Specific things around before this can be run as a as a as a even earlier configuration if needed.

A

Yes, so init containers for a part is is like you can specify more than one in it container. So this is so. If you look at this example here we are adding two init containers into this part that we will restore right, so the init container, restore hooks will be like the first will be amongst the first set of unit containers that will run and then once those are run, the rest of the pod can resume, as is which will include the init containers that were already part of the part definition.

B

B

So do we have also this kind of immediate container hooks in there in the back up or also right.

A

uh No, so that's that's like a difference between the backup and the store. So when you're performing backup, which is the application, is already running and has made some progress and has stored some state that you're intending to backup, at which point we want to run quest and unquest of the application uh having a limit container. There would not be useful because the unit container runs when the part starts for the first time. But here the part has already started and has made some progress and stored. Some state.

B

Yeah, but my understanding would be, um those questions will be, will be io, centric right because you're gonna back up the the stateful things you, you worry more about the story side, so you want to make sure storage is consistent.

B

That's where you you do the choice, mainly yeah, I just just kind of top off my head- is that anything else need to be done, uh because the the container will be a bigger hammer rather than just just a hook, which kind of requires an io for the from the data perspective right, yeah, yeah, yeah,.

A

Yeah, but without these hooks, your backups will just be crash, consistent, which may not be enough uh for certain applications or for certain stateful applications, so that is where creasing and unquestioning comes into picture and uh other than questing and questing. uh Before and after a backup, uh I, I can't think of anything else that you will have to do to like successfully back up a stateful application, yeah true yeah.

A

A

B

Questions another question is the implementation of this kind of need to contain all the exact.

B

The hooks is pretty much the same as the it's pretty much the same as the kind of way you start the application, where you you add it add it uh added the init container as part of your yarn for the for the stateful application as well right, it's just it's just like the the the timing or the for the way it started is controlled by the uh by the way you know to trigger this as a first and then uh stable application will be the second be the later too right.

A

Yes, so for for init containers, it is pretty straightforward. We just update the uh manifest to add this init container that the user requested, uh but but for running, exec hooks. We we have. We need to wait for the part to be running uh in so that we can like exec commands into. uh We can exec into the container and run commands.

B

So if you say this exact hook need to be wait for the part to be started before it's running, so that means the the pause. The pause original init container has been running before right before your exact c is running here. Is that right.

A

Yeah, so uh you can either you can either use init, restore hooks or exec restore hooks, but if you can, if you use both, then the init container would have run. But if you also need a certain command to be run, then you can specify an exact uh restore hook which will to run the command inside a part. The part has to be running right, okay, so that is where we wait for the part to be running uh before attempting to run commands inside it. Yeah yeah.

B

Yeah thanks yeah sorry.

A

The questions you all may.

A

A

If there are no other questions, I will go a little deeper into the rustic integration that I mentioned uh the uh so I mentioned that I previously mentioned that you can back up volumes used by your stateful applications uh when you're backing them up and if you're running outside of our cloud provider, environment or if you are running in an environment where you don't have volume, snapshotting capabilities, we have this integration with this tool called restic, which is a third party tool for valero, which we integrate uh into whether to perform file system level backups of volumes.

A

uh What this enables is this is agnostic to any cloud provider and will work in any environment that you're running in and the other thing this ensures is like your volume snapshots or your volume. Backups are portable cross platform, which means you can backup volumes running in aws and restore them into azure using resting.

A

So that is a capability that rustic brings up like enables or facilitates, um if it, the installation uh of the restake and integration of the stick is all handled by valero. For you, all, you have to do as a user is tell valuero that you want to use resting.

A

I don't want to go too deep into how rustic operates and how how how valero uses this uh this operation uh uh straight uh in today's episode. But uh but this is definitely something we can dig deeper into in subsequent episodes of the office hours.

B

So this rustic use case is: is that assume did that work on top of the like uh object store, or is that only.

A

B

A

The volume so the stick is just uh is uh rustic is basically an alternative to backup your persistent volumes.

A

If you are not running in, let's say you're running in, like your own volume provider, environment, which does not support, snapshot apis or does not have a valero plug-in uh to perform volume snapshotting, you can use rustic to perform file level backups of your volume itself, and these backups are stored in the backup, storage location.

B

Okay, so, for example, if I use the local volume which happened happen to have does not have the snapshot capability like local file system right, this can be used to back up those into uh into another persistent world which could have uh different function or provided by different vendors.

A

Yes, so this this is basically performing like uh like capturing the contents of your volume and storing it in the in the object store.

A

Okay, I think that's a good segue to look at like the format of our backup that is stored in the object store, uh so we store like uh some metadata and then als along with the uh tarball containing the backup.

A

The metadata is basically uh information for valero to uh on, to use while restoring and also to show as uh fields of what show information of what is in the backup.

A

uh The backup is basically like the start.gz uh file that is stored in the object, storage location. uh We we have recently started in. We recently introduced, I think two releases ago, uh versioning or like versioning of file formats.

A

This will allow us uh to kind of make progress without breaking uh or to make changes without making without breaking uh older installations of bolero.

B

So basically, this is metadata of the.

A

Yes, there is metadata and then there is also the manifests for uh the applications like the yaml files.

A

We store them as json, uh and then there is also like information about uh volume snapshots that were part of the backup and stuff. And if you, if you were using restic, then the file system backup would also be stored in the in the object, storage location. Alongside the.

B

B

So basically, it's a it's a it's. A json file, along with the tar right in the top, cannot compose of one valid backup.

B

Okay and the end of the yarn, for the namespace inside of the command x is. Is that, like the plain text of this json file or it's part of the gz.

A

uh Okay, can you ask that again sorry.

B

uh I think I think my question is for the for the manifest of the of the of the, um I think, the manifest of the name space or whatever uh stateful inside of it. That's the young file for those stateful is part of the tz right.

A

Yes, yes, so that's all! That's all uh packaged into our gz file.

B

Okay and the json file, the vernandle dash uh backup.json, does does not reference to those young file.

A

uh Yeah, so this is the uh the dot valido-backup.js file is the meta. Is the file containing the metadata where we have the spec to indicate what was backed up, uh what was excluded and then also if there are any volume backups, we list them here so that when you say valero, backup, get details or valid backup describe details.

A

You will find all this information.

A

Without actually having to unpack the backup and inspect it manually,.

B

Basically, it's a it's, basically, the the five, the the kind of the the name of the backup right you know in the legacy world is like this is my last week backup for this applications.

B

It's just some kind of metadata related to that particular background when it was when it was done how it was done, uh timestamp of those right, yeah.

A

I just noticed that we did not put in or write in answers into the hack md to the questions that we answered. So that is something that I might do it after. After wrapping up here.

C

I can also do that. You've put in all the effort to yeah to answer the questions I can write them.

A

A

B

Can you can you talk about a little bit kind of the ecosystem like besides valero? What are the kind of similar uh type of kind of project or product in the same kind of domain area? um Sure.

A

So backup and restore has been something. That's that's been. That's come up quite a bit uh over the past year or so, uh and for that uh the community's upstream community has started this working group called data protection working group, uh basically trying to standardize uh what standardize the kind of things that we've built into valero and we are- we are hooked in there as well.

A

We are trying to uh help people adopt uh the valero features or or get gather learnings from valero and make make it as part of an upstream or a fully upstream build solution that is still in uh kind of early stages at the moment.

A

So that is, that is something that is happening other than that there are other um paid, uh backup and recovery solutions. uh One that comes to mind is casting uh yeah, that I think they have two parts. uh I I I'm not. I don't remember those things very well, but that is like a paid subscription that you can use or you can purchase to backup and restore kubernetes applications um for belero.

A

uh Ours is uh fully open source. uh You can open up each open issues requesting for features reporting bugs. uh You can also open pull requests to contribute to valero.

A

Let me show you our all. Our code is on github.

A

So this is our main repository where we have issues and pull requests, and we also have this page, that is all the adopters of valero or all the all the various solutions that are using valero as part of their backup and recovery solution. So we have all these people who are using weldro in their products.

A

uh We also have an open governance model, which means that all the decisions that we take are in the open are discussed in the open, and the roadmap is also something that is part of the discussion that we make it a point to have discuss more in the open.

A

We also have this support process where you can ask questions uh either in the slack channel or or our community meetings or our latest office hours.

A

So this is this is our office hours, and this is our community meetings. You can also email questions to our google group.

A

So that is mostly that I don't know if I answered your question.

B

Yeah yeah, I think you point that out quite quite assignment information, yeah yeah.

A

A

I think I would all the other thing that I would like to mention is like the resources tab on our website, which has uh links to our community meetings playlist on youtube and also like the deep demos and deep types that we have done. We also have like various talks featured from kubecon kardicia who's.

A

One of our maintainers has done various interviews with uh like new stack and other other uh on and on a few other youtube shows uh uh and other podcasts that have featured whether or not that we feel are informative, and that is all those are all linked here and uh you can. You can check them out, and there are also a few blog posts that you would like if you would like to read like how to backup certain kinds of applications or like gotchas, packing up certain things, uh certain store in certain storage environments, etc.

A

So I would strongly encourage you to check this out as.

A

A

And just as a note, all these office hours uh will also be available on youtube too for folks to watch.

B

A

I think we are eight minutes to the top of the hour uh if there are no other questions. uh I think this is a good stopping point uh and if you have suggestions on what you would like to cover for the next office hours come with questions or tell us beforehand that way we can, we will be in a. uh We will be able to help you better uh instead of trying to investigate on hand.

A

So I think, with that, I'm going to stop my recording uh and before I do that. Thank you every thank you all for attending or watching this later. If you are watching this on youtube, be sure to check out our community page. If you want to attend uh our future office hours or our community meetings, you can also join the google group for updates about our releases and all announcements related to palero and also be sure to follow us on twitter, which is where we will post again new releases, exciting announcements, etc.

A

A

With that, I will stop this recording.

A