Kubernetes Data Protection Working Group, 27 Jan 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes Data Protection WG Bi-weekly Meeting 20210127

Description

Kubernetes Data Protection WG Meeting - 27 January 2021

A

All righty, you guys, can see my screen right.

B

A

Nice uh today is january, 27 wednesday. This is the data protection, kubernetes data protection working group meeting. uh Today we have uh a couple of items in our agenda. I majorly focus on one presentation from velora given by nolan and dave.

A

The first agenda is that our white paper is in progress. I think many of you are either have or either been assigned an item to work on. uh If any one of you are interested in contributing to this white paper, please uh feel free to join.

A

uh This is a work in progress. The second agenda is that remember in the beginning of this working group, we started to discuss. How do we understand uh store backup vendors?

A

uh How did they do backup in the kubernetes context, and we plan to do a series of presentations from different backup vendors?

A

I think we did one for casting that was given and xin, and I discussed about this and I think we figured out this is a good time as the multiple efforts come in concurrently in this community uh to see whether those uh designs we are making a progress, we're making a fit into the picture of existing for existing backup vendors. Today uh we are glad that lorna and dave is going to give an introduction to veloro.

A

This is a open source tool which can backup kubernetes clusters, and if anyone is interested to do this kind of practice, please reach out to me or shin on that manner and then, lastly, we will do some open issues if we have time uh and then with that I will hand uh the presentation to norton, I think david has not joined yet no names or.

C

Yours, thank you very much share my screen here.

C

All right is that visible.

A

Yes, yeah, I can see it.

C

Cool all right, so uh my name is nolan brubacher. I am uh tech lead for the open source components of valero um dave. uh Smithy cheetah is also here at vmware and leading up a lot of the um things we're building on top of it.

C

So I'm going to just hop in and talk about what valero is. So uh we can start by thinking about what we need to protect within kubernetes. Folks on this call probably familiar, but just as a recap, you can protect the uh the yaml for your stateless apps, which could be done in git ops.

C

um You also want to protect any sort of state stored in a cloud service, uh so things like um amazon's, rds or any any sort of database. It's a cloud service.

C

uh You also want to store the state that or back up the state, that's in your kubernetes persistent volumes, because those are not actually in the api. The kubernetes api server and those are critical to getting your your work back up and running. Then. Finally, there can be some state stored in the kubernetes api server. Things like a persistent volume aren't necessarily stored inside of your git ops, repo. So you've got some resources inside the the kubernetes api server that are actually representatives representative of state.

C

So what valero is is an open source tool to protect these kinds of storage. um It has two components. First, one is: it backs up the kubernetes resources to object storage, so this could be s3 compatible. Google cloud azure. We have a plugable interface for that, but those are the big three api types that we support for object, storage, the other component is snapshotting or backing up persistent volumes.

C

The volumes themselves on your provider, so avalero backup is composed of backup of the kubernetes resources that lives in object, storage and then snapshots or backups of persistent volumes which may or may not be shipped up to object storage.

C

We have support for running kubernetes anywhere and that could be on vsphere. It could be on aws google cloud azure. uh We do not use direct xcd access, so that also includes managed offerings here in vmware. It is part of the tanzu umbrella tonsil umbrella, and we have integrations with kanzu mission control, dell, emc power protect and red hat's openshift.

C

I'm sure there's others that we're not fully aware of, but we've got um like a presentation from caverno yesterday on integration with our policy engine.

C

So, like I said, we can back up all of the clusters kubernetes resources, uh but we can also do selection on this, so we can backup uh based on namespaces or select backups based on label selectors.

C

We also have support for getting the persistent volume data via rustic for style file, system level, snapshots or doing snapshots at the storage system level.

C

We offer support for execution hooks for quiescent applications or other operations. So if you've got a database that you need to pause or file system that you need to pause, something like that. Valero has hooks to do that. We can operate our backups on a schedule.

C

You can interact with it via our command line or by submitting crds, which is essentially what our command line does and you can restore backups into a new cluster migrate to a new namespace or you can restore into an existing cluster. Say you deleted something by accident. You can bring that back in to the existing cluster.

C

I'm not going to go too in-depth on this picture, but this is a kind of a a rough outline of how valero works. You have the valero cli or cube ctl or your own api that submits crds all right, custom resources.

C

Those get submitted um here on and uh then are processed by the valero server, which will invoke plug-ins to manage the different object, storage, um persistent volume, storage, and we also have plugins that act on specific kubernetes kinds or api groups.

C

Through csi, we can also create backups and, with our rustic support, we use constructs called volume backups, where we backup pod persistent volumes via rustic and with rustic. We send that data up to the object, store.

C

So inside of valero, we capture the cube resources and we put them into a tar file that we store in object store. So um we can modify objects on their way out with a backup item action plug-in um often we don't and we save the modification for them on restore. But sometimes this is useful.

C

During this process, we also make snapshots via plugins, and we store a reference to that snapshot id in our object storage. So we have a list of all the snapshots in that backup and finally, during this situation, uh or during this process, we have the execution hooks called and process any data that they need to.

C

If someone is using restic their file system data is also transmitted to the object, storage and that information is recorded in the tar file and we also have some additional backup metadata stored in the object store. So the back of it was associated with logs.

C

Shorthand for everything that was created in it like a manifest, so that is that side on restore. We fetch this backup info from the object, storage and we restore the kubernetes objects from the tar file. We have a customizable um priority order for restoring, so this is our default that we have right now.

C

We basically process all the resources in this order, one at a time, and then we get to everything else at the end, in alphabetical order, and we submit things in these this order in order to make sure we can kind of stitch things back together on restore.

C

So I mentioned we have rustic. um This is kind of like an escape hatch. If your storage system either doesn't have a valero volume, snapshot plug-in or uh doesn't support, csi snapshots through the kubernetes api restic is storage system agnostic. It works at the file system level, uh but there is not a snapshot and the application either has to be quiesced or frozen for consistency, and uh it does take longer because it's on the file system running in essentially the same node as the work.

C

The workpod volume snapshot plugins are very similar to what csi snapshot snapshot. Drivers are now these existed before csi was kind of standardized, but they will interface directly with the storage system to take snapshots and some of those plugins will ship those snapshots off of the node, such as the vsphere plug-in.

C

And like it says, these are usually block oriented, so they don't have to operate at the file system level.

C

We can support cross-cloud migration with restic, since that's at the file system level we're not dependent on a particular block api or having to do any kind of format translations. So we can send all the data into object, storage and then, as long as both clouds can reach the same object store bucket.

C

Then a target cluster can pull down the rustic data and restore so you can use this between to move between two clouds.

C

So for the future, we're looking to work with additional backup partners and kind of bring people in so that we have broader understanding of what people want from us and also to make components of valero more systematized, so that you don't have to install all of valero to get some of your functionality.

C

We want to present an engine to users that they can plug into we're, also interested in doing application integration so that valero can snapshot applications right now. Valero works very much at the namespace and label level and doesn't have a whole lot of intelligence around what an application is things like helm, charts definitions, the app crd.

C

This kind of stuff would be really useful to read and build a graph of the objects that are being backed up and with that I'm going to open it up for questions.

C

A

Thanks nona uh any questions.

D

Yeah this is this is rob esker um a couple, quick questions, um so I noticed you know just kind of looking through and I've watched valero. I guess going all the way back to its arc days um it it. um I don't see a lot about sort of governance.

D

Obviously it's under apache 2 and you know I see that you guys use contributor coverage, for um you know, guidance on a contributor sort of um conduct, but I, but I I'm curious if you could speak to governance, and why is this not a? um You know what what is your view with this vis-a-vis like, for example, cnc, cf sandbox, and I guess, depending upon how you sort of um respond to that. I had some questions. What you about back further back up partner, integrations and what that would look like.

C

Yeah so you're right, it's not been uh moved into the cncf um part of that is we're trying to be as open as we can, while controlling it. For for full transparency, um we do want to get good integration with vsphere.

C

That being said, we're not trying to lock out support for any other projects right now. We don't really have any plans for donating to cncf sandbox.

C

However, we do have this governance file and we'd certainly be uh open to having those discussions.

E

Yeah and I think we're really interested in having partners work with us use valero contribute to valero. We want to be really open about. um You know how we can merge, you know, work together and integrate and the vsphere is a a factor, but it's not the factor, and we pretty much figured out how to uh you know it's a it's a clear goal: that bolero is not going to be a visual product and it's not going to be a vmware only product, but it is integrated with a bunch of vmware products.

D

Yeah, thank you. I'll have a look at the at this.

F

I work with these folks. It's also integrated um to be clear with some of our competitors products um and we uh we embrace uh the opportunity for a larger community um with with everyone, so.

C

Yeah we've worked with with red hat uh on improvements to valero that feed into openshift solutions. uh We have done some partnering with with microsoft in the past. So and this is it's not um meant to try to lock people out but um yeah. It is still not on the timeline to look into cncf, but we definitely welcome further contributors and input.

E

Yeah, I think one of the other things too is we've. Had people embedding it in different ways, so, like openshift, for example, uses it as the backup component, so it has the schedule and all that stuff is, is used there. Power protect is embedding it more as an internal component more as infrastructure, so we're looking at breaking out these layers so that we actually break out an infrastructure layer separate from the product layer, and then we can really talk about how that infrastructure layer moves into things like cncf.

A

I got a couple questions. The the first one uh nolan you mentioned. The execution hook is not a global level thing because you don't have an application concept. At this moment, I guess.

C

So that is something we tied to the workload pod. So we we, we have a variety of different methods of inserting it to a workload pod and then we essentially call scripts that work that live in the workload pod. So um it requires doing some annotations on your workload, pods or trying to remember exactly how we execute at the top level, but we don't have anything that is like a backup wide execution hook. So it's on a per workload pod basis.

C

For example, if you have a mysql pod you're going to want to run the hook in that pod.

A

Sounds good. Another quick question will be the csi plugin right now. The availa calls into csi snapshot aps directly, not interact with the worm. Snapshot controllers right, not creating one snapshot resources. Is it honest.

C

Oh yeah, so the the csi plugin actually does use those volume snapshots here crs so it'll make the volume snapshot and then basically include that in the valero backup the volume snapdraft plug-ins, the these kind of the lack of a better word legacy ones that we started with those call directly into the apis. So the aws volume snapshot plugin calls ebs apis.

C

But if you were to run aws with the csi plugin, then it would instead submit a volume. Snapshot. Cr wait for that to be ready and insert that into the backup.

A

Okay, so by default, which one is we uh is the is the current status.

C

Right now, uh the status quo is using our our volume snapshot plug-in api. We would like to move to the csi um model, but it's not as widely supported yet over time. I think we're going to see our volume snapdragon plugins get deprecated in favor of csi we'd really like to go that way, but right now, partly for getting the largest number of kubernetes versions, we can as a backup product we're not doing that um and there's more work to be done on our side to make sure the csi plugin works. Well, it's correct.

G

There's a there's no default right. I think this question is: is there a default.

B

Plugin, you have to specify right, there's.

C

No defaults included. It really depends on where you're running, but um we in terms of what I would call a default is our volume, snapchat or plug-in interface. But when you're installing valero, you do have to install the plug-ins for your cloud. We don't install anything by default right now.

C

Ideally, we get the csi plug-in, moved either moved into valeria or we reorient things so that we're just using csi.

A

Got it and last question sorry, I have a lot all related to the con. You know. Ongoing efforts in this community will be rustic uh when you do a rustic backup. First of all, um if I understand you correctly right now, the warning works for file systems right, that's one and they scan the files.

A

Prior to that you mentioned, there is a part running locally on that node, where the volume is attached to right yep. How does that work with the uh does? Does it require the one to be attached amount to your part?

A

If it is not a single rider, then uh then you need to an amount from the workload part and amount to your part. Is that how it works?.

C

Yeah, so the rustic support goes through the volume directory on the node. We install it as a daemon set and we look at what volumes are mounted to the pod and then use that data path to get at the volume um so you're asking about rewrite many volumes right yeah.

C

That support um seems to be hit and miss on depending on implementation. So we can back up certain nfs implementations by mounting through this data path, but there are some we've seen that we do not get all the time uh that there can be errors.

C

So we don't unmount the pop. We do not. We never unmount a volume. It's probably a shorter way to answer your question. um Longer term we'd like to see the rustic integration be be pluggable as well, so you could swap in. I know, um there's folks using copia for this kind of thing or um there's another project called borg, not the google board, but longer term we'd like to make the the file system level integration, plugable and then take then use it in conjunction with snapshots.

C

So you take a snapshot and do the file system level, backup of the snapshot. So you're not having to mess with freezing stuff and all of that.

A

Right, the interesting thing is that for local ssds at this moment it doesn't support snapshot at all, because it's very yeah, it's just literally a locally attached ssds to that specific node.

C

Is that for gcp.

A

Not only for gcp, but also for many many other solutions as well in kubernetes in general, gotcha.

E

Well, um that's an area where we could look at, for example, getting some file system snapshotting in there like if you're running zfs as the file system on the node, then you can take a zfs snapshot of the local storage.

A

Right, that's yeah. It depends on the infrastructure you're running up on yes, yep.

A

Yeah and uh and the the the rustic backup is the change based right, because you only search for changes.

C

Yes, yes, uh it'll it'll be incremental at the file system level, but it still has to do scans and it still has to do that. Diff and all of that, so it is, it is slower um and our snapshot support is not directly incremental. We we are able to use like azure's api, to request incremental snapshots, but that's not consistent across all the providers.

A

Got it so how? How does this do you guys have planned to extend velora to have management plan, because how does the end user uses? What are our backups for restoration and how do they manage the deletion of this? Because this, under the hood is all managed it? It seems to be the user. We need to understand quite a bit at this moment, but I'm curious whether you have plans on that.

C

Yeah so we'd like to so one one issue we have right now is because we back up through the kubernetes api server. Even if we back up at a name space level, we still, the expectation is that we get persistent volumes which are cluster level resources.

C

So uh you can't back those up or restore them if you're, not a cluster admin, um so like user self-serve is not really there yet and for backup deletion when, when a backup either hits its expiry date or a user, a user issues, valero backup, delete, valeria will go in and the volume snapshot plugins have a deletion api.

C

With our csi snapshots, we modify some things to go from like retain to delete um and then once that stuff's cleared up, we delete the stuff in object, storage,.

H

A

That's what I have. I have a lot to think yeah, no problem.

A

Anyone from the community has any questions.

G

Oh, I have one good question. I think uh just move uh to slide seven, the one above so this page about the resource sequence, but the priority order. So I thought crds are not restored by laura, so this user has to deploy the crds uh lara resort crs, but not crds right is that right.

C

So valero will install its own crds on install so like valero when you install it brings crds, but valero does grab the cluster level crds. uh We have logic that says. If I see a custom resource, I will go grab the definition and put it in the tarball.

G

Okay, okay, is that, but is that by default it will not be restored or something I thought there was something there saying you will have to include it for the crds to be deployed.

C

Yeah, so we had some bugs in um one three and one four versions where they didn't. They didn't um execute in this order reliably. So something that would happen would be that your customer resource definition would be submitted to the api server, but by the time the restore hit the custom resource.

C

The api server wasn't ready to make new crs for that crd. So we had a race condition that should now be fixed that we properly processed the discovery api. So there were some bugs around this previously, but by default we should grab the crds and on restore put the crd in first and then process. The crs.

G

I

I have a question about the use case, so here we talked about backup and also we mentioned using zfs, for example, to snapshot.

I

um You know local ssd type volumes, but you know those dfs synopsis are local to the machine where these ssds are they don't get pushed to any object store. So my question is really about what use cases is are trying to cover as far as backup application, migration migration from one cloud to another, for example, like looking at this chart here, like we have storage classes, the car we look, I assume, like storage classes, would change from one hyperscaler to another or application migration in scope for valero.

I

Like what use cases are you trying to cover.

C

That's a great question so originally valero was largely based around these hyperscaler snapshot constructs and those are often available throughout the whole hyperscaler infrastructure. You might have to do some manual migration of the snapshot, but to use amazon as an example, ebs snapshots are stored in their own s3 infrastructure.

C

So uh right now, valero is a lot of valero's. Common use. Cases are within within the same availability zone or similar construct and moving from one cluster to another. In that availability zone. We're doing disaster recovery in that same one.

C

We do back up storage classes and with rustic and our restore plugins we can uh modify the storage class on restore to a new cloud, but admittedly that is a kind of a not as common of a case.

C

We'd also like to see uh get to it, get to some point where we can um duplicate our snapshots across az or across clouds. um But it's not there. Today.

E

Yeah, so in terms of directions, um so the the current. So as nolan said, the original model was uh like aws, so you take an ebs snapshot. That's you're good right that goes into s3 and there's all kinds of magic behind it to keep it durable. For you, um however, you're still you're stuck at aws, so being able to extract the data from your evs snapshots and maybe just ship it to another cloud.

E

Just for durability is uh something that's there, but once you get the dirt, once you get data extraction, you pretty much have migration.

E

um So that's on the roadmap and uh there's stuff that's coming out of what we did in the vsphere plug-in, because the vstr uh storage is a on-prem local snapshot setup, and so we built in stuff there to move data into s3, we're looking to move that up into valero in the longer term architecture.

E

Then there is the issue of like filtering and modifying resources. We do have the ability already to you know, put plugins in here to filter and change things. um There's the cross cloud aspect of it there's also even just like kubernetes versioning upgrades. So you have an archive, you know you have like a. I don't know what we're at like 119 and you're on 1.25 and do we need we do a restore?

E

Do we need to modify things to fit in there and so we're working on filtering mechanisms in there as well filtering and modification mechanisms.

I

Thank you is this filtering account as a part of the open source solution, or is it like proprietary.

E

It will be open source. All the stuff we're doing here in valero is going to be open source. Okay, thank you.

G

I think uh some time ago is that raphael who gave a presentation on that right. So, if you have a your source and target, have different kubernetes versions and how to uh how to? How do you do that? How to support that right? Is that.

E

Yeah, did you do that you did that here, yeah and so he's working with valero to build that into valero, okay yeah. In fact, there's even a pr outstanding at this point, yeah.

C

There's a pr for for handling it. On the restore side uh we we have the foundation for in the backup. We have the data we need, um but we've got another team not directly on the gloria team, but another team within vmware that is contributing to the open source effort to do that. Negotiation on restore.

C

I see a question in the chat from stephen manley about how we, how we approach or how we want to approach app support question is: is the priority helm, integration or a push for more kate's, app definition or something else um we do. We have issues um open for doing helm, integration and doing doing um helm as an entry point.

C

um Admittedly, that has been pushed down the the priority map our priority list to get some other building blocks done. um I don't think we've got this fully decided yet I I think helm is great for meeting people where they are today. um I personally would love to see uh more of a kate's app definition, um but dave- and I have also talked about using stuff like istio definitions or or junior service mesh definitions.

C

um So I don't think we have a clear answer on that. Yet um do you have more input there.

E

I have an architecture in mind, so the architecture that we're working with is uh in the asteroid project and that is to start having uh what we call protected entities which could be pretty much anything so the idea being there that we start to expose snapshot and disaster recovery apis for different types of objects.

E

So one way you might do this, uh I have a prototype that I'll demo in a few weeks is uh say you have like the you have like an operator so, for example, like the postgres operator, um ideally in my in my world, the postgres operator should expose an api that says, snapshot your database or snapshot. Your object really, and you know valero or another backup app comes along, says: hey.

E

Would you snapshot yourself and the operator goes out and does whatever it does to snapshot that thing and then comes back and says yeah I got a snapchat, here's a snapshot, I do and then later on we can come in. We can say hey, you know we need the data from you. You know. Would you give me the data in your format and we want to extract that and copy it somewhere? So that's kind of the big model, and then you can see things like.

E

For example, we can build a protected entity that sits on top of say, a helm truck and we can say well. uh This application doesn't have, for example, an operator for it, but it's defined by a helm chart. So we're going to have the helm chart protected entity manager expose out a snapshot api for helm, chart x and inside that you know the helm chart protected man protected entity might go and say: okay, here's a helm chart.

E

That means that you know I'm gonna walk across it, pull these things together, snapchat these pvs and I'm done or the same might be true for, like the kubernetes app definition, what we're trying to stay away from is building in one definition of application into valero, because what we're seeing here right now so, like you've, got helm, chart definition. You've got uh kubernetes app definition: okay, well, you've got two already and you're going to come up with n.

E

You know so we're not going to push everybody into defining their applications in terms of the kubernetes app definition. I don't think I don't think that's going to happen and we also need to be able to handle things that are external to kubernetes.

E

So that's kind of a long-winded way to say yes and you know kind of externalize it and uh start looking at things as objects is my view.

J

But dave that that's great uh great explanation, I have a quick question regarding what was the exact name of the project where you're thinking about this you, you mentioned something which I missed. uh This is astrolabe.

E

Let me put in the uh the chat, um the documentation. I need to get back to the documentation and fix it up, uh but it's currently it's sitting there in the github repository next to valero and we actually used it. We did a lot of the the vsphere plug-in is built on a lot of this, so we got some miles on the apis and the concepts.

J

Yeah, so so this is good, I mean um so what you, what I think, what you're trying to say that the kubernetes app definition is not.

J

You know it's like there are many incarnations of that, and and and maybe stay away from that and come up with a more a slightly more abstract concept called protected entity, and you can kind of have that definition, wrapped around a helm, chart or a or an operator but which expose a bunch of you know hooks, or you know, actions that that they can take, um which is an interesting way to look at it. Actually.

E

Yeah, because well so I've been working on this for a couple of years couple three years now and one of the things- and I came out of the v'ster side, where my goal was to get backup vendors to support new vsphere objects, because we keep on coming up with new stuff. And what would happen is the the backup vendors. They would look at our new stuff and they go well where's. The money. Are you? Do you have enough customers for this and we go well?

E

We don't have customers because they want to install it until they can back it up and they go. Okay. Well call me when you got some customers, because there's there's no money, and so what I thought we should do is start flipping the model where there's a standard api that we as data protection people can use and that, instead of us trying to get data protection, vendors to chase us and data protection, vendors going around and chasing things that we meet in the middle and we say: okay, there's an api.

E

If we provide you with this api, you should be able to back it up and vice versa. You know they can say. Okay, did you give me an api, so I can back it up? No okay! Well, when you got that we'll talk right, so that's kind of the big, the big picture goal.

J

Yeah the the only issue I encountered this is with. It is great when you give the postgres example and- and it's easy to explain, but as soon as it's kafka, it's you know, people get okay. What do you do with that? You know so. That's captain's problem exactly no seriously.

E

I mean that's, that's the kind of thing yeah, maybe not that's right and that's where we've got to flip the model, because, instead of as data protection, vendors us all chasing after all, these different things and coming up with four different ways to back up kafka vsphere vms. What have you? Maybe we should say? Well, you're kafka. You should understand your data protection needs. You tell us and you you give us the api, so we can do the right thing.

J

But then how does this? How does this work intersect with the quizzing and quizzing hooks work? That's going on in the data protection working group. I mean, I think it's very similar right. It's not! You know it's a similar concept. It's just the protect. You know the the the thing that you're trying to define it with, which is the protected, indeed is sort of new, but it's not all that different from that thinking.

J

No, it's not so. I think.

E

Right now, with the quiet hooks, um those are like a primitive that we can put under the snapshot, so a snapshot should quiesce whatever it's snapshotting capture, the storage or whatever it's capturing, pretty much storage and then on configuration and then on class. So I could see the quiesce hooks being something that you can use like in your operator. So you're writing an operator that is going to snapshot your whatever you build.

E

You can gener! You can build this somewhat generically with things like class hook, persistent volume snapshots and you should be able to build a snapshotter that works without doing a huge amount of work for past one.

H

uh This is called from yeah dell emc, I can enhance. uh I can add a little bit on my experience with the level on the app consistent aspect that I wanted raise the question. So the way we do it is available provide us a hook, a hook for on a part object, and we can specify uh queer as command in that hook.

H

So when the level run a backup, for example, it is an issue apart and it looks into the annotation on the hook for the hook and and when we see the annotation up, it will execute that command that we specify in that. So that's how we can uh implement the queries and mqs at the port level uh when the product in creates an nps. We can go ahead and do our snapshot via the vsphere plucking. uh So that is how we uh employ uh uh integrate with develop our activist and backup.

J

Yeah, I mean that's, that's actually understood I mean. What's interesting, is the higher level abstraction where you know it's not just question quest, but you know, create a snapshot, create a backup or move. You know whatever it is. You know and and let the app do whatever is needed. You know under the hood to make it happen, which is interesting. It's just a higher higher level than just twice and unquies, because you usually quiet because you're trying to snap you know things like that.

J

I

I need the motivation for integration to, but not doing everything at the app level is whether things can be done more efficiently at the storage level. So, for example, if you let let's say kafka to do backup, you may have to copy all the data to a remote site, whereas you know you can do it more efficiently.

I

If storage is involved and can you know only send the dirty deltas to a remote site, so um I think that's that's where, like this forum really can help, because it's not just I mean there's cert, there are certain apps that can you know, take snapshots. Certain apps quieting is implicit. Some are explicit.

I

um You know some may back up data locally, just take local snapshots that may backup data to a remote site. So this is, I think, where we're talking about the building blocks, that can benefit a whole range of applications, and I think that's why we're having these discussions, because we can't delegate everything to applications. I mean we can, but it may not be the most efficient way of doing things well more like what we want to.

E

Do is build out abstractions for applications and provide the applications themselves with building blocks. So, for example, like say say you build out the kafka snapshot.

E

We want to be able to let the kafka snapshot or make a decision about how it snaps its data, and we can provide, for example, like protected entities that are on top of volumes where kafka says. Okay, the way that I snapshot myself is, I call the snapshot api on all my persistent volumes or you might have an application that consists of kafka and a mongodb, and you provide your own snapshot api at the top. But what that does is just snapshots and kafka puts the two together plus whatever of your config data.

E

You need and says: okay, that's my snapshot, so the idea here would be.

G

E

Wind up with like a hierarchy or um it's a it's, a dag, it's not a tree, it's a dag of the components and have the option at a level like kafka may say. Well, you know I'm going to be completely opaque, I'm just going to expose out my mo what I do and anything inside is hidden or it may say, yeah, I'm going to show you that I snapchatted a bunch of volumes and that's part of my snapshot and let you figure out how to move the data across that's the goal.

G

So dave: do you expect, like the operator, like kafka operator, to do the class or who is going to do questions in this case.

B

E

Be so it would be part of the snapshot. So yes, so the kafka operator would decide what it needs to do to quiesce. So, for example, let's say we have a volume group and it can just say: okay. Well, here's the volume group I'm going to snapshot my volume, that's my quiesce that works for me or you may have a single volume. You can say: okay! Well, I don't need to is all I need to do is snapshot my single volume or it may say. Well, you know inside here I should run a script.

E

There's some kafka stuff that happens. There's a there's, a kois hook that I'll call and that'll have the kafka app do the right thing, but that should all be what I want to get is that all hidden behind the api so that in the dp space we're not dealing with? You know long lists of hooks and ordering of hooks and all the rest of that stuff, because that's kind of like what we're doing right now, like we run across the pods.

E

If we see a pod we'll execute a execution hook, we don't know what that does it's just an execution? Look, it may do anything. So we don't really know if we're supposed to call a unquies hook. You know, what's the exit path on this, um those are all things we need that should really be inside the snapshot. Api. In my opinion,.

G

So if that's case it seems to me that then the the app operator will be doing everything, then what does the backup vendor do? In that case,.

E

Call the snapshot api but so like, for example, like um so uh right now, I've taken the valero kubernetes serializer and I put it behind the projected ending interface. What we should be able to do is um take that serializer and say you're an app, and you say: okay, I'm just defined my app. Is this namespace so do a kubernetes snapshot on.

E

However, you normally do it on my namespace and that's my snapshot, so I think there's a partnership between the um the infrastructure and the app developer and the data protection vendor. So we see a lot of value in data protection, space, scheduling, backup management, security, data movement, uh data um data storage.

E

So those are the things that as a backup vendor would be handling, but things like yeah, I do a snapshot of of x. We really want to push. I believe we should push that down to the apps.

J

And and um what about an application say a complex application or a polyglot that that may use say a database, a messaging broker like kafka and and say I don't know some web component. um So so you are kind of a snapshot on each of these things will be three different actions and you're exp. You know expecting a higher level entity to understand that, because you know we I've seen you know there are.

J

You know, applications that are kafka as a service or postgres as a service and and and in those you know, it's pretty easy or okay.

J

The operator does the thing that you want them to do, but things get quickly kind of uh they fall apart when, when you have these polyglot applications, where you know kafka is there, but that's not the only thing it does so so I guess you still need some level of serialization at a higher level entity which will do these things in some order, because you cannot just say you know snapchat everything at once. Maybe.

E

No, you can't, um let me we're getting into astrology. How are we doing on time? um Seven.

J

E

Chit chat um can I share my screen. Nolan.

D

G

Oh, I think I need you and to make you a co-host hold.

G

On okay now try to share.

E

Okay, so this was uh uh I've been meaning to put this presentation together for this group. This is actually a presentation I did with uh veritas geez in 2019, um so the idea here would be that things start to. We start to look at things as a as a as a dag, but pretty much you can think of it as a tree.

E

Right so say you have an application that embeds kafka, that embeds postgres well, the applications protected entity should be able to call into those apis that kafka and postgres are providing up upwards so, rather than having the top level flatten everything we should be able to descend down into this dag and let these individual levels take care of things. So you may, for example, say yeah. My postgres database is scratch. I don't care if it goes away, so I'm not even going to bother backing it up, I'm not going to snapshot it.

E

It's not going to be part of things and- and you hide it from the from the other pieces.

J

Does that make sense?

J

Yeah I mean this. Is I mean maybe at some point you can give us a little bit of uh you know a broader presentation on this. So exactly, but but if you had to follow this project, astro lib is the way to go right, go and look it up and see. What's going on, yeah and I've been.

E

Meaning to do this presentation here for a while and just keep getting sidetracked on other things, so we'll do it.

H

Hey I I have a this is song again. I have a small input on the on that. um I think one of the questions was how to serialize um to you know when you do backup, how do you serialize things to backup one before another um in uh 5.1.5.2 the level one? The final two I have introduced a new feature called order resources, so you can specify, uh for example, if you want to start a backup file port, and you want to backup this.

H

The first part like, for example, the application port, and the second part, would be like a like a front end port or something like that. Then you can specify them into that in in the backup. So it will execute the backup, according to the order that you specify, that is one of the features of valero 1.5.2.

C

Yeah and I all the stuff about astrolabe and protect again these, like there's, there's an early implementation in the valero vsphere plug-in. I think the the longer term goal is to kind of get get valero operating in this worldview.

C

um It's it's not there today and, like tong said we have, we have some some building blocks, but I think where valero valero is at right now is it's very uh very much more aware of of the actual process of doing things um and like what we try? What we're looking to do in the future is say: well, valero is more layered and at the top level it's an orchestrator that receives all these snapshots and then maybe replicates them somewhere does scheduling?

C

Does policy, does permission, control um and then having a layer below that uh that does the protected entity, graphs and then below? That is like serialization snapshotting.

C

That kind of thing um so, like the the astrolabe and protected energy stuff, is very future looking and kind of where we think we might want to go.

A

All right, uh we are at 9 58 last question: if there's any.

A

If not, thank you so much lorna and dave. It's very interesting discussion and uh learned a lot me personally learn a lot all of it. uh Everybody in the community is encouraged if you want to talk about a specific backup product like this, uh please talk to me and she will move forward and with that. Thank you all for today's meeting I'll give you two minutes back to your life.

E

And uh actually, if everybody, if anybody's more interested in bolero, please join our community meeting, you can find that from the valero dot io page.

A

G

Thank you bye. Thank.

J