Kubernetes Data Protection Working Group, 4 Nov 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: [WG Data Protection] Meeting 20201104

Description

No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).

A

Hello, everyone today is november 4th 2020.. This is the kubernetes data production workload meeting.

A

So today we are going to go over some cap status and after that uh we have one topic about our data production workflow white paper. So this one is about uh backup repository, uh so I'll have uh preshanto and dave to talk about this.

A

So let's first go over the caps, so we are trying to bring snapshot to ga. um The cap is already merged. Api review is also done right now. There are prs on adding the metric support and snapshot controller that is being reviewed, and then we also have several e3 tests prs that are really in progress, um and I just want to note it here that we used to have this uh snapshot and there is a monthly meeting on the morning snapshot.

A

So now that meeting is cancelled. So if you have any questions related to snapshot, you can bring that in this meeting uh and then I want to briefly go over the graduation criteria in the only snapshot cap because um we're trying to bring this to ga uh now we actually need to go over this uh production readiness review. um So, let's actually go over this. I think this is a new in kubernetes.

A

I think we are the first in sixth stretch to go through so not even uh clear on what has to be done, but but at least there are documents, so we do have documents for those uh it's just. We haven't really done anything like this before for any other features, so uh so for for snap for brilliant snapshot. Ga we here we basically just listed. What are the things that we are uh working on so like the matrix for that I mentioned earlier: that's working progress and then also attacking the crd schema validation.

A

So that's also. uh We have a pr there. It's been reviewed um and then production readiness review questionnaire. This is actually very lengthy, actually a lot of detailed questions, um so uh the first one is for about a feature, enablement and rollback. I think this is mostly for a ava feature. I think in our case we're actually already beta trying to go ga. So um we did try to answer those questions, but I think this one is less important now, because when your feature gate turns to beta, it's already uh turned on by default.

A

So this one basically says like what will happen when you enable it. What will happen when you disable the feature gate right so, but in our case from beta to ga, there's no feature gate change, it's already it's already um by default, so it goes through this one. But then there is also uh some questions about rollout upgrade and the rollback plan. So if you go from beta to ga you're supposed to use for backward compatibility, so you know upgrade what type of tests that you've done for upgrade.

A

uh Does it really support um backward compatibility so like a beta version of one snapshot should still work after the upgrade things like that, so uh and then uh also uh there's questions on metrics. So this is something that we are currently working on, um so it needs to have some metrics and yeah.

A

So the questions on uh how you, whether you have tested, upgrade roll back, but in our case since we are still just uh still trying to bring this to jira, we don't really have it yet so right now uh we have done some manu testing um it's uh well, since it's not available, it's not really possible for others to really do this test. Yet so it's just uh for us, it's manu testing for now and then um so.

A

This is why it's about application, so we don't have that and the monitoring requirements uh talks about how you so those are mostly about those metrics. So we do have matrix uh to measure how long it takes for snapshot operation and then also some error. If you know, if error happens, we also have those type of indications. How many times we have those failures, how many times we have success, so we have those type of metrics.

A

So there are those, and then I think, there's also uh I forgot where that is it talks about like the size of this change like um you're, adding this api right so um you're, basically adding more api objects to kubernetes api server, so like the size of that api object. So there are questions like that as well.

A

I'm not sure where that is. I remember seeing that and then um cli this uh sro service level objectives.

A

This one is this: one is kind of uh hard to say what exactly is the is the reasonable excels.

A

um And then dependencies does this depend on any any services running the cluster.

A

So I think in our case, uh because snapshot is a crd, is auto tree, so we depend on easters to package those and deploy them crds central control, and then we also have this snapshot validation hook, those should be uh destroyed by the distro uh and then and then the sea, the side, cars and the caesar driver. Those are responsibilities of the vendors story- vendors, uh oh, okay. I think this this one. The scalability is the one that was asking like. What are you adding um what other like the size of those uh ap objects?

A

What other api calls that you're? Adding? Because you are basically adding more costs, uh basically increase the load of the guest server right. So it's those kind of questions.

A

A

Yeah, so this is the size question size, question.

A

A

Okay and then uh it uh basically talks about the troubleshooting. How do you troubleshoot this.

A

So I think we can, we can look at the metrics uh and then we can also look at the logs. So um that's further.

A

Troubleshooting yeah, so that's it and any. Are there any questions?

A

So I think mainly right now. We need to uh make sure that we get all the all the et tests there.

A

We, I think we have a test plan somewhere, but if you test yeah, so we uh we have to have the we have to have tests for um for the finalizers so that actually already we already have a e3 test, that's been reviewed, we need to add tests for the secrets um and- and then also we have a stress test- that's also being reviewed um and then of course, once we add the mattress support, I also need the e3 test for those. So, okay, all right.

A

So if you notice no, if there are no questions, I will go back to you here. um So I wonder uh so. This is a question for you guys I'd like to know who are building products that are using css snapshots or who have already got like release, products that are using css snapshots, and that will be really helpful if you can share like if you want to put your uh company name or it's even better. If you can put a link there, you know if you have a uh some paid some website that shows.

A

Okay, we have this product using cd snapshot just to help us know um who are actually using this and who would like this one to be. Who would like this feature to go ga yeah? So, if you guys want to, you know, enter this information here, that'll be awesome.

A

That will help us to push it to jay.

B

A

All right awesome, so we see casting k10 truly. What awesome- and I know velara also has a- I think it's called a beta beta feature that supports csi snapshots as well.

A

So I'll add velour here.

A

All right, okay, uh so for the next one, is a container notifier uh for content. Notifier. uh We have that cap, that's still being reviewed. I think there's still a couple questions that we need to get resolved, so we will need to uh schedule a meeting um with the aka reviewers to figure that out yeah. So um next one is the generated populator ben. You want to give a update on this.

C

um Just like a status update for people that don't turn up.

A

Yeah meetings, yeah.

C

Yeah, um so we've we've been having a weekly meetings on uh tuesday mornings, pacific time uh to go into detail on this. We've had three meetings so far and we've uh we've made some progress. The proposal going into 120 had been to implement like a validating web hook, uh to uh to notice. When pvcs had data sources that that were never going to bind, they were never they were going to result in a pc, they never bound, um but we we didn't like.

C

We decided we didn't like that design and decided to go instead with a controller model where we just basically post events to pvcs that look like they're never going to bind, because because it's possible that the populator that will bind them is going to get installed later. So so it's actually okay to have those pvcs sitting around just waiting for a populator to get installed. So so that was the first sort of decision we made that was deviated from the original plan.

C

We've spent some time talking about how we're going to implement the data populators themselves, and we were fairly happy with the the scheme that I prototyped that involves basically creating a secondary pvc that is empty.

C

That mirrors what the original pvc looked like running, either a pod or some other process on the empty pvc to populate it and then rebinding the pv that got bound to that pvc back to the original pvc that the user asked for so that at the moment their pvc binds it already has the data they were expecting and any pods waiting on it can can immediately start up. We've did some investigations about.

C

Can we support models like wait for first consumer with data populators, and we believe the answer is yes with a little bit of extra work, and we spent some time talking about, uh given that there's going to be some fairly complicated code to implement the mechanics of the of the the watching and the rebinding and um and respecting all of the kubernetes rules.

C

Like wait for first consumer, we would like there to be some common code that can be shared among various populators and it's an open question whether that common code will be like a sidecar container type thing or just a library.

C

You can import for the sake of expedience, we're going to start off with the library and uh but try to keep our options open so that you know if, if, if we can turn it into a sidecar later on, we'll we'll try to do that, um but but basically um we're making some changes to the to the cap in 120.

C

The feature will remain alpha in 120 um and we're going to see how far we can get towards actually implementing the controller that posts, events and samples of the of the populator mechanism and reusable code, so that we can get things rolling and maybe get debated in 121..

C

Any questions about the that.

A

Effort, I wonder if uh maybe not everyone understands that uh if you know the importance of wait for first consumer, yet I think normally at the restore time people probably just do static provisioning. I guess right so.

C

I I don't know what people do do it restore time, but.

A

I'm just guessing I'm just because without this I mean you can't really do real. um I mean you can do dynamic progression, but you can't really support a wait for first consumer. Without this feature, I don't think so.

C

Well, I mean like like today: the only thing you can restore from is a snapshot or another volume right through a clone, and those do support wait for first consumers, so that they would get scared.

A

Yeah, if they, if they do.

C

A

I think uh probably backup vendors, mostly probably can't really use that feature directly at the resort time, because the data, if there's data, is not really managed by the csi driver, etc.

C

Somewhere yeah, if you have, if you have a backup scheme that involves like creating a pod and attaching it to the volume that the user will be using um then yeah like then, you can't have a situation where you specify what the application should be. At the same time, you specify what you want to restore it from you'd have to sort of go through a whole process to do a restore and then tell kubernetes. I want to run this application so yeah. It's um like.

C

I view that style of backup and restore as like, being layered on top of kubernetes, which is fine and great, but but what we're looking forward to is a version of all this, where the the restore happens inside kubernetes, as if as if you just specified a data source for a pvc and boom, your data pops out and like you didn't, need to do anything as a.

D

C

That that's what I'm looking forward to.

A

Yeah, I'm not actually not sure um at restore time if people normally, I I think they normally just assume that the original data source still there I mean, but I could be wrong- maybe maybe uh it's uh not true, maybe already solved that problem.

C

A

C

A

Oh, I'm just saying that uh normally your individual static provision right, then you kind of assume that it's the same, uh the same storage right at the restore time, you're not really changing anything. It's static, provisional, not really dynamically, selecting anything. So you don't really know if at the restore time, if, if that, um like the story, location still has enough capacity or anything you don't. You wouldn't really know right because you're not.

C

A

Dynamically deciding where to place.

C

You you can, you can today create a scheme where, as where you implement what is actually a backup and make it look like a snapshot, because we know that some snapshots are implemented as basically copying all the data from the volume to another place and then right.

A

So if you yeah, if it's kind of hidden under the csi driver yeah, you can do that right. You can just do it.

C

From snapshot that you can do so if your implementation is inside the csi driver, you can make something that feels like a backup. But you call.

D

C

D

C

You could just do that today. What we're trying to develop is something that is more properly a backup that doesn't have to be implemented inside the csi.

D

C

But from a kubernetes perspective still feels very similar to just restoring a snapshot where you just say: hey.

D

C

Volume that has his data go and, and it we just want to basically break out of the the shackles of the snapshot. You know the limitations of you know what you can and can't do in that model um and come up with something more flexible, but um but still have it be very much like a kubernetes data source.

D

Okay, in this case, the uh external probation is out of the picture right.

C

Not out of the picture I mean the external provisioner might be pulled in to create the empty volume onto which the data will get to get restored, um but.

A

If you, if you do that, then you can't really guarantee that then it's all. If you create an entry volume, then it's already bound and then you can't really create a part ahead of time right well,.

C

It depends. This is why this is why we we do the the rewind trick, and the second and.

D

Second, pvc.

A

C

Around those issues at.

A

C

In the prototype so sure I'm happy to go into more detail about how that all works. um But I'm hoping that.

A

C

We just have code that demonstrates it all, and then it will be more obvious.

D

You're saying that there's a bi-weekly is that right.

C

D

B

C

A weekly meeting so far the invite was sent out to the sig storage mail list. I don't know three weeks ago because we started.

A

Three weeks ago, yeah, actually we can also yeah, but I don't know if uh everyone during this meeting also have time to join that meeting right. So that's why I would like I think, but I think that actually.

C

A

Important for people on this call as well yeah.

C

It's been a fairly high bandwidth meeting so far, but like we're, we're slowing down now, like we sort of had a bunch of stuff to plow through at the beginning, and now now it's just a matter of coding. There aren't that many open questions, so I'm not sure what we'll discuss next week. We might want to reduce the frequency to every other week potentially, but but I I can, I can post a link to the agenda doc, which has the zoom link and all the other information in this agenda. Doc. If you want.

A

I was so so bad, I'm thinking, maybe good for you to like do an overview. Just explain why we need that. I mean I know, because currently I think people already have their own solution. That you know does not rely on that. But then, if you can explain, uh you know why we need that. What's the, what is the advantage of uh using a feature? That'll be very good right, so maybe, like.

D

A

I I think, probably you probably need to uh yeah. We didn't even talk about this one, so we are. We still have some other topic to go so maybe next meeting, if you have some time just to.

C

We've gone through it a few times in this venue like.

A

C

A

C

A

I wonder if you like, if you can draw like a simple diagram, I don't know that's too much trouble, but uh maybe.

D

D

Under discussion, then, maybe wait until ben is more comfortable. I would say.

A

I was just thinking of like explain to people why why we need that feature? Even I mean not even going to the details on how to solve the problem yet right, because we're not really that.

C

A

Working progress.

C

I I don't want to. I don't want to give the impression that, like we need this feature like there are many ways to to skin the cat of a backup.

A

Right right but you're saying that what but like when you, when you introduce the feature when you write the cap right, you have some section explaining the motivation right, so you might, in my case I found this. I didn't realize this. Is that important until I realized that I need to wait for first consumer? So, but maybe you know in general of course- and we know this is useful, but for me that was the one that I think it's really.

A

We really need that if we want to solve this placement problem, which we currently cannot so okay.

C

A

Just like high level what.

C

In this meeting um or we can, I encourage.

A

C

To to come to the to.

A

C

In more detail next week, um you.

A

Could do that too yeah we can yeah. So I can uh just fold that in my to this mailing list as well, so people can.

C

That's one way to do it um sure, but also just maybe post a copy to the agenda doc in this agenda.

A

C

People who have that can just find it on their own okay, without going through the email.

A

C

Sure, let's, let's do both.

A

C

Are welcome to to come.

A

Okay, all right thanks, uh okay, so uh then next one cozy, basically right now it actually has been very active. uh There are still weekly meetings, poc progress and the cap is merged, it's provisional! uh So it's uh doing a pvc. It's a poc poc implementation for this release, trying to go off in the next release, 1.21 and what in group so there's a cap there that has been reviewed. I think I need to schedule another review meeting. There's there are some uh questions being raised in the previous meetings.

A

So that's why I have not uh um scheduled meeting for. I was trying to see how to how to solve that problem. So I updated the cap. Take a look if you are interested and then I will also try to schedule a real meeting for that. Okay. uh Next, we uh have a topic on backup repository.

A

um So, let's see, if prussian okay, I see pressure there dave, I'm not sure. If dave is here so appreciate the this link, you added here, it looks like there's only half, it doesn't have the full link.

E

Let me just put that.

A

Chat: okay, so.

A

All right, do you want to go over this actually do you want? I don't know if it's better that you share. Actually I can make you.

B

A

Okay, try that and I'll stop sharing.

A

A

Work: okay, yeah: we can see it yeah.

E

Cool, thank you so much shin, so hey everyone, um dave myself and shin have been working on the backup repositories or the backup target piece for the overall data protection white paper and uh overall I'm going to go over. You know high level requirements of what we need for a target, but just kind of level setting. Why are we doing this?

E

You know when we do uh backups or capture data from a kubernetes environment. We need a location to put those backups into. We obviously have snapshots and snapshots technology is good, but most times you know, obviously not not the cloud providers, but most times snapshots provide a single point of failure. uh So you know you do need a centralized uh location where you would want to move your backups, especially from a migration perspective.

E

um And again uh you know. Data has been growing at a very exponential pace today and uh we need to ensure that the data, whatever we are storing or capturing within the repository um you know- can handle scalability performance aspects and you know, can be stored in a much more efficient fashion.

E

So with those items in mind, the high level objective of whatever we have done here so far is to provide some requirements for the target repository, and then you know, along with that, any kind of guidance that we may have on specific items.

E

uh I'm not going to go over all these uh items over here because they have been listed as characteristics in the table below so we'll just start with this table.

E

So uh the first item that we have is supporting multiple protocols right so from a target perspective or where you store your backup data we want to make sure or customer would want to store it. uh You know probably use object, storage for scalability, uh you know there could be existing file servers. uh You know nano file, servers or emc or storage systems providing nfs storage or something on those lines which are already available in the customer environment which they would want to connect to.

E

Along with that, you know, you may have data domain type of appliances which uh you know which may be a use case again. You know this is kind of looking at it and prioritize use case perspectives of what we expect to happen more or more uh so obviously object, storage being. First, then, you know some kind of file storage. You know using technologies like data domain and so on. Next and then the other point was you know, we were also looking at uh good. Customers want to store their.

E

uh You know, backup data on direct cloud storage like uh ebs or azure cloud cloud storage. Instead of you know, using azure, blob or s3, uh for example, uh we weren't 100 sure on this. You know because of the cost reasons- uh and you know the usability aspect of it. Unless and until you know, customers are doing a lot of on-demand and continuous migrations and want to use that ebs uh volume.

E

As the you know, as the central connection connection point between two clusters, you know uh that is a use case, but it's not that uh well defined in our opinion, yet um suggested improvements. uh You know, we've been talking about. Cozy cozy has a lot of uh you know apis to create the object storage pieces, so that can be something that can be leveraged there. Could be certain improvements that could be made to csi as well, but because I believe, cosy is already down that path uh pretty much.

E

We probably can just leverage cosy to do all that. Obviously, certain things need to be kept in mind. uh You know object storage for one entity may be different to another entity in terms of how it's implemented underneath the covers, and uh from that angle you know migration use cases from this is migration from one target into another target uh probably needs to be thought through. You know what would be the underlying changes in terms of storage capacity, how it is laid out performance. uh You know, performance, deltas and so on.

E

uh So after defining or listing the protocols, uh you know, we would want to make sure that we are able to do this on-prem in cloud-based environments, whether it's you know, azure blob, aws, s3, google cloud storage. uh That would be the next kind of requirement from the target perspective. You know one is obviously supporting those protocols and then the other one is supporting those protocols in different environments.

E

Again, cozy provides the apis to do this and can be achieved pretty seamlessly through that from a redundancy perspective. Customers to want to ensure that the target has some high availability. This could be uh jio based high availability. You know if you're using cloud based technologies, aws azure, google again, um you know you can have the geo availability from a specific cloud provider or you know you can also have uh customers looking for cross cloud uh redundancy.

E

You know having a target created in aws and then having that replicated to another volume or another kind of object stored within uh gcp or something.

E

The next item here is long term archival. So once uh you know, once you start storing the amount of uh data into your, you know different s3 buckets or you know, azure block, storage buckets or something like that. um You may want to have a tiering uh policy that basically moves the data from s3 into even uh colder storage or even lower cost storage than s3. Something like uh glacier. I forgot what it's called on the uh azure side, but they they also have something similar, so the other.

E

So this point basically talks about taking the data and basically based on you, know, retention policies or archival policies. You automatically move them into a lower cost storage. If it's available- and you know having the target or you know whether it's the target storage automatically moving that or you know the backup vendor doing it, but that would become a requirement underneath the covers.

E

I'm just going to take a quick pause, any questions so far.

F

Okay, you mentioned uh cbt is there, um is there this that requires kind of storage integration as well in some way, usually right, correct.

E

So I think dave I was hoping dave was on the call, because dave has some information around. You know different kinds of target repositories which he's defined as raw passive active. um You know the raw target repositories are basically, you know just kind of dumb storage where you put all that data into it.

E

uh Passive inactive is where you know those repositories kind of become a little smarter and help you with you know apis to provide that information around cpts or doing full incremental snapshots, restoring that directly as a full synthetic backup directly from the target.

F

Yeah makes sense.

E

Okay, so moving along the next one is uh permissions. You know once you have the capabilities of you know. Obviously this this probably all falls into the cozy aspect of it. uh When you're creating the target you want to be able to create, you know different kinds of object, stores in different locations and apply the right permissions around it as well.

E

So we probably would want to move this one up over here, because it's all kind of provided by the same set of apis that cozy presents so overall seems like cozy would be a you know, pretty uh innate way of hooking into targets and doing pretty much. You know the probationing and handling of all the permissions for uh creating a target within kubernetes or for capabilities.

E

uh The next set of items are around encryption. You know encryption. Obviously it could be in multiple things right. The encryption can be provided by the backup vendor or it could be provided by the storage vendor. That is at rest. So one of the biggest things with encryption is obviously integration with.

E

You know different key management softwares and being able to interact with that key management software to get the right keys to decrypt the data and move it into the target storage or if it's not not encrypted at source, then you know taking that source information uh encrypting it sending it over the wire or it could mean you know just having the target doing it at its level by doing data addressed kind of encryption, which would be kind of agnostic to the backup vendor, but would be handled by the target provider itself.

E

So I think the the net net the way we look at it is it should be kind of open. You know if the storage provider has the capability, the uh you know. The backup vendor should enable the storage provider to leverage that, because from a speed of backups and everything that may be faster, and obviously uh you know if that does not exist, then the backup vendor would want to provide that as well as a added feature.

E

So I think, depending upon you know how the customer's environment looks like this. The encryption could be offered as choice.

E

uh Next items deduplication and compression uh similar to encryption, um you would want the ability to enable disable the duplication compression again. This can be provided by this can be provided by the storage target again directly or can be something that the backup vendor can handle uh before sending it to the target. Again. This should be something around choice that should be available to the end user or the end customer uh deduplication and compression will. Obviously you know speed up the transfer protocols.

E

You know how much data you store on the target will also be reduced, so avoids not only performance savings burst, cost savings as well and then finally, the uh last one was target management by multiple clusters.

E

So, let's suppose, if you have one or two clusters mapped to the same target which cluster gets to own the data and which cluster can access the data or delete, the data should be well thought through and there should be some kind of locking mechanisms, or you know some intelligence underneath the covers through the software, which is actually placing the data on those targets to ensure that there is no overlap.

E

Or you know, the data is not uh incorrectly deleted or is actually accessed and deleted by the right source, so especially useful in migration scenarios, something that needs to be taken care of, as well from the backup notes perspective provided some information on cozy. I believe this information is probably better, probably better provided than what shin is sharing in her main document.

E

um Along with that, there is astrolape. uh This is another open source project uh which focuses on you know kind of standardizing the target spec that we've been speaking about so far, so it would be interesting to take a look at that and then the different backup repository types. You know, as this is something that they've had come up with, which I agree with as well.

E

You know initially, right now we are working with all raw raw kind of storages. You know nothing, no intelligence. On top of that, uh everything is handled by the backup vendor to do you know full synthetic backups merging your incrementals to the uh base for snapshots, so that is the world we are living in, but uh you know, as things improve and as we abstract more layers to you, know further application deployment and delivery.

E

uh We will kind of see more of the passive and the active targets as well, where you know a lot of the handling of the incremental snapshots and merging them into full synthetic would be handled at the target level without really having to worry about it. You know from a backup perspective, so just kind of you know some points to think about to chew on you know, as uh as we kind of grow through this data protection realm, uh you know as products get a bit more mature.

E

uh You probably jump from raw to passive to active.

E

A

So I think the this part, the raw past activate. I think this overlap with the what we talked about in the cpt meeting. I think we talked a little bit about like if, uh let's let's say, if we support bt, then should we have the backup repository um managing those.

D

A

And trying to do trying to do the merge or those.

D

Type of things.

A

uh And we, I think, at least at that meeting we last time we talked about that- that we should leave that to the backup vendors and not really introducing that in our kubernetes api. I think that's, uh that's it.

E

Yeah, so is dave part of the snapshot. What is it the cbd meeting as well.

A

Yeah he was there as well. That was the after yeah. That was probably after this one.

E

Probably that was where he probably brought it up as well. So yeah, that's right! It's uh it's only provided as suggestions.

A

Yeah yeah so yeah.

E

um In a high level, this is, this is kind of the crux of you know what we think and again the you know inference right now, based on talking to you, know, customers, and so so on we see 80 are kind of aiming for cloud-based targets. Obviously there are a lot of different reasons for which you want to do on-prem, but your combination of both you know dealing towards cloud mostly.

E

Any questions guys.

F

uh You're, mentioning backup uh repository uh kubernetes apis um is that is that cozy or is that additional stuff as well? Which point are you exactly referring to well so this is kind of expanding on zhing's question. If we're um you know what what apis do? We want to make to support different backup repositories um kind of if any right, because I think cozy is a pretty low level api that you know, gives you an interface to audio storage right, so you know maybe the raw raw tiers, the apis.

F

We want to define which we have. Essentially, we have block devices and.

A

Yeah cool, so I I don't know if cozy I don't know if cody is raw or passive um passive, meaning what meaning that we provide and common api right. What's the difference between draw passive pass, meaning we.

E

That was the same question. Actually I had for dave. I think he explained it to me was more around. uh You know having an api, obviously to talk to the passive piece, but uh raw. There would probably be no api. You would have to directly. You know.

A

Call the native apis bc right. That's what I'm understanding, so I think so I think cozy is also a kind of passive api because that's trying to provide a common kubernetes api, but.

F

It doesn't, it doesn't, give you snapshots right, it's not a backup target.

A

It's not, but it's I think, uh yeah. Okay, so maybe yeah, that's true! Maybe not uh yeah uh he's yeah! I just pinned him he's actually driving right now. So.

C

The expectation is that the buckets that provided by cozy will be used as backup destinations and sources for some implementations of backup right, but those have to layer on top of cozy um yeah.

A

But I think, if I look at what they were saying here, the passive heming this api actually can can provide. You can actually get the snapshots uh not just because I think cozy it only provision bucket did not you can't.

D

A

Objects very scary, so I think what he was talking about. I think this is like one. The other layer.

E

Probably like contributing some code in the future to make it that way, I guess.

C

Yeah, I I see cozy as a building block to building an eventual backup api, but by itself it's just object.

A

Storage, it's just about it, yeah, it's just bucket, so more more like a raw, but it's a but it's kind of kind of complicat for other.

C

Buckets but the question about like a kubernetes backup api like that's what we want to build here in this team and it doesn't exist yet but like we're working on it, so yeah.

A

So if someone says there's a.

C

Kubernetes backup api, like they're, pointing to something.

A

Oh, no, that's the backup repository. That's that's not really backup api. Yet backup repository is just the deposit representing like the your object, store or nfs, or something or backup device, or uh you know, on-prem backup device. That's the backup repository! That's not the backup api! Yet um the backup api uh yeah. We don't have that. Yet we have uh andrew's dock.

A

He usually talked about that.

C

Does did andrew large present something, and I missed it,.

A

That was very, that was like almost in a very first meeting. I don't know.

D

Okay, yeah yeah. He.

A

Was just talking about, I think I remember he talked about okay. This api should be very similar to snapshot, except that it has this uh um pretty clear definition of what becca means. You know, backup, meaning it's a different device. Not your primary storage snapshot does not have that uh differentiation. Basically.

C

So he- and I agree on that, but like I've been hoping that he would post some more details.

A

uh I think I think someone is is that alexis he's working on this right he's, I think alexis says, he's working on the backup apis. If.

A

Yeah, so maybe you know, maybe we can uh hear from that yeah. So that's the backup baby. That's still not the backup repository yet um yeah.

F

Yeah, I think, what's maybe we can figure out what the motivation of this is. My understanding of the white paper is basically that we want to kind of put together a summary of what we need to be able to perform backups in kubernetes.

A

F

Then the fallout of that will be the the components in kubernetes that we can build. That will be useful for that, and so it seems like the motivation for backup repository. Is you know? Maybe we expand cozy to include the actual um object interactions or maybe different interfaces. You know, but I think that's for me. That's kind of what falls out of this section. You know the need for an interface um and in fact there already are common interfaces out there, that you can use right.

F

There's libraries, that kind of abstract away cloud provider interactions, and so the question is: do we need to add that? Do we need to define one of those in kubernetes? You know: is there something correas that would have to directly interact with objects? Is there something that or can it just use kind of off-the-shelf interactions right, try using sdks or other existing interfaces.

A

I think those are open questions right. I think uh it's I thought. Oh, I think someone uh is that maybe alexis and elite are also working on this. Is that right or no a couple repositories.

G

A

And now we can okay yeah.

A

Right, so I think this is still tom. I think that's still like an open question, um probably that's something that we just. We need to just still need to discuss and uh clarify what else do we need to do so now we do have this cozy.

A

uh That is, uh I think, it's pretty active project actually, um but from that do we want to add anything else uh for us to use that as a backup repository, I think that's still open question.

F

Yeah and maybe you can treat it orthogonally for the white paper here too right, that's white paper. I think I mean this as is even summarized it's kind of a it's. A good summary of what background product stories are, what they.

A

F

An ecosystem, and maybe like as a separate task or a separate discussion.

A

Yeah, like yeah, we not yeah. We don't necessarily have to talk about like all the api requirements in the white paper, yeah yeah.

E

I think, probably from a p0 perspective, this is good for now. You know once we can.

A

E

This then we're going to move into you know figuring out what apis and everything I needed.

A

Yeah yeah, I feel like what they were saying. The task is more like you actually have the api that can retrieve a a particular snapshot. So you can. You can actually use that at the restore time thing kind of like that at that level level, correct, yeah, basically right.

E

You don't have to yeah so.

A

That's what the the astrologer has done so about.

E

It, oh okay, that's where he was going.

A

So that's the the plug-in. This is the velar plug-in for visual uses that asteroid layer to do those yeah. It's probably what he means. I guess, and then I think active is more uh yeah, another layer that you actually the the backup repository actually also kind of managing all the things that are stored there, uh because if you're just using um let's say s3, it doesn't really manage those like you have your change blocks. You have your you know. It doesn't really manage them for you right.

A

It does not really construct them for you, that's actually pretty complicated, so uh yeah. So I think that's what I mean.

E

Basically, I think this will kind of centralize. You know how you interact with the target, whether you use one provider with another provider. In the end, the target basically takes care of how you know how the incremental blocks are put and merged, and you know.

A

Yeah yeah, I think, yeah I think, but at least last time we when we talked about speedy, we thought okay, this at least right. Now we don't want to let's we leave that to back our vendors, so.

F

One thing we might be able to add to this as kind of the survey is including the apis that exist already for different storage providers. Right, if we're interested in figuring out what abstractions we need to build.

A

F

You have there's kind of two sides right, there's like the storage side and then the the repository side for this, um especially with cbt cpt's, pretty hard to come up with.

A

Yeah for cpt that's another. We have another topic for that right so right now this is just the backup repository so um yeah. It's sometimes a little confusing like we're talking about.

D

What we're talking about.

A

It's ability to have a different set of uh apis right or are you uh tom? Are you only talking about like how we like do merge for those vts on the backup repository layer? You mean in that level.

F

My view of that repository is that it's able to handle consuming uh like change blocks. Oh.

A

F

Help make it efficient, but that may not be. That may be like a higher level layer right. I think it's actually talking about the primitives that, um like the storage primitives, that backup repositories consume and so there's kind of two sides right, there's the what does the repository need even a higher level repository layer and then what does the? What do the volume storage providers provide?

F

And that's, I think, maybe that that's what you're talking about is in the cpe discussion. So.

A

Okay yeah: maybe we can uh talk about cbt more and then come back to this one.

E

Sounds good anything else, guys.

E

Anything else we need to talk about or anything else. We probably need to add here.

A

Okay, so your so this talk is a link between that agenda. So uh please review it. If you have any comments, feel free to add to the dock.

F

Preshanta, where do you think uh rustic would fit in this framework? That's interesting.

E

um I think that rustic would probably be another. I think my knowledge on wrestling is not that great. But what is the underlying storage that the elastic requires? Any? It can be anything right. It's an object, store, typically, okay, so probably it would probably something much like one of these. I guess.

A

Does resting required that the back end to be object, store or is it can be a best.

C

Well, I don't know, maybe the the form that we use. It is always object store. So that's the only one I know about, but.

A

Oh you use rustic as well.

B

C

B

Is really cool.

C

um uh But uh yeah, I don't know if it could also use like an nfs store. Oh.

A

C

You can just give it like a volume and give.

A

C

Bucket and it'll back it up and restore it.

A

E

Yeah so so, if that's the case then it would probably be you know one of these and then on top of that it would be super imposed kind of like abstracted, but.

A

But that I don't know it seems to be itself, it's not really. It's still need a backup repository for the data stored. I think it's not like.

C

A

You can contain.

C

Some metadata to remember where your backups are, but.

A

C

But I mean itself.

A

It's not really it's not like a s3 orchestra type of thing right, it's just so. I think she that should be some course something else. I'm not sure what um it's almost that's. That's the way to uh do the backup for you right. It's not really.

C

Yeah, it's like the the the primitive layer of just doing the backup in the restore.

A

C

You can build on top of that, but.

A

It's not yeah, it's not just the repository repository, meaning that this is like the place where you store your data, because the graph using wrestling you still need to specify where to store it right. Otherwise, it's not going to work. I guess.

C

Well, I I guess I'd be careful around using this term repository, because what what matters is that you know you have an object that represents a place that you can do a backup to and.

D

C

Object that represents uh what you can restore from, and a repository is a nice way to conceptualize what that is, but it's it's kind of rigid and not as flexible as you might want, or you might want the ability to move backups between repositories or you know, do other things with like have meta repositories. There's all kinds of other constructs. You could imagine what matters is that you know that when you take the backup, you know where it's going to go and when you're doing a restore. You know where it's coming from yeah.

E

I'm just looking at the rustic uh documentation right now and it talks about preparing a new repository. It can be local, it can be s3 mini or wasabi azure or google cloud storage.

E

Different uh yeah.

C

Rustic is like a swiss army knife for data background restore. I.

A

Think it's just performance is not good right. I think that's the only downside.

C

Yeah I mean it because it it reads all your data to to do the backup, which is, if you don't, have anything better, it's better than nothing. But it's it's yeah. It's not always the best way.

E

Yeah, that's a kind of b can be a backup vendor in itself, because you know it provides like even encryption, and you know selective backups and things like that. I mean obviously not 100 for a kubernetes environment, but someone could extend on top of it. Yes, like valero.

F

Builds on top of that, for example, exactly.

A

F

I guess what I'm asking is you know, how do we want to define the boundaries of what's in the repository? Is it because it seems like there's many many layers right? You know what? How do we, how do we? How do we name these layers? How do we talk about them? um That's kind of my method.

D

Tom, I think the repository just a placeholder right with regarding what gets stored into the repository is not really or a decision like birthday can choose to store something and some other mechanism can choose to store. Others.

C

I think I think, having it having a conceptual repository is good when there's more than one choice for where you, where to put the data, but once the backup is created like it's it's in a place and and the object that you'd use to restore from better know what that place is without any external references.

C

C

Like it, if you can only get back up to like like, uh but the way snapshots work right, you don't have the option of like. Where do I want my snapchat to go? You just say I want a snapshot and it goes where it's going to go. You get zero choices or you get one choice: the the snapshot destination so.

D

Like snapshots still have a reference like the snapshot handle okay, what I mean is.

C

That there's there's no choice to make about. Where do I? Where do I want it to send the snapshot? There's just the snapshot with backups. The the extra detail is: maybe you give the menu of options I can back up to here or here or here, and you know which one do you want.

F

That, but it's on both sides too right, because you also the things you consume are very different right. You might want to consume. For example, output from you know my sequel dump. You might want to consume a foster directly, um whereas with volund snapshots, you know, you know the thing you're taking the backup of yeah. So that background you know, I think, there's a lot of layers and backgrounds that make it a little more complex than the snapshots.

A

I think we are running out of time uh yeah. It looks like we maybe need to talk about this more uh and just define like what are the layers, what should be handled by black color repository? What is not.

A

Yeah so yeah we can. Maybe we can have some follow-up discussions on that. All right. Thanks appreciate it.

E

A

All right, uh I think, that's it for today, are there anything else, otherwise, we'll meet in two weeks thanks. Everyone.

E