Kubernetes Data Protection Working Group, 22 Sep 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes Data Protection WG Bi-Weekly Meeting 20210922

Description

Kubernetes Data Protection WG Bi-Weekly Meeting - 22 September 2021

Meeting Notes/Agenda: -

Find out more about the DP WG here: https://github.com/kubernetes/community/tree/master/wg-data-protection

Moderator: Xing Yang (VMware)

A

Hello, everyone today is september 22nd 2021. This is the kubernetes data protection bohenku meeting today we have one main thing in agenda, which is uh the cvt design discussion uh so fun. uh I think you can get started. I already made make you the co-host so, okay, thank you.

B

Okay, let me start sharing my.

B

Let me let me start sharing my screen.

B

So um last last time we met we, we were discussing about making um the cbd the difference of snapshots as a csi service.

B

Otherwise you know it's just be like a private for each company right, so um we I dig into that uh csi service a little bit more and I can see that um we only so far. We only discussed about uh almost like one half of the the component, so I want to bring it down here, so we can discuss in more detail. So there are two half of the the first half is.

B

We have been talking about that. A lot is the the change on the csi drive to add this service, so each um storage vendor, if they want to add this service into their csi drive, they can implement this differential snapshot service on their csi drive and the um the api right now is just um should have one which is get chain block and then the response will be. um You know the the chain block respond, that we have been discussed in the last few sessions that we met.

A

So should this be a separate service in csi, or should this be part of the controller service right now? This is all.

B

Controller, I look I look into uh so right now. If a csi a vendor, provides csi driver, they implement these three interface by the identity, the controller and right.

A

So snapshot is in the controller, so I wonder what should be actually part of the controller? It's just. I wonder.

B

A

Can enhance this controller service by adding a controller? This is many. There are many controllers. It's not like just one controller snapshot has a controller, and you know for pvc. There are several controllers. Three sizes controller right. So it's not like you have this one uh controller you can only oh, you are talking about. Are you talking about from the csi driver side procedure?

A

I'm talking about the csi.

B

See it's a driver side right.

A

B

Yeah this one is for the csi driver right.

A

So yes, if we want to make this like a separate css driver side, if you want to make this like a separate part of something, then yeah, then that needs to be a separate one, so yeah so.

C

B

C

To mention that everything in the csi rpc today, at least, if it's part of the controller, the only thing that can consume those rpcs- are things running in the same pod, which today is like all of the side cars like unless we're planning on, like writing a sidecar that consumes this. It doesn't make sense to put it in that api, like maybe a separate pod. A separate service uh starts to make more sense, but I still would really want to understand what is the client for this service? What does it look like?

C

Where does it run? Who writes it? All of those things.

A

Yeah, this part is not. I think that part is not not sorted out yet. I think we still need to talk about that.

B

Yeah, that's why, at the beginning, I say that we have two half sites this. This one is only half of it right, the the the vendor the started vendor, providing this as part of their csi drive package.

B

They implement this service right and then the api to talk to this service we've been discussing before and when the vendor created that I mean up support that service, then they would in the get plug-in capability.

B

They would have to put this differential snapshot service, along with other uh services that respond to in the response to this gap, plug-in capability right. So that is, uh that is the first half of the the the problem that we I mean of the structure that we're trying to uh create here and this.

B

So I think I put all the material that we have it discussed on the related to this api in this document, but this is not complete yet because now we need to talk about the second half right. How can we, uh how can form a backup vendor right from from backup brand I want to, particularly if you want to do this service? How can I talk to this this service that we added here from the vendor right?

B

So I was looking at the um the csi, uh the csi, uh the sample for the csi uh architectures right, and I think that we might have to enhance the um the kubernetes um control, the the kubernetes snapshot, controller to or or maybe some components you can.

A

Have a different control: you don't have to use the same controller, but so that would.

B

A

So it can be a separate controller if you know we'll see what makes sense.

B

Yeah, that's, but that is what I'm thinking that we need to enhance.

A

So this does this also I mean: does this need like a common controller part and a sidecar part, or that I think we have not even figured out that yet.

C

That's why we probably.

A

Need to look at your the overall flow first yeah.

C

And we need to think all the way back to the the the clients of these apis who's, creating snapshots and doing things with snapshots. How do they figure out what they can and can't do like if you have assists, if you have a system where there's multiple csi drivers, some of which can do this, and some of which can't do this? How do you find out, if it's even possible on a given snapshot to to consume the service yeah.

A

You have to think through what they do. Okay, csr has the capabilities you could.

C

Right right, but like the the consumer of the api, doesn't know if it's going to work or not right like they create.

D

C

Object and they may or may not get a snapshot and then that snapshot may or may not be able to support these. You know diffing apis. You might just get a snapshot that doesn't support it and then what do they do?.

A

Yeah so seeing that they have to do full backups, then right. If uh so, they check the reality. If it's for this, but we have not decided whether this is just.

C

That user is not talking to the rpc they're talking to the kubernetes api and there are no capabilities in the kubernetes api really.

A

uh So I think we also needed to have a sidecar right. So the same same thing.

B

I think that's a very good question that that we like starting from the user of this service right who we're going to talk to right. I.

A

Mean we can probably think about like what do you do today with the snapshot with the snapshot capabilities? Do you actually check those? Do you the backup software, always assume that you can successfully create a snapshot whenever you request it? Is that what you do? Do you actually check capabilities? I guess that's the one yeah.

B

We, if I create a snaps balloon snapshot or whatever from the client part of you. I continue checking on the volume snapshot object until.

B

You know act on this.

A

So it's possible that you can. You are not really getting the snapshot back, even though you request it right. Basically,.

C

Right yeah, yes,.

A

Then they'll take the csi plug-in.

C

Might not even include the sidecar if they don't have support for snapshots they wouldn't they definitely would not.

A

Create a volume snapshot class right if snapshot is not even supported, and then this one should not be there. You know, I don't think we want to support those without snapshotting. What's the point, so the.

B

The problem that we have right now is that for volume, snap fault, we have a kubernetes uh object for it uh service. We don't, but.

A

Right, well, that's the thing that that's why last time we were talking about it, I think we actually still still have it. That's why I was saying that uh I think shenzhen, uh unfortunately he's not here. uh He brought up this uh api extension server. uh I think I think we can look into that.

A

I think the reason that we're saying we don't want is because we know concerns regarding performance, but I think we should look at the api extension and the server and see what we can do with that, because otherwise I mean, if we can't even consume it from kubernetes, then do we really is this three.

B

Could you tell me a little bit more about that api extension server, because that the concern that we have the cr the object that we create will be too large, especially in so sure.

A

You have a separate server in case. You have a separate, I mean not separate it's like extension, so it's like. I don't know the details of that. So I'm hoping that something's not here, but maybe now you can take a look of this offline. I don't know if anyone else on this call know more about that. I.

C

Think the the way that the way that api extensions work is basically kubernetes delegates all of the the rest api for some resource to a separate http service. The cube api server isn't even isn't even handling those requests, and that would definitely help with like performance issues around.

C

You know just the the raw talking to to the storage uh driver and getting like the list of blocks like. If that's a lot of data yeah, you want that to be consumed efficiently. That's very important, but there's there's a concern that comes even before that which is just you know. How does.

D

A user, the kubernetes.

C

Api, like know on a per volume basis or a per snapshot basis, what to talk to or where to talk to it, because we've designed csi to allow multiple csi drivers to sit side by side and users typically, don't know that right. They just have a bunch of storage classes and snapshot classes, and you know typically, you just create pvcs and create snapshots and expect something to happen.

C

But here's a situation where, like something might not happen, you know you might get a snapshot that you can't do anything with, or so we have to answer that question first of like how do you even get started with with you know, creating a snapshot figuring out that you can do this and then once you know that then yeah, of course, maybe an api extension that gives you all the actual change blocks is the most efficient way to deliver that.

A

Yeah, we could also add something.

C

A

The css csr driver spec, so that's uh you know some of those so so.

B

When I, when I volume snapshot, object, for example, right, I specify who is the snapshot right and.

A

Yeah, I was just saying that on snapchat you have the same problem today right I'm just saying: yeah.

B

Yeah, so so, should we enhance the api to instead of just specify the volume name we also by the the snapshot or whoever we're move, an act on this differential or snapshot service like a vendor or something like that, can we somehow.

C

Snap snapshot classes have a have a driver name in them, so if you have multiple csi drivers and one of them can do snapshots and one of them can't do snapshots, you would only expect to see a snapshot class for the one that can, and you would know which one it was by looking at the volume snapshot class.

C

So we one could imagine like some top level field on the volume snapshot class, which says supports diff, apis or whatever you want to call it right, and then you could, when you created the volume snapshot class, if you, if you supported it, you would have to assert that fact and then maybe the rest of the stuff only works. If, if that field is set on the volume snapshot class, all snapshots of that class would then be assumed to support this feature.

A

Oh, no, it's not here. It's not here. It's the snapshot class! Oh it's now! I'm I'm.

B

Not in the csi.

A

Call itself yeah.

E

So I think this is a. I think this is a larger general problem that perhaps we need to find a way to actually expose the csi capabilities through kubernetes, somehow right like expose them in the csi driver, uh object or or something like that, because this is an issue with like volume, cloning and there is no snapshot class for that.

A

Yeah yeah we yeah, we do have the same problem with attitudes, so we do use csr driver to solve some of those problems, but there are still you know, other capabilities that we are not really.

C

What exactly happens if you try to clone a volume that can't support cloning, because do you get a useful.

A

Error message: it's.

C

A

C

A

Does it at least.

C

Does it at least generate an event that tells you why it's not binding?

C

A

A

Even should have something right: the pvc have anything or p at least p or pv have any event telling you that well, there should be something well at least you've seen the logs, I'm pretty sure it's in the logs using logs. We check the capabilities.

B

When we create a volume snapshot, we do specify the volume snapshot class name and then the controller will, based on this volume, snapshot class name, to look up the uh for loop snapshotter to perform the snapshot right. Yes,.

F

B

Can we do similar thing here? Can we, when we create yeah.

A

It's actually the same problem, then it's. This is not a new problem, even if you also, if you think about resize right, you know we actually have that like a lot of word expansion, but what? If a user used the wrong uh storage class that you know it's actually not really. This driver actually does not really support it, and you still get this problem. It's going to.

C

Yeah you're right, the the problem of not supporting it at all is is a long-standing one that we need to do something about, but but I'm more worried about the case where you have two different ones that both do support it, and so you have two copies of the service like in. In that situation, an api extension is.

A

Like one, what do you mean in one driver, yeah.

C

I could have two different csi drivers that both support the snapshot, diffing feature and so.

D

C

Going to be two copies of the driver running two copies of the http server that the front end's at you know, there's going to be two copies.

A

It's going to send the request to both. Well, what what I mean! It's not because he's scrolling.

C

I'm saying you have to know you have to know which, which is which right like.

B

There might be a map or service mapper somewhere right on along the data path. There must be a central place where right and.

C

A

Then, if we have that in the snapshot class we add some entry, then yeah yeah. No, I'm.

C

Saying for everything we've done so far, we have side cars. We have built into the kubernetes api everything we need to figure this out. I'm saying the moment you go for like an api extension, you create.

A

A bunch of new problems, I'm not familiar with how api extension works. uh Oh you're, saying that we don't have a way to figure that out. Well,.

C

The api extension itself might need a multiplexer to support multiple csi drivers. Okay, on its back end- and I.

A

Don't know if that's like some open issue, that in phone can you write that down? I think those are the issues we need to look into if we use a api extension how we can resolve this, maybe add a lot of complexity to that api extension server.

B

Yeah we, it sounds like we need a much multi multiplexer to directing the the service request to the specific csi drive. That's supporting this difference of snapshots right for that specific volume, because for each volume you, when you create a snapshot for it, it belongs. That's that for loop and the snapshot belongs to some sorted vendor on the story right and that's only the started vendor that that's that's csi drive is the one that actually will have to do with the differential snapshot.

B

So if we have a way to direct this volume snapshot to that, then the same way we're gonna direct, this differential towards that csi device.

B

um That is, though, that what we need what we right now, what we don't have is is a center like a a a front, end service, or something like that uh to do the search instead of listening to the creation of the kubernetes objects right right now, for the volume snapshot did have a controller, the external snapshot controller, not listen to this volume, snapshot creation.

B

But in our case uh this this is a service. There's no object created to it all we or should we create an object for this and direct it into the external and the api extension server. What do you guys think.

A

Same server should also be able to handle those api objects right. Otherwise, uh what what does it do? I mean I, I would spot it has to handle those uh api objects, but that's the thing we I haven't have never used it. So I'm not really sure how that works. So then, if you yeah, I I'm.

C

Sure there's a way to do it. I just I see a deployment challenge with you know the way csi drivers are packaged today. You know your vendor has a bunch of yaml that you run that spins up some pods and some demon sets and in a way and voila, you have a csi driver, but this is a the deployment.

A

Can you um can you go back to that uh that diagram deployment diagram of the csi driver, so the extension server will be like talking to this cube api service on the right hand, side right so, like my uh well, I'm just.

C

It would be a whole other pod and it would need its own connection to the csi driver. Somehow.

A

Right but I think but but but I would think your let's say this is uh uh well. Let's say we have a stable side of diploma of something that will still be. I mean that should still work the same way as before. I would assume it's just uh you have this. uh You have this other other server right. That server will be communicating with the.

C

Right, I just I don't think if.

A

If we go in api.

C

Extension route, I don't think you can have two of them. You only have one which means. If you have two csi drivers, you need some some multiplexer to stand in front of them.

B

Like a match between.

C

I haven't thought through all the details, I'm just I think it's it's going to look a little different than what we've done before, so it makes sense to like prototype it and show some a way that we could do it.

D

So I could imagine a prototype where we have a csi spec extension that generates the diff in a side car that maybe calls that. But then the question is how to connect the backup software to the sidecar. Yes, you could sort of push that problem into the backup software.

D

But that would not be too friendly in the sense that it would have to figure out what csi driver it's looking at and then and then figure out how it's deployed and where that service might be packaged with its controller. It'd be a little messy.

A

And there's security implications. I think you still need to have the side card static, controller, okay,.

D

Yeah, I I agree, that's what I'm saying the sidecar controller would be packaged with the driver and called the csi extension. But then how does the backup.

E

D

Talk what's the security problem.

C

That I don't know.

D

C

Just just that, today, controller pods, don't listen for for http calls right typically, because because they they're typically run on sensitive nodes because they have sensitive.

D

C

And you don't want them to get hacked, and so by creating a new sidecar that listens, you're, creating an attack vector right into a sensitive piece of software.

G

um Possibly not understanding everything's going on so, given that this interface, we're talking about, is exposed through sort of a grpc connection rather than like a standard kubernetes api. um What if we solved the complexity problem, could we write say a go library that would basically solve all the things that that thomas was talking about um like locating the csi driver container and and open up a connection to it and verifying that it has. You know, snapshot diff capability.

D

Yeah, I would think you could you could write a go client library and you would have to have certificates to do the tls validation to solve the security problem that were known between the backup software and the sidecar, and the driver that handles this.

B

That that security concern is evaluated, but I think we have to figure out the flow here. First right, the data flow between component and what component to be involved first and then we will look into detail on security.

C

Yeah, I think I think the important thing to bear in mind is in a world where you could have two csi drivers in the same cluster. How is it going to work? Something needs to multiplex the calls or there needs to be a way to to find the service for a particular snapshot.

B

Yep, that is the oh.

A

How how does how do you handle this today? Let's say without this: I think we oh. She also know that part right yeah. So in.

B

Right in in, uh like let's say, vmware right vmware, I I just make a call to the the the vm the v3 to get me the difference with the cbt between the two. uh You know snapshot right, so you basically already know.

B

I have to know that I need to know the where's, the address of that vsphere uh service and so on connection credentials, and so I just make a private connection. There.

A

For this, so how do you? How do you decide I mean so if there are multiple csr drivers, how do you know which one is uh like? Do you just look at the snapshot class? Would you decide or how do you uh basically, so you need to actually make calls to the student provider apis right, yeah, so.

B

A

Just look at you actually need to know which one is which so yeah that's part of the configuration.

B

When we create the.

A

Configuration you use that one, so it's already pre pre-configured, it's not like dynamically designed based on the snapshot class or something well when.

B

We, when we, uh when we run the backup right the color, the the color of my service, will providing me uh that information how to connect into that vsphere. Okay, yeah.

H

Can I ask a more fundamental question, so I mean before we delve into implementation. Details like you know, api extension, csi security, all those things. I want to know what problem we're solving if the problem we're solving is basically so. I think that I can talk more in terms of how netapp does things, but um I would like to hear what other vendors are doing in this space.

H

So as far as what netapp is doing, you have like a data replication engine which is you know, our staff mirror apis which are used to move data incrementally efficiently from you know a to b, and we also have a difference engine that's called snapdiff, it's it's only for nas, it's not for block. I understand this proposal is only for blackwatch, but it basically gives you a diff of different files between two two snapshots right.

H

So as far as like the efficient and incremental backup are snapper apis, which are the you know, data replication apis.

H

They do the right thing behind the scene, so there is no need to expose the different blocks to anyone else right so efficiently. We can backup data from point a to point b and only copy the data so that that's that's one capability that we have. I don't know about other vendors, but the second thing, which is about the difference engine you know like you know, but the main use of that is to build a backup catalog right.

H

So, for example, you know this file and that file change between these two snapshots and that catalog can be used, for example, to do selective, restore right.

H

So this kind of makes sense in the case of you, know, nas, but I don't know how that information is going to be used for sand and block protocols like ha ha like so so. Basically, I want to know there are two things like especially um I directed stuff about the vmware integration like right now,.

H

Outside of kubernetes, how are you integrating with different vendors and for what use cases like? Is it mainly for efficient backup or is it look from point a to point b or is it more for building a cadillac which vendors is it only like the system that you have in place? Is it only for vsan or is it or integrated with other vendors? I want to know more about the use case.

H

Does that make sense.

F

We've seen this mostly for efficiency. um You know the other approach to take like block backups is to do. uh We call it differential, meaning you. You have to scan the entire file system first or cross number or volume device um first and then find things that have changed.

F

uh You know using like a new kind of engine which is really really slow on backup, because you have to read everything: if you have apis until you will change, you can backup. Only things have changed.

H

So my question is, you know if, let's say vendor, a has efficient has an api for backup. That already does the right thing by only starting the dirty data is that you need to expose the dirty blocks to true csi.

H

You know look. Maybe we should focus on just having a backup api that I mean hopefully does the right thing behind the scene. I understand that it may not be supported by all vendors, but that's a different route. That's a different alternative. You can't take. Are you asking.

F

The difference between data path apis and like the control plane apis, where you say take it back up. You can figure that your system, with a data sync and.

H

I don't know too much about other vendors and I'm not the backup expert, but I'm just asking you know like if uh if a vendor has a backup api- and it already sends the incremental daily data, is there a need to expose the dirty blocks through csi through some other mechanism to outside? If the vendors api already does the right thing.

A

So adela, I think this is forbaca vendors, who want to support different distorted systems.

H

And they already do that, you know like they already do, that right.

A

But they have to go outside of kubernetes so, but they have to go for different uh vendors apis right. So here we are trying to come up with a common api. That's that's the goal here right. So, of course, if you only support uh one storage system, I guess there's no need, but here there's backup vendor want to be able to backup volumes created by different storage systems.

A

So we don't. If we don't have this, then they just have to go. Call use storage providers, uh individual apis right. It's amazing right and.

H

They're already doing that, you know like for all the.

A

Yeah, but that's not efficient right, so that's that's what we're talking about! That's not efficient, and we want this same as csi. Why do we need a csi at all right? um You can already do that. You just call each vendor api. It's the same same reason. Actually.

H

No, I think, there's a difference here. The the difference is that, let's say if I have a data replication api that already sends blocks efficiently from point a to point b.

H

Okay, so I just say you know back up this volume, that's the api and the backup vendors can call that api to csi. You know they don't necessarily have to know about the change blocks.

H

They just call that api and the right things happen behind the scene by the vendor. So I'm not basically saying we shouldn't support backup through cs. I'm saying we should, but maybe the use case for having an apa for change blocks is less compelling.

A

You are basically saying like uh it's just based on what vendors support that every time you just uh you are uh so that that has to be like incremental every time right so because.

D

A

How do you know uh what is because here we're talking about two two snapshots? uh You know one snap, you know the diffs between two. So if you want to do this one automatically, uh then that will be always incremental. I guess right. This is like. Is that what you mean? Did that get that right.

H

So I mean part of the goal with this is to improve. You know: incremental backup, snapshots, right, incremental backups right.

B

So I I can, I can answer your question in the way that I understand right right now, for example uh in in my mac vendor right, I can backups form uh vmware, um first class disk and also backup form a bunch of csi drives right and uh in order for me to make it efficient backup, I have to talk to vmware vsphere with using their own protocol and for csi right now for other vendors.

B

That's the csi vendor that uh I want to make an efficient backup. There's no way for me to do that. Unless I have to dig out what is this vendor is doing and the I have to call that api for each of the vendor.

B

For example, if in my cluster that I'm backing up there's five different vendors, then I would have to call vmware vendor api and other other sport, and I have to add incrementally uh adding the support for these vendors api into my backup software uh that make my backup software very complicated, and so, instead of doing that, the proposal here is that, can we have a common way that I can make a difference or snapshot instead of calling individual api to get a snapdeal?

B

Can we can I, as a backup vendor, just call csi's differential snapshot and get the different and make the backups efficiently? uh If I only have to support vmware, I'm just happy to call in vmware right. That's that's fine, but if I have to support both netapp, you know vmware and- and that will be a lot more work for me to adding support for them.

B

So that is my real answer, for that is that if I have to support multiple vendors and if they don't have a common way for me to make a difference in snapshot, I will have to eat next. Individual api call for the fund.

D

So another problem is: if you had a common backup api, wouldn't the targets that you could back up to be fixed by the vendor, meaning they might support s3 or they might not uh in this mode. So how would you go about standardizing that and backup vendors often have well in in the case of uh data domain? They have their own backup appliance that stores and dedupes data and things, and I don't see how you would be able to support something like that.

H

I mean what I imagine is something you know like uh how we have snapshots snapshot class. We may have a backup and backup classes, you know, and then, through backup classes, you can specify different targets, or you know things like that.

B

That's actually a good question so whether whether we can, let's just say, backup from one vendor to um the data protection, I mean the the the data store, the backup storage of belong to another vendor, how we can make it.

F

That's a good question. I.

H

F

Both use cases erlang, I see the case where you want to make api calls directly to control the data within the storage provider or other vendor, but I also data path. Apis are also very important um because different vendors have different. You know efficient systems for for collecting, backups and managing them the data, the data path, part of it yeah.

H

I can definitely see I mean benefits to both approaches, but just just being realistic, you know like in the case of netapp, like these apis are proprietary, so you can't really anyone cannot call them.

H

That would include our own csr driver trident, but um I don't know about other vendors, like is everyone exposing these apis tobacco vendors to the whole world, or just to select backup vendors like I want to be like if you're designing it? I agree on this design. I want to know what percentage of vendors are actually getting to support this.

F

Cloud providers expose it and uh you know, vmware exposes it. Certainly.

F

But if you're proposing apis that let us control the replication schemes within in different storage vendors themselves. I think that also makes sense, but I think that's, it's kind of a separate use case.

B

B

Can we mount volume on the uh part right this it just explodes it exposed to us as a fire system or as a block right? Is that common enough for the um the story vendor to like uh should be? You should be based on that to answer the question whether no.

H

I guess you know yeah what I'm proposing is more along with you know. If the storage vendor already supports backup capabilities, then what I'm proposing is more suitable for that you know, but if a storage vendor doesn't support backup capabilities, then we need this. You know the type of integration that you're talking about. So I agree. You know there are use cases for both, but um I want to know like you know, so we talk about vmware support and this hyper skater's supporting this.

H

But I want to know, like you know what other vendors are thinking about this and if you have to make a component case internally for me you know I want to know like you know what other vendors are doing already or what they're planning to do.

F

I think fong put together some uh other apis that expose this right.

A

And there's another there's another dock: a google doc has uh has a list at that point we actually collected.

I

B

I I I have that dog. You want me to show that dog it yeah. Could you show me I couldn't find it then right now yeah. I can stop sharing. Okay, I think.

B

B

It might not be a complete list and, and some the information might not uh be up to date, but that's what we found.

A

Yeah, so so this vmware and then okay, so this is basically vivo right, uh so uh other ones. So you see that the netapp actually wasn't super sensitive. Now that it's actually there's a wee ball. Okay, so all of this list here so.

H

Yeah, so people is something different because you know we've all is not something you support to kubernetes.

A

I'm just saying.

H

That it's possible.

A

For you to support the those change blocks.

H

Now I know we do, but I'm saying.

A

H

This is done through proprietary apis, so it's not something we exposed to the whole world. You know um only select partners are, can access these apis so and.

C

It's not it's not blocks. It's files.

H

It's only files too right yeah. We.

C

Have snipping for things.

H

Exactly we don't have the thing for blocks.

A

All right so an aws.

A

Rbd has something as well.

C

Those would probably all be change blocks, not change files.

A

Yeah change about, because we are talking about this- we're right now, focusing on blocks right. We were actually not really, uh I think that's the end goal. Change of our list is non-go because that's different a little different.

A

uh Is this the other apis? Okay, so I think we listed here there's a google on gcpd and then there's a dell emc. I know they can support us.

B

Yeah, it's not a complete list because there's so many actually.

A

Yeah yeah, so those are the lists we, those are the things that we listed here. So uh this definitely it's not just uh not just like one or two vendors can support this.

A

I think that's that's all we have we captured in the stock, so there should be more.

H

Okay, yeah, I don't want to put anyone on the spot, but basically you know what other vendors are thinking about this it will be like I mean it's wanting to support things through v-wall. You know through proprietary apis and integrations, but it's another thing to support things through open source drivers like or through csi interface,.

H

And you know it would be hard painful if you go through this exercise and nobody really signs up for it.

F

Can anyone consume uh the v-ball interface or is a at least a consumption side, public.

F

You know, could we like make a shim, for example, that would implement this, this interface we're proposing and then consume these all these other storage providers.

A

um But if you use viva, then that's only in the vicious environment right so yeah, it's probably not enough.

B

I I I think, definitely there's an interest in the community, because when we start talking about this cbt common interface back like one or one years ago, many people- many engineering, formed uh at least of the people who have contacted me, including you know, microsoft, google and vmware is interested in having a common uh interface for this differential snapshot.

B

I can see some of them are in this in this meeting today and also they're contributing into the design dock and so on and so forth. So they're. Definitely they are definitely be some interest in the community, but uh we do not have an official. You know commitment from specific vendors or anything there. Yet if that is something that you are after.

I

Sorry so what was the question whether there is interest within the community to support this feature.

I

Is that the question.

B

I

D

B

Was about more like if we have to do a lot of work to to get this in command common interface like this, uh you know, is there any vendors who are interested in such interface um or to make it worthwhile etcetera? That's what I understand.

H

That's right, yeah, basically like yeah the apis.

B

H

We have for diffing, these are proprietary, apis and they're, only accessible to certain partners such as vmware, um and I don't know.

B

About other vendors, you know.

H

But, for example like in the case of you know, file incremental backups, you know having a catalog make sense because, for example, you want to restore individual files in the case of blocks. I would imagine I I don't know I mean I. I don't understand like exactly how that backup catalog would be useful, but for for um cbt for blocks, which is the focus of this proposal, I would imagine the main benefit is really incremental backups right and for incremental.

D

H

Many vendors they already have apis that efficiently. Do that, you don't have to expose those change blocks to a third party like the backup apis already just sends copies of their dirty data. You know to the backup destination.

H

I

Why I'm questioning.

H

I

Okay, yeah, so I mean so in terms of from customers themselves right. There is a lot of. uh I mean interest that we see from the different customers we are talking, but now question is: can we standardize? Now I mean as goal of this working group, um can we standardize it through the csi rate as a csi driver, providing this through a site or something providing this functionality so for the backup vendors to consume it? So that was the goal.

B

Just want to have a time here we have 10 minutes left on the meeting.

D

Yeah uh as someone working on csi drivers, I have sort of the opposite question, which is how many different backup vendors would be interested in using the interface. Clearly, we at dell are going to support dell's, backup needs and if it has to be proprietary, then so be it.

D

But um I think the the question that uh you know fong and sikarna are asking is: uh could we make it more open where the backup software could be more portable across different types of systems and different different customers could choose different backup software and have it work with their systems? I think it's a reasonable question.

H

Yeah I mean both backup vendors and storage vendors. They have to kind of um agree on. You know what they're willing to do.

A

Yeah, so we can actually collect some more feedback. Okay, we well. We already have something in that document. We already have uh different uh vendors participating and interested in that, but we can actually we can maybe get more, uh so we can actually maybe bring this up in the sixth story meeting tomorrow and just to you know, just ask vendors to uh tell me as well but from this working group, because we have uh many backup vendors in this scrolling group right.

A

um Maybe you guys uh also want to chime in and see if you are interested. So let me actually talk to uh shanghai, apply and see. uh How do we want to collect feedback? So we know that we have that document that has listed a few companies.

A

Let me let me sync up with him first and then see what we want to do with this, but I think tomorrow I will bring this up in the 640 meeting.

J

If I can have one thing as a backup vendor, you know, these type of apis also enable uh us to overcome same network and topology challenges.

J

uh You know which I imposed by you, know more innovative ways of deploying clusters which might hide a lot of network access, so we're even in situations where we have all the vendor apis. We can't use them because of the network challenges imposed by the topology.

J

So interfaces like this right, you know people who create those types of environments, kind of incumbent on them to provide such interfaces so as to enable uh you know use of the of real software in those environments. So these interfaces would help a lot.

A

So you've been saying that we have this common interface so that it's easier, you don't have to figure out like how to get it to work. If there's any like uh vendors have some special advanced api that may not work right when you aren't right.

J

I mean case in point: right is vmware's uh tanzu s right. The the the network topology will not allow access uh to the guest clusters right and it's hidden because.

A

That that cannot be taught by this. That's different. If you had that, that's different, that's that's! Really different! Okay, we yeah that's like a different cluster when it is a classic deployment. It's not not right when we had. If you had the differential blocks, map.

J

Right in the in the cluster, uh one could potentially get to the blocks.

A

J

A

This this does not address that point you still need to, but I think we are on the topic. If you want to talk more about that, we can talk.

I

About that offline, but.

A

I

Perfect, I think.

A

I

Going by the technology right so vmware vm is the platform. There is a clear defined way to take uh snapshot, differential, cbt and all so now. If you come to kubernetes as a platform, there is no way. So if we can, as a community standardize that, then vendors can support it, and I mean the storage vendors can support it and backup vendors can consume.

D

B

Sorry guys we only have five minutes left, so I want to summarize what we got so far and then we might continue. We might have to continue with the next time yeah.

A

Can you uh uh share your song.

B

Again, yeah, actually, I haven't. I haven't uh write down anything on the the um the document for for the cap yet because I don't think this, this is a mature time to write kept. Yet what I want to got so far is that we we we are discussed about having like a fun and multiplex that able to uh for the client can talk to and then that multiplexer can directing the traffic to the specific um container or sidecar that actually serving the different snapshot right. We needing that piece.

B

I think that is one piece that I I think we agree on today that we need that multiplexer that able to look up whether that uh volumes have been mapped to what vendors and and if that vendor supports snapshot, differential snapshot or not that that multiplexing is needed right and another thing that we need information for is the we need to talk to. We need to get more information about the api extension server so that we can see whether uh we can use.

B

uh We can serve that large um requests of multi many many thousands of uh mesa that are blocked without corrupting the the the the kubernetes api server. So that is something that we have to go back and discuss more and dig out more, and I also um see a lot of uh question or some question related to whether we should have a common um storage api so that, when this kind of thing available of this kind of differential snapshot, api service is available.

B

How we can leverage that in the common storage api. So that is something that I don't think we can answer immediately. I also see a group of questions related whether uh this api would be this. This service would be uh be an interest for different vendors, so I think uh ziggs have mentioned that we need to go back and collect for more vendors to see if their interest. So that is what I got on the note. So far, do I mean anything.

B

I think that's it. Okay, I would have to go back into this video and and make a. uh I will walk through the list. uh Make sure that I did. I got everything, but it looked like. um Let me do some more research on this multi black component and I will try to propose something I might have to talk to some of you in private, because I might not have all the expertise in this area, so I might need your guy help from there.

B

A

Okay yeah, uh can you uh yeah thanks? Funk you uh stop sharing, I oh you stopped, uh so I I think, uh okay, so I actually uh added this one right. uh So, if you can, you know, add your company's name. Who are you know who are interested in using this uh cbt api?

A

uh I mean that that's that's one way of collecting this yeah, so I'll put them there, and I know I will okay leave sean chang's interest of this food yeah.

B

Emc definitely.

A

As a backup uh vendor uh this ad, you are interested in that as well right from microsoft,.

A

G

A

G

A

Okay, I'll just add.

A

uh Is jung you're? I think red hat is interested because you have been attending those meetings.

E

Red hat's, interesting.

A

So, uh who else, please feel free to add your company name? There cassandra says tom.

E

We are oh yeah.

B

All right yeah, I mean this is just like an interest. There's no commitment, yeah, yeah, yeah,.

A

It's fine yeah, that's fine!

A

As long as we got enough support, we can make progress yeah. uh So, okay, I think we're already at top of the hour. uh So uh you know, if you you know, other people are interested. Please uh add your company's name here. I will bring this up again in tomorrow's sixth gorgeous meeting as well.

A

J

Thank you very much.

A

Okay, uh that's all for today, thanks bye.