Kubernetes Data Protection Working Group, 30 Jun 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes Data Protection WG Bi-Weekly Meeting 20210630

Description

Kubernetes Data Protection WG Bi-Weekly Meeting - 30 June 2021

Meeting Notes/Agenda: -

Find out more about the Kubernetes Data Protection WG here: https://github.com/kubernetes/community/tree/master/wg-data-protection

Moderator: Xiangqian Yu (Google)

A

Okay good morning, everyone today is wednesday june 30 2021. This is a data protection working group meeting. uh We don't have a lot of topics today, mainly we have some leftover items from last meeting talking about worm snapshot.

A

She and I feel uh maybe we can utilize. This change opportunity to kind of you know, explain a little bit. What are the difference between the discuss, the local and the remote snapshots in the current csi implementation and another potential possibility we'll discuss a lot in last million is to introduce a a first party field that allows csi providers to upload the snapshots to uh to some remote backup repository uh shin, and I had some discussion over that. I will share with the group uh what we were thinking and the next major item is.

A

We feel the we have a lot of good materials on the white paper for with regarding data protection in kubernetes context, uh there are still couple of sections needs a little bit more updates or editing, but I think in general, it's in a fairly good shape. uh This has been a very long and long process and a lot of effort. Thank you.

A

Everyone who have worked on this xin and I plan to take a pass on the existing white paper, write ups and we, the goal, is to with what whatever materials we have as of the day, to publish our first version.

A

uh We are planning to do that by july and and then after this agenda, we were going to open the discussion. uh If there's any of the issues, anyone wants to discuss uh shin. Do you mind sharing the slides.

B

Yeah sure, can you stop sharing first.

A

All right, you got.

B

B

Okay, so this is just some follow-up questions from the previous meeting. uh I think when tom was presenting uh the options for some cloud services backup and then I think those questions came up so uh we're just following up with those the first one came up is uh the difference between different snapshot, semantics that are supported by different story vendors.

B

So I just want to make sure everybody is clear with this, and so there are two types of uh snapshot in semantics. The first one is a local snapshot, meaning when you create a snapshot, the snapshot will be placed on the same source system as your primary volume.

B

So this is the the first one called local snapshot and then the second one uh we call it either remote or durable snapshot. I mean that's just name, but what that means is after snapshot is cut. It will also be uploaded to a remote location, could be an object store in the cloud, so storage systems supports either local or remote snapshot today, and because of these differences in the css back, we actually have this uh ready-to-use field to indicate whether the snapshot is ready to use.

B

So if you, you are a solid system, only take a local snapshot, then normally after snapshot is cut, it's immediately ready to use. So you set that flag to true, but if this needs to be uploaded, then normally after you create a snapshot, you set that flag to force waiting for the upload to complete and then after the upload is complete. Then you set that ready to use flag to true. Then you can use that snapshot. uh So you know there are pros and cons of each, but this is just a you know.

B

Different semantics, um both are supported. um The duplo snapshot is a name suggested, so it's actually. This actually becomes a backup now, because this is actually um uploaded somewhere right. So that's the difference. I just want to make sure, because there are some questions from previous meeting. um Are there any questions? If not, we can just move on next.

B

All right so seems to be clear. um I so another thing is this also came up in the previous meeting.

B

I think there was some suggestions in previous meeting that maybe we could have a parameter in to indicate that the snapshot is local, all remotes. You know somehow kind of uh specify something so that we know whether this snapshot can be uploaded somewhere. um So I think shenzhen- and I talked about that- we were thinking.

B

Should that be a first class field, because in last meeting I thought there seems to be a lot of interest in the last meeting, but after we talked about it it's this is just a it's actually very complicated, because um I think different drivers or different source systems handle this differently.

B

um It's actually pretty hard to come up with a first class field, type of design that meet all the requirements right, so the backup repository could be. It could be an object stored it could it can be nfs. It could be something else.

B

um So so, after talking about that, we thought it's still better just to just to keep this under the the parameters which is uh which is still a quick field, so vendor can can specify that I think I believe some cf drivers already do that. um So this way you know uh the see the driver can have the freedom of choose whatever it wants to do um so this could be.

B

I mean this really depends on uh csi driver what it wants to do right so this that was I just this just came to mind, and I didn't really think through about this. I was just thinking how to um connect this with this uh cozy design. So I was thinking, maybe it's possible. We can specify a bucket access request and then somehow connect this back up so specifying this uh saying this will be uh we'll be using a cozy bucket as a as a backup repository. So this is some possibility here.

B

I actually pause here and see.

C

So sure so this expects the csi driver to be the one moving. The data to.

B

uh I think if we do yeah, if we do the in this way, then it's going to be to see the driver, because this will be this will be in this will be in here right now: volume snapshot class uh but but the backup event. I was wondering okay, so if that's safe for backup vendor only uses, does that work.

C

No so yeah! I was just going to ask that. So if the backup vendor wants to write this snapshot to another secondary storage, um okay,.

B

Yeah, can you use this, we can think about it. Can you use this field as well? uh It's.

C

Actually possible.

B

I think it's just depends, but then, but then I think uh the tricky thing, of course, since this is opaque right, I think it depends. Then that depends on the driver right. So we don't really have a it's, not a first class field, um so you have to know how the driver behaves. I guess that's my that's my question. I think right. So um this is if it's the first class field, then of course you can probably depend on that.

B

I think then everybody uh will be using the the field the same way, but this is the. This is the on the parameter, so it's opaque field, so the driver can decide what to do with it. um So if, let's say when a backup, let's say if a backup software wanted to use this right so because the backup driver will also a backup vendor will also be taking a snapshot.

B

uh Let's say: if you specify this a special parameter in there, um you can actually you can probably use that right. Can you yeah? You can actually you so if, if the backup software is the one who is uh creating the snapshot, so you know what is in the snapshot class right.

C

So, in that case, uh that means this doesn't help in any way, um so the backup driver, I mean the backup and the software already knows that a backup repository um right.

B

I don't need that yeah.

C

Yeah, it's true.

B

I think this just came out, I wanna is. It is ben here today.

D

Yeah, so I'm a little late shane, but I'm here now.

B

Okay, because I remember you were here when we were talking about this uh in last meeting- uh so that's probably maybe that is for a driver's point of view, because I think you mentioned that you are doing that in your driver. Maybe um is that.

D

B

Okay, so is that just some.

D

No, any driver can in principle, do whatever it wants with snapshots right. The only contract that that both csi and kubernetes provide is that we're going to give you a handle to something that you can create new volumes from um and whether that's going to be fast or how that's going to work is, is undefined so the the driver, if it wants to use an object, store it can, if wants, to use a local snapshot it can, if it wants to do both it can.

D

If it wants to do something amazingly sophisticated it can as long as it can meet the contract that the handle that we give back can be used to create new volumes. That's the only guarantee that kubernetes provides, which is nice, because it makes it very easy for lots of different snapshot implementations to be compliant, but um but it's it's not nice. If you want a specific behavior out of snapshots um so yeah, we have at least one prototype that if you take a snapshot, your volume gets shoved into an object store um that.

B

Is through this, so the bonus natural class right, I assume that's probably the place where you can actually.

D

B

What type of information do you put it? It's like like the bucket name or something, or what type of information.

A

I assume it's secret as well.

B

A

Mission secret.

B

D

I mean you could use information that was part of the the snapshot class. You could use information that was part of the snapshot. You could use something proprietary, I mean again it this guy's yeah. It's.

B

All different yeah, that's, I think, that's the tricky part, that's what I was talking about is it's like it's like impossible to come up with something coming, uh because we so.

D

It's impossible to to because the the contract for snapshots is so loose yeah. I don't think we can ever take snapshots themselves and make them better, but you could create something new that was snapshot like that, had a tighter contract and said, and then you have a special kind of snapshot that has these specific requirements.

A

uh Ben, are you saying the contract? How what do you mean exactly by tightening the traffic contract? I mean.

D

The existing csi spec only has only says two things about snapshots. It says you can create well, not not in counting like listing and deleting, but you can create them from volumes and you can create volumes from them and everything else is left undefined right. You also have to be able to list them and delete them, but those are kind of not relevant, like people have asked like well, I'd like to be able to revert a volume back to a snapshot.

D

B

That's a different api, that's not. I mean yeah.

D

B

That's something that we can support right, that's actually. We actually have that.

D

B

D

Asked for it, but like the.

B

Problem is no no that that's a possibility, but that's like another feature, so yeah.

E

Don't look at that capability.

B

Separate the api, for that I mean we need something else. It's not.

D

Yeah yeah: it's not.

B

D

The existing snapshot capability- all it says, is you can create snapshots from volumes. You can create volumes from snapshots and the details of that are entirely undetermined and so, like yeah.

B

But that part is fine. I think the I thought you were trying to say is. I think that I think the challenging challenge right now is probably because there are two different uh semantics that that I mentioned earlier. So you don't know uh when you take a snapshot in it could be a local. It could be durable one. You don't know right so then you actually have to depend on. You actually have to know what storage that is. So you know that what type you're getting then you have to handle them differently.

B

I guess that's the I can see that's one.

D

Challenge you have to have some some external knowledge from from, above that, what is going on.

A

um But yeah, I think that's expected right. Whoever use the storage system, they need to understand or exactly the api they're using right.

B

I'm just saying, I think he's that's probably he was saying that it's not like tight. If it's tight, then that then the snapshot will always be local over always always remote. Something like that.

D

B

But it's not possible because we are supporting you know so many different storage systems. They behave differently. So you can't say: oh a cloud provider, but you cannot just upload yourself. That's just not going to work. You know from the beginning right. So that's the yeah. So that's the challenge is because we are supporting a wide range of storage systems. So that's the challenge of designing commedia.

B

So that's we are well. We are today, um but.

D

But like, like, you were saying about, like a snapshot, revert new feature which would have a new yeah.

B

That would be the difference that that would be like.

D

uh Basically, an object, bucket snapshot, feature.

E

D

Was you know, drivers could assert or not assert that said that, like if they have this capability, then they're going to provide certain guarantees around how their snapshots are created and then we could. We could design.

B

Words differently: that's a totally different feature that that can be added on top so but yeah it has that's a different one. So that's something that we actually have that in the beginning of the design, I think somewhere in the design dock. We have that same.

D

It's debatable whether that's even the right way to get to like uh some sort of a common backup api. I mean yes, it's something that can be done. I'm.

B

Not we're not saying that has to be part of the backup, api right and just, but also it's not probably not. Every sources can support that. Some can support some kind of right. It's an advanced feature.

C

So in that case shouldn't we separate this local snapshot from durable snapshots. So here, if in this design, then I mean uh it's the csi driver right that is now doing the backup as well to secondary storage wow. So then this doesn't account for scenarios where a backup vendor may want to back up to a different storage.

C

uh So the original plan was like.

B

So we are still right right now, we're just brainstorming. There's no.

C

B

No proposal here right because I think this came up in the last meeting, so I heard a lot of voices saying. Oh, we need something, that's why. But then, when we uh when the shantae- and I talk about this- I'm like okay, it's not what we thought yeah I uh this is just because this came out a lot in that spin. So I thought we can talk about this so and see what everybody is thinking right.

B

So this is, I think this is more like a csi driver itself wants to support this, and then it has once you have a way of saying whether this is the local or whether it has want you want this to be uploaded somewhere. I think.

D

Yeah and nothing, nothing stops any vendor from doing something. Proprietary like this today, yeah.

B

This is already there I yeah I actually already implemented. I mean we did this in my previous project already uh so so this is, you know something. Of course, any driver can do already. It's not.

D

B

D

Think I think the goal of this group is to hopefully come up with something that can be supported by a plurality of of different implementations, because because there's value in having a common way of doing things, so that users don't have to have specific knowledge of the underlying storage and they can just say. Okay, the api gives me the following guarantees.

D

I'm going to take.

C

D

Of that, I hope we can arrive at some some common implementation. That's right.

B

I think the challenge is the challenge is not all study systems for local snapshot and not all supports it's durable snapshots right so that I mean so when you're saying you just have this one interface, then you know everything without any extra knowledge. I mean that's probably hard, just just look at those differences already hard, uh unless maybe I don't know if, if we can tell so if somehow, let's say in the snapshot status, if you somehow can tell if it's a local remote does that help? I'm not sure if that helped at all.

C

So, in order to support uh durable, I mean local snapshot has to be supported. First, and I mean the storage system.

B

Some student system does not support local at all right. A lot of cloud.

F

Providers doesn't support.

B

F

um So just come to a couple of basic questions, one and try to understand the context of the discussion a little bit. I think I missed last meeting. um So is the intent here of this discussion to come up with a way to change the the backing store for snapshots or come up with a way to specify a backup location for a snapshot so that it exists both as the native snapshot and a backup in some other repository.

B

uh So I think there are some csr drivers uh actually has done this already uh so originally right. Let's say this is that the the three system only supports local snapshot, but now with this, it actually passed something in the parameters so that the snapshot actually will be uploaded somewhere.

B

So this is something I know some drivers already do that, but I think the.

F

Like my question is: is that that that the thing that gets uploaded does? Does that then become the backing store for the snapshot is represented through kubernetes, or is that just an alternate copy somewhere else?.

B

uh It'd still be, I think in that case, I think it also depends on your invitation, but I think mostly that were all it's just. It looks the same as a snapshot. It's just now. It's actually coming to another place, so maybe then you can, since you.

A

B

Already done that.

A

I think uh to answer your question to the end user of the snapshot feature. It doesn't matter how underlying storage system handles that to you, you just need the handle and that handle is available for you to use to restore a wire right. That's it right. uh It should be transparent to you.

F

Sure I understand that I'm just trying to understand sort of you know mapping in the context of like. If your volume is already in cloud, uh it is remote and durable. um Then.

B

You don't need that yeah, yeah you're right so, okay, you're talking about like, for example, as you write, is that already uploaded is that the upload that will be uploaded to object store right. uh If that's the case. Yes, I think this is not needed. I think if it's already uploaded, I think this is more for uh the story. Systems that are traditionally uh needing local snapshot, not uploading.

B

Okay, so I think I think now it's big. I think it's because now this uh our css back supports different semantics. I think there's a I think, so people actually look at the different type of snapshot and and trying to figure out what's the best user experience. So uh especially, uh you know if a user are using multiple different types right. So if users say the user starts to use uh cloud provider snapshot, then maybe this user only know this type. So then we'll always expect those to be durable.

B

But then, uh when you start to use the different uh solid system, which only has a local snapshot, then then he or she may be surprised. So I think there is some learning curve here as well. um It's just they're different. So I think there are some sort of systems that only support local snapshot are also looking at.

B

Maybe it is possible to upload it somewhere so making it durable. So that's there's some advantage of doing that, but I think of course there are pros and cons right. So I think so let.

G

Me let me share some context or point of view here and see if that helps. It's just my opinion on understanding how storage works and backup vendor uses it. Usually, in majority of cases whenever a storage system has exposed a primitive, the primitives are basically taking a snapshot which is hydration and restoring the snapshot. Refrigeration.

G

That is done in a very local space or storage area, where the main storage is. This is for various reasons right.

G

At the same time, the process of durable or remote snapshot comes into picture more from providing that enterprise guarantee or safeguarding your snapshots. If it isn't local, it is possible that, along with the data source, it can be compromised, deleted, misused, etc.

G

So for durability, safekeeping, the backup data snapshot from the local storage is replicated elsewhere in a remote, offsite location, it's a secondary copy majority of cases. This is how it works. It might be a counter condition where some storage does not have local snapshot at all. It directly does it in remote. Maybe that's a very corner case. Secondly, going back to your third slide, usually, I would want to look at backup vendor decide where the backup repository should be how it should be and how they are going to copy it after the storage system.

G

Does the job when I say storage system does the job the csi driver takes the snapshot and says hey I have successfully completed, then the backup vendor takes that local snapshot and not do anything that he or she wants to do and push it there to the repository.

G

If we look at this context, what I have shared the separation of responsibility, if we apply that, I would want the storage system not to worry about backup repository where it should be copied, etc, because that job is done by a separate software company or tool called backup. Software and the storage software can belong to company a company b, backup software can belong to company, xyz and and xyz decides how they will copy it to the secondary location, remote location right, that's the separation of duty.

G

You might also see some storage company or software also does copying of backup to secondary location. That is because such storage systems as part of storage software, they are selling value addition to their enterprise customers. Saying hey, you know what without storage system you get cross region or cross zonal replication of your data when you take snapshots something of the sort right? That's why some storage system also add a let's say a subset of backup capabilities in there inbuilt, but that's it. They are just building a backup capability on top of their storage software.

G

This is my industry point of view on how customers use backup versus storage and, in my point of view, if you would want to consider the backup responsibility need not be decided by the storage in the csi driver.

D

So so krishna, the the points you raised were the same ones I was going to raise. I think we have to look at it from the perspective of like what what are the users of kubernetes not getting today and and one thing is, it would be nice to know for a given snapshot like.

D

Is it local or is it remote so that if you're trying to implement backups on top of snapshots, you know if, if you have any work to do or not right, because if the snapshot's already removed, maybe there's no more work to do?

D

But if it's local, then yeah to make it durable, you would want to copy it somewhere uh and then the other half of uh what he mentioned is is if, if the storage system does have the ability to like push the snapshot to some object, store somewhere like you might want to be able to control which one um you know because object stores are, are many and plentiful.

D

A

ah Ben, if you meant that, then then uh the worst uh proposal to add a status field, one status field in the snapshot, content interface, to say that uh yeah reason it was healed because it needs csi support. Basically, the csi driver implementers needs to re return, that field back to the corner. Yeah yeah.

B

I'm sorry, I think I missed that. What field is that.

A

Is the field telling whether this snapshot is local versus remote.

B

Oh yeah, I think we were. We were just talking about that right. So uh what was it.

D

B

We don't want that. What.

D

B

Not it would require.

D

A csi spec change, a new capability and a bunch of stuff.

B

Yeah, no! No, that that's not what was the reason that we don't want to do that. Just I mean, in addition to it's a new field.

A

Sorry, it's because.

B

A

The csr driver providers need to support that field, and then we don't know the default behavior. uh Oh you're saying it. The first.

B

But normally when you add a new well normally, when you add a new field, then it has to be a pointer right. So you don't have to set that. Then you don't know yeah.

D

You could have an unknown value, that's.

B

Actually, that's this is actually possible. I think that's, but I mean is there? What is there a value I was just. I don't know if there's a value. Actually I was just thinking about it.

D

We know we're putting our finger on a gap on a this is a gap in the existing design that, because the existing design is so flexible, you don't give any information to the higher level user about what's going on when a snapshot is taken and that's but.

B

I mean I'm just saying that there is a possibility of adding that field, but how does that help I mean? Does it because it.

A

Helps it helps backup window or users understand whether there's anything left to be done if they're doing if the product, but.

B

Then I think they still need to know which is free system. That is right now, do you have any uh yeah.

C

So this I mean the field itself doesn't help right. I mean this is.

B

C

For like, if the storage like what krishna said, if you want to add the storage vendor, wants to add more value and provide and upload the ability as well, because.

B

Then backup software student does not know like because it depends on which storage system it is. It has to do things differently. Depending I mean based on that flag. I guess I mean well.

A

Yeah, so if we responsibility anyways, I I don't. I can't imagine the place where backup when they don't understand the storage system and wants to do the backup right.

B

But I mean: does this additional information help? I guess that's my question. It doesn't help, then. Why do we need to provide it right? We're talking about whether does it add any value by adding this status field?

B

It looks like the answer. Doesn't.

D

Help everyone, but it helps the people who are concerned about their data durability right.

B

That's it like a user, you mean, and a user or check. This looks like a backup. Vendor probably don't need this. It's like they can't really do anything based on this flag so and the user will check the slack you mean, but that's in the content.

B

If, if any user check, it then has to be in the women's snapshot itself right, oh just I don't know.

D

If your high level goal is to make sure that all of your data is backed up to durable storage, you could just go, take a bunch of snapshots and for the ones that were local back them up and for the ones that were already in the cloud not back them up. But.

B

You're not talking about it, you talk about a user. Do this uh manually. What do you mean just to like you? Could.

D

Have some data.

B

Normally, you would use the software right, the software, then I'm just saying that's what I was asking is it then backup vendor should figure that out it's not going to ask user input for that right. It should already figure out like what what this is and also.

D

There's a question of whether we can have the separation of concerns that krishna was was pushing was was pointing out, you know, can we have you know a backup software that can work with arbitrary storage implementations and can we have you know, storage implementations that can work with arbitrary backup software? It's like. If the answer is no, then what are we doing?

D

There's no point in having this working group. If the answer is no, I think the answer is yes, we can have a separation of concerns, but that requires defining something in the middle, which is the common language.

G

A

We try and understand.

G

If I try and understand what ben is saying here is my point of view ben, you can correct me if I'm wrong, we are taking snapshots. These snapshots are supposed to be in a way backups for for for durability, reason or whatever right.

G

There needs to be a way to describe uh the readiness of the backups and more describe about the backup in itself. Let's say one of the backup last sunday, backup is a sunday recovery point right. That recovery point should be descriptive in nature. Where is it located? Is it located in local store? Is it located in a remote storage?

G

uh Is the remote copy completed or not? I think uh the description of the recovery point a past backup is what we are trying to arrive at here. Is that uh right way to understand them.

B

But that's the backup.

A

C

Right now we are actually.

B

He's talking about the snapshot.

C

So I mean so, the question was asked right. So what is it that we are missing in kubernetes, that's available, say for other um kind of platforms? um So uh if you see here so I'll, just take like on-prem like a vmware vendor right, so you have a snapshot functionality and then there is another functionality to read the snapshot and then, um in addition to that, there is uh optimizations around um taking incremental snapshots right with jcbt.

C

So that is what is missing here. Where vcsi spec gives us the. There is no data path, functionality there right where you can actually read snapshots or add cbt functionality and read only the change blocks. That is what we are missing. um So what if the storage vendor wants to upload, I mean that's already there right, I mean what you were saying: the parameters is there those, um but what we are missing is those constructs.

B

Okay, so you think that we actually really does not have to worry about too much of others, but instead we should really.

C

I mean depends on what the goal of this group.

B

Is actually yeah cbt is actually that's yeah. That's uh is it's a function yeah we. Actually we do have that one right, but it's I think we have to have not to come back to that for quite some time. um Okay, so maybe maybe we should, uh then you should. Maybe we should wrap up this one. This is just because we there were some discussions in the last in the last meeting. So that's why this topic is here.

B

So we don't have to, we don't have to uh continue talking about this one. I think. Maybe we can move to the move to the next.

B

Because even this one, when we talk about here, this is it's. You know this is under parameters. Then uh vendor can already do that. We are not really, you know, we can't stop lender from doing anything. This is opaque right, so that's the purposes and vendor can uh interpret those.

B

So that's already there uh and then, since we're not going to propose any first class field, then I think we don't really have to uh talk about this, I think or unless, if somebody out anyone else want to continue to stop with this topic. Otherwise we can maybe move on to the next yeah.

A

If there's any more uh discussions, uh uh pre uh shin, can we maybe create a slack channel for that and then whoever wants to.

B

Well, do we we have that data for anyone, group, selection,.

B

uh Questions want to continue with this topic. Yeah. We can wrap this up yeah. I think this. The only reason we brought this up is this came up in the last meeting, so uh maybe there's some confusion, so I think looks like at least right now I mean we don't really have to keep talking about this. Okay, so I'll, just stop sharing, maybe go back to shenzhen. You want to continue yeah.

A

Yeah, I don't have too much uh the as I mentioned the white paper. Thank you all for the hard working the white paper. At least I personally went through. I think we have fairly good amount of data over there, and most of them are really well written, so shenanigans might, um or the idea is to kind of you know, uh take this one step further.

A

Instead of waiting on some of the sections to be done, we just publish a first version so likely for those who own a particular session or have worked on a particular session.

A

You will hear from us too very soon with regarding to that and then once we done that path uh by the july and we're looking to publish into the community and before we do that, of course, we will, you know, sit with sit together with this group and uh go through the mainstreams we're not going to go through in very detail, but at least the outlines, et cetera, et cetera, to see to collect final rounds of feedback.

A

uh If everything looks good, then we are looking to publish that document.

A

Questions: okay, uh we got uh 18 minutes left any open this or any anyone want to discuss anything.

A

B

So can we just maybe go back to the.

A

Oh, the slides.

B

Know that the the agenda dog uh I I can't, I think I want to check and see if a fan has a tiny update, because last time.

B

The cvt section uh I don't know if I have you, got a chance to update.

G

Yeah, I I have updated the I look into the map lift and I update the document with a possible flow that you know match with the snap lift. I have not looked at the rustic yet so I, uh but I think the the the the example that we have for the um cbt.

G

uh It's it's the the workflow that conversivity it should be able to cover the file system as well, because it is the same. This is very much the same way. uh The only difference is when we're backing up with that, whether we backup a block by block or five by five, so that is the difference there.

G

But I again I have to look into the restrict to know the detail.

A

Okay, okay, thank you, wong uh again shin and I will be reaching out to um those who working on specific sections and do a final pass very shortly. Okay, any other questions going once twice all right. Everybody thank you have a good day.

B

Thank you, bye.

F