Kubernetes Storage SIG Miscellaneous, 12 Jul 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Storage - PVC/VolumeSnapshot Namespace Transfer Design Meeting 20210712

Description

Kubernetes Storage Special-Interest-Group (SIG) PVC/VolumeSnapshot Namespace Transfer Design Meeting - 12 July 2021

Meeting Notes/Agenda: -

Find out more about the Storage SIG here: https://github.com/kubernetes/community/tree/master/sig-storage

A

uh Okay, so uh uh this meeting is about the pvc volume snapshot next space transfer, uh the initial design was started by john griffiths. Here. Excuse.

B

Me, can you make it a little bigger seems to be pretty small sure.

A

B

Yeah yeah perfect.

A

Okay, uh so just I put the bullet points here to summarize, what was finding you? Can you have access to the document? There is a link for the kit, so basically everything will be to allow the bbc's to be between transfer between name spaces, so the pve will stay the same, but we will do transfers for the pvcs uh they would.

A

He proposed accept an approved mechanism, uh and this will be based like back in the day, like I think, two years ago it will be annotation in the pvcs, and this annotation will speak out the target volume that will be used, so it will initially was targeting the volume clone and volume snapshot and the original pvc will be deleted uh before transferring so before you start the transferring uh you will delete the pvc and then sorry it will start to transfer and then, before the the target pvc can use the original pv as the original pvc has to be deleted, uh also the reclaimed policy of the pv.

A

If it was like a delete contamination, it has to be overwrite already to keep the pvc to be a pv uh yeah, and this was a first attempt and there were many discussions happening and then this was closed and then another uh attempted by michael hendrickson.

A

uh He started here and he proposed uh uh another crd, which would be storage, transfer, request and storage transfer approval and uh and then there will be a controller which will detect uh like if there is a match and once there is a match, the transfer will be initiated uh and again the pv as the pv will be updated. Accordingly, the source pvc will be deleted and the target one will be created.

A

He also commented that this instead of having these changes in the api itself, we can have crds, which has a storage transfer, request and storage transfer approval, and then we will have external controller instead of editing the pv c controller entry. You have external controller to take care of this uh transfer request approval and it will maintain the conditions and open the condition uh will be matched it will initiate a transfer.

A

Just I have the link here, so you can have a better idea of yes, a better idea. How this can you see uh the github link or.

B

Yeah we can see your screen.

A

Okay, so this is a crd for storage transfer request. It will have the namespace the target namespace and the target name, and also the source uh namespace that's coming from the pvc. uh Sorry, the transfer will happen uh and there will be approval. That's happening from that uh from the other side from the namespace that will be receiving the pvc, and this is a crd for the storage transfer approval and it will have also a similar similar to the crd of the exploration request. But of course we will have.

A

You know we'll swap the approvals the target name, space and source name space, uh the condition that the transfer will be initiated open are listed here. So he assumes that r refers to request and a refers to approval, and once these conditions are matched, then we can go ahead and start transferring.

A

uh Also, he emphasized that once the matching is complete and we start transcript process then before before we start transfer or one of the things that we have to do is to change the uh claim policy of the pv to retain. So we make sure it will not be gone by when the original or the source name. Pvc will be deleted.

A

uh We will delete the pvc as a source one, and then we will bind the pv to the target. Pvc, sorry yeah to the name space and then we'll create the pvc which will have the same respects of the original one and how we bind to the pv and then, after that we can uh change back the claim policy to the original one.

A

These are also like ghost trucks for the prototypes. I think it's the same as the crds.

A

He also yeah, he proposes stories, uh but it's the same as the original kit, uh but this is our very things are important. I think I should uh emphasize on that. He also emphasized that these things can be crds instead of entry uh like which is a api's entry of the community itself.

A

We can have these crds and external controller. Take care of this thing uh exactly. Instead of doing the implementation of entry.

B

Execute what does that mean? Is the cap really required? I think this definitely need a cab right. I'm confused this question.

C

B

A

C

Yeah, so I I um this is mike, I a couple years ago or a year and a half ago, um yeah, I I um I guess I just wasn't and I'm still not, I'm maybe entirely sure what requires a cap and what doesn't and if there are no changes to like kubernetes kubernetes it does. There have to be a cup.

B

Yes, yes, we do to cap.

D

C

D

uh Yeah, if something is like a project, that's going to be like supported and sponsored by the sig. It's still good to have a cap for it.

C

A

uh Okay, so there are also some questions uh about what how to deal with the current cases. Yeah.

E

Just asking a quick question on this one: uh this is I like this style uh just this is luis and my question is um on the crs uh the approval one. uh Is it a uh a namespaced uh object? Did I miss that part yeah they're, both namespaced okay?

E

So does that mean that the um let's say a person's q config only allows uh to store uh on a certain name space as the target, and then I say, okay, I approve for uh a request coming in from someone to to move a volume to my namespace.

E

That's the way we do it. Yeah.

C

Yeah um so uh so I started working on something uh around these way of red hat and the way that I started it was. I had a cluster scope resource where you could just define everything and then I layered uh name space scope resources. On top of that I mean that's just an implementation detail, but I think ultimately um you want the name space.

E

Yeah definitely yeah it's easier to see. If the person owns both namespaces, they can move them.

F

E

Okay, but okay, I just want to make sure it was name-spaced. That's the key.

A

uh Okay, so this is uh just test plan and uh there was no code uh correct me for if I'm wrong, it's good that you are here. There was no code or poc for this, not.

C

At the time um no.

A

Yeah, so just to make sure that we are on the same page, so these are the crds and then christian huffman uh continued. This work uh expanding also uh on not only in pvcs but also to uh one snapshot, um and he also follows the same uh approach as having crds. uh As you can see. It's almost the same just add some things that specific to uh the secrets, uh which are part part of the storage class, but I think uh it's the same approach mainly uh and even the same conditions.

A

The the token just something is specific to the secrets, but otherwise it's almost the same conditions.

A

He just added some things uh like, instead of instead of having yeah, we can have a finalizer to make sure that the bbc source will be deleted instead of deleting it manually. Also. He said that we set the claim policy, as michael said, uh yeah, and then uh we just wait for the pods to start consuming and do the binding uh and again reset the same claim policy and remove the finalizer. So we have a normal ppc uh specifically for for this work, evolving snapshot.

A

uh We would create the snapshot instead of the pv networking space and do the binding with the content. Sorry yeah, one snapshot instead of pvc and the volume natural content instead of pv, will be bind to this on snapshot, uh and then we will delete the source. One. uh The caveat here is that the external snap shelter will be need to update the immutability of some fields. These are immutable right now, but we would have if we would go this approach.

A

We need to change this to be immutable uh because.

B

Excuse me, I think I I yeah. I think I added comments for that. I think this one we uh it should not be should not be mutable it should. It should be immutable.

B

A

So your comments are here. I know yeah.

B

A

Yeah uh but yeah, of course, this uh problem and that's why uh we didn't it's not implemented yet uh yeah, and this is rosetta's plan and again there is no work that have been done this way and then there was a comment on this christian huffman work, some masaki kimaru, I don't know if he is here, he talk actually kicked off the first approach initiated by.

A

Sorry, I'm very bad with names by john.

G

A

Yeah exactly, but instead of having annotations, he just added uh sync like some specs to the pvc uh specifically to be on the same page. He added the async, expect transfer destination, namespace and destination name and the source name, space and source name. Also, he added the status condition type because we would need this while doing the binding and transferring he actually already implemented poc for the ideal case, not any corner case, and he had some tests correct.

A

I will just quickly follow you over what he proposed here, so he said instead of doing the new uh crds or api objects, we can just add this field to the bbc that I mentioned earlier earlier and also yeah exactly so transfer source transfer destination and the status condition type.

A

uh This is an example for the things that will be added to the pvc uh and then, but also this is relying on user interaction. So the user will modify the pvc as a source, one.

A

uh You will specify the destination in space and destination name for the for the created pv uh and user b. You have to create another bb c uh with the source destination, namespace and source name of the ones that you would consume from, and this is the control behavior in which we will do the transfer.

A

So he has to check the status first and then he has to change the condition. uh I think there is actually better approach. I think it's more clear here. If you can see uh he clarified this with comments here and the interaction diagram shows this very clearly. So the user a will have a pvc. A and user b will have bbc b.

A

So uh initially bbc a will be created and will be bound to a pv, as you can see here and then the user user, a will change the specs of the pvc to uh target this previous destination name of the of the to be the bbc b, which is this one and then the controller will check the condition and we'll start binding pvcb to the volume and once it's bind, we will do the unbinding of the source, pvc and then we will change we'll do the binding here for for the bbcb and then we'll change the conditions.

A

As you can see here, the condition like there's a phase and condition both we check for both. So here he's here's a condition is bound, but sorry the phase is bound, but the condition is transferring, and here the phase is bending waiting for it to be transferred, but the condition is receiving here after it's already bound. So we have to unbind this from the pvc a so we just set it to lost and already transferred, and then we can go ahead and declare this as bound to the bbcb.

A

uh He also actually added here that he had twisted this normal cases. Only. He created pve c number one here and then he after his bound and then he modified this added the spec.

A

And then this was the result and then he created another ppcb and changed the spec.

A

And then user a has to manually delete this.

A

Yeah and uh from these comments he just like here he's following up on this: he he said that, from his point of view, this is better than crv or the api changes, and also we can, after that wrap this into another implementation to target volume, snapshot.

A

Yeah there were many comments here on discussions.

A

However, the last thing was that he like just wanted to show you that he had already poc, so he actually, this one was the main thing, the biggest changes and uh he just checked for conditions in the attached touch, controller and the pve controller and open these conditions. He implements the changes and also he did not actually make a lot of changes inside because he used the util functions which take a bbc or pv and do the most of the logic.

A

uh From his point of view, this is actually a working uh implementation, uh but it's in my opinion. We have to make changes for the entry and uh I think the crb approach is better. Maybe it's quite harder to implement, but it's much more bitter and I just want to go yeah to here where he proposed exchange yeah. So this is a most recent changes for this kit and I would like to hear from you your opinions and how we can proceed with that. From your point of view,.

E

I'm fairly new, but but thank you so much mustafa for the uh summary you have given uh the from my point of view again very new. uh I feel that um transferring ownership over storage volume we have to be very careful on that uh because of the security uh implement um situations that we could cause uh by transferring the ownership of some data volume to from one namespace to another. So I I really like uh the acceptance model because it is trackable, it's audible and it it kind of maintains a level of security.

E

Saying I allow uh this uh volume to be transferred. uh I understand that we can transfer. You know with the other models, but uh from a multi-tenant point of view, uh I I feel that is very necessary to be very careful on how we do that. So I I like- and I'm not talking about implementation, I'm talking about mainly the usability of it. uh I I feel that uh you know it feels better to have that type of model. It's almost like a cert uh request, an acceptance model there it's already in kubernetes.

E

So it's just my opinion.

G

Yeah, I know I know when we when we initially started looking at this um one of the one of the reasons for the the cr and the transfer requests and stuff like that was, was exactly that. That was actually a big part of it. A big part of it was um one being very explicit in in actually making one of these transfers take place, but the other was also in terms of um it gave an audit trail in terms of the objects that were created in the events and everything else.

G

Now I I don't know I mean you may be able to achieve that same thing with with the alternate proposal of just modifying the pvc. um That's certainly neat, for some reason. It just seems to me. We talked about that and I thought it was something we decided wasn't doable or a good idea, but um things change.

G

That may be a great idea now. I don't know.

C

um The other thing about the api is just. It applies to many types. Obviously, the p, the pvc one is specific to pvc editing. You know the people um so having a generic kind of transfer api could be useful for more resources, different resource types, so you know the pvc and snapshot api would basically be the same.

C

um I think that that was another argument for the request, the crd approach um and yeah. I I don't know that semantically like if someone has edit permission to a pvc, does that really give them the right to you know, give it to their friend in another namespace. I don't know that's right.

G

That was a big. That was a big question.

G

Is there we wanted to have limits on on? Who could? Who could? Actually, you know, do a transfer, and that was part of it, so you're you're right on the money. I think.

A

um But so far, everything we discussed uh or like the following up discussion were about pvc and we would like also to target volume snapshots.

A

So I think we need like more generic.

B

B

It sounds like a so far. Everybody on the call prefers the crd approach. Does anyone have a different opinion?

B

uh Osaka is not here today, right.

A

I actually myself also like the crd approach. uh The question is, would we make changes in the pvec or sorry make controller and three changes, or we should have external controller to take care of that.

B

I think, typically, we are pretty hesitant on adding new fields in the in pvc, which is a core entry object.

D

B

D

The the question is more about where the implementation is going to be done, not necessarily the api.

B

uh Okay, so the implementation- I think this can be. This can be an external controller right. It does not have to be added if okay, so if we are doing this indentation using the approach that modify the pvc itself, then it has to be intriguing. Modifying the entry controller mustafa does the okay. I actually did not look at the the poc, the third with the which time is it the fourth attempt actually brought up by uh um osaka?

B

Does he propose to add another controller or just modify an existing one.

B

Are you talking? Are you talking.

A

Yeah, sorry, I wasn't mute, I didn't know so he actually changes uh in like entry. Pv control so attach dash controller, the pv controller, okay yeah, because he did the changes inside.

G

I think I think that it's kind of interesting- I I don't know.

G

I don't know what that would actually look like to have a cr and then actually implement the changes on the existing controllers and how the rest of the community would feel about that. But I think it's kind.

B

Of interesting.

G

B

No, he didn't so. He I was talking about the this. Poc is actually not adding cr new field to pvc.

G

Yeah right and what I was, what I was actually wondering is um to the question of whether you could modify the existing pv and pvc controllers right even with the cr.

B

If the cr, then we should not be that we cannot that's it.

D

I guess that's what happened. The.

B

D

The the question is: is either way: um are there potential changes to certain states or conditions that would require changes to the entry controller like even if even.

G

D

We did it as a cr. Are there some assumptions we're making that require changes to how things currently behave today.

B

Yeah, so that's probably something that we actually need to investigate more, because I don't think we have any poc for the crd approach.

C

um So yeah I've got something similar to the crd approach um that I've implemented uh and I think that there would be some nice synchronization if, if we could do things in the pv controller, um like you know, I I think everything work you can make everything work in um with an external controller, but it it is a little can be a little racy like.

C

So, if you know at one moment the you know, no pods are using the pvc and then you start this transfer process and then a pod starts using the pvc um you're going to have to you may be. You know in the middle of a transfer, then you're going to have to wait for that pvc to become unbound again, where you wouldn't get in such states. If it was in the pv controller, probably.

B

uh So in your invitation, you actually modified the entry tv controller, no.

C

I didn't I didn't.

B

C

I I think that it would, uh in some cases, give us extra nice synchronization if we did have and.

B

There was a race conditions. Basically, if we do this one completely completely so so, okay, so first, is it even possible to do it completely outside of street? Is that something that you have already done.

C

Yeah, it's something I that I've got it and it works.

C

The problem is it's just you know what what could happen is um you know we do all the matching that we talked about earlier and then we start the actual transfer process and you know we have to wait until no pods are using the pvc and uh that's fine, but then you can start moving things over and then, if someone starts using the new pvc before it's deleted, you're gonna be in this weird waiting pattern where you started to transfer, but it was interrupted.

C

Maybe uh you know the controller was written, so it'll eventually converge, but um if you know the same controller that was doing the transfer was um you know uh the pv controller, I guess maybe for uh it wouldn't make that pv or pvc available to the to another pod.

B

But so if we are making these changes in true, does that also mean we need to add some new states there.

C

So I've done everything out of tree and it was didn't need anything, um but I think it may be. uh It may be cleaner to do it entry. uh I don't know if we want to right.

B

I'm just saying, but if we are doing entry we does it require us to add some like new states to. uh I don't think it's required.

C

But it may be nice.

C

Like to have some way to um like to say.

B

C

B

Like don't get you think.

C

Don't give this pvc to any more pods or something.

B

Okay, it's just.

C

Like making you kind.

B

Of a lockheed or something.

C

Yeah yeah, if if there was some way to say you know yeah once the pods they have, this pvc and user are done, don't give it to anyone else would be nice, but again it's not strictly necessary. I don't think.

B

Because you didn't do this one because you did auto treat you can't do this anyway right, so you didn't really lock it, but you may have to wait for longer.

C

Yeah, so I there's definitely um yeah, so we basically have to wait until no pods are using the pvc and then delete the source. Pvc and then once that thing is really gone, is when you know that no no one else is using it, and then you can um create the new pvc and up you know, update the pv reference and all that.

B

uh But do you even allow so who is going to delete, so you just wait for somebody else to delete those right. I mean the pods that are using the pvc. Do you yeah so when.

C

The the transfer controller, as I wrote it, will if the transfer is matched, but it's still in use by a pod it'll just wait until the pause.

B

Somebody go delete them: okay, yeah, okay, but that how can you make it better? If it's intriguing, you still have to wait. I guess you. You can just go ahead and uh well.

G

You still have to wait, but the the thing that mic that I think mike is pointing out and the way I was gonna. I was gonna cheat for this, um but but the issue is what can happen is, as the pod can release the volume all right um and then what happens is this?

G

This transfer process kicks off um or or at least it starts, um but at the same time- or you know, milliseconds or seconds after or whatever another pod is, is launched that attaches that pvc, so you've got a scenario there where, where bad things can happen right, um what I was going to do is actually just cheat and reuse the in-use status and and say that it was in use by the uh by the transfer process. I don't that's not very it's not very cool, but.

D

I mean we have the pod protection finalizers on pvcs that might be able to help solve some of these race issues, but.

G

It's good to think.

D

It's good to think you know think through the scenarios um and and see if the race is already covered by our existing finalizers.

G

Yeah finalizer might actually be the answer, but.

B

But that the existing one still will not help in this uh new case right. This is a I thought, uh john you were thinking. Is there a way like say, do not do not try to come and use it? I mean no newport comes out with transfer right, so we need like another finalizer seems to be.

B

The existing finalizer is only for the pod.

C

Yeah, he didn't think that the existing finalizer would work but yeah something new. That would um just mark the pvc to not be you know, assigned to any more pods.

B

New finalizer, I don't know if jan likes another finalizer, he does not want to move analyzers, okay, um yeah, but it sounds like nah. Don't say we need to somehow lock it.

D

um So is someone taking notes on sort of next steps? I think that might be good.

B

uh Some I'm taking some notes. Okay plead about it. Maybe other people they're also taking, can.

D

B

D

So I guess it sounds like you know. uh Most of the folks here prefer the crd approach.

D

um I guess in terms of next steps is, there still seems to be a couple design items that need to be worked out um specifically around dealing with a lot of these races that we see.

B

uh So should we uh go ahead.

A

You know I just want to say, like uh usually, we should start from somewhere and then under testing. We will figure out these corner cases. We cannot get everything right from the first shot. I guess.

B

Yeah, I think we probably at least need to write down the you know the problems. Let's say if we sounds like more people likes the crd approach. Let's say: if we use this approach, uh I think we should at least write down what are the risk conditions?

B

uh Maybe you can also sync up with a mic of languages that you get because it looks like he had to have down the work uh implements external, so like the external controller. So what are the problems that he run into and maybe at least we should write down those problems and then see uh if we can do better.

C

Yeah, I think everything can be done in the external controller, but it might be nice to have like just a something we could mark on the pvc. That would kind of um not hand it out to any more pods.

B

A

Would it be possible that we continue in this document? Everyone can share uh what his souls uh anything comes to him and his mind, and then I mean like regarding the corner cases or things, and then we can. I can take it from here.

C

Yeah yeah, um I I closed out my cap because I thought uh christian huffman was continuing it, but if someone else wants to create a new one or or I reopen mine and or um I don't know, we can say bernie's offline or something, but um I stepped away from this for a while, but I'm willing to come in and and help a little bit now. If um you know, if that's good to whoever is wants to be in charge of things now,.

A

Sure thing uh I mean like it's, not I'm not in charge, just I'm trying to revive it a bit and, of course, everyone who wants to help is more than.

A

A

All right just uh for everyone, who's in the call who has any saying uh or current cases came in mind. Would you please uh just share it in the document would be highly appreciated and uh yeah. So I think also the uh the crd approach. uh I I believe so it's better as well.

F

C

The main thing that I encountered in implementing this was there were certain time in times when you would could make some metadata updates and, like the pb controller, would um generate an event that you probably don't want. So I had to make sure to get the order of things correct so that they weren't ever like two pvcs pointing to the same pb. That would um signal an event I think, um but yeah. So uh I can kind of write down my learnings in this document or some other document.

B

Sure yeah, that sounds great.

A

All right, so uh I think uh uh from my side I don't have anything uh more to say, but uh I'm always hoping for uh comments, discussions uh and yeah, although uh we can pick it from here.

B

Yeah sounds good, so yeah, so basically the next two steps is just to write down uh your comments, your suggestions in in the stock. uh So I think this is already.

B

This is shared with everybody right, the dog yeah. So I added this in this chat, uh yeah and then we can. We can go from here. Okay, sounds great anything else.

B

Let me show you have anything else. You want to add.

D

um No, I think when, when we're getting closer to actually getting a design ready for review, um I think it would also be good to just have a section talking about the previous and alternate designs, and you know pros and cons and sort of why we decided to go with this approach. I think that would be helpful for the reviewers.

B

Yeah definitely yeah, so at least this way we. uh If there are other comments, then we say: okay, this is already covered uh yeah, so at least we we have a record of all of those.

F

Great sounds good.

B

All right, then, uh okay, if there's a nothing else,.

C

So I I guess we I we can write down ideas here, but I'm wondering how do we want to continue on the cap front because they see that? I guess christian isn't here.

B

Yeah, so basically uh so christian says he's he does not have psycho to work on it anymore. So mustafa has agreed to pick it up. So that's why you know he is. uh You know uh driving this.

B

So uh I think, maybe after we consolidated the of ideas in the stock and mostafa you can you can maybe submit another cuff or you can. I don't know if you want to continue on questions. I don't know which one is easier. I think it's up to you.

A

Yeah sure so I think uh christian one has the most recent thing, so I can take it from here. Yes, I create a new kit with a new changes that we will make and the final approach we will take over and yeah. I can lose okay.

C

Great I'll uh I'll I'll write down as much as I can in this document.

B

That'll be very helpful since you already implemented this, so I have some experience now. Okay, great good, uh sounds good. Anything else.

B

All right, then that's it for today thanks everyone. Thank you have a great day. Everyone. Thank you. So much.