Kubernetes Storage SIG CSI Implementation Workgroup, 27 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Storage - CSI Implementation Workgroup 20211027

Description

Meeting of Kubernetes Storage Special-Interest-Group (SIG) Workgroup for Container-Storage-Interface (CSI) Implementation - 27 October 2021

Meeting Notes/Agenda: -

Find out more about the Storage SIG here: https://github.com/kubernetes/community/tree/master/sig-storage

A

um If you have some other topics, yeah, please also add them um in case. We cannot finish all, but we can continue next time.

A

uh I'm not sure. Can I share the screen. Sheen.

B

uh You should I made you a: are you not a host.

A

Okay, I'm not sure okay, so I think, uh as we mentioned, we want to talk about issue young kind of mentioned the last time related to csi volume, stage and stage, for I think um if the volume uh handler is the same, but it might be represented by multiple pvs um and the second item is uh angry's phone, no shutdown proposal.

A

So we want to talk the the young's issues. First is that okay, and uh typically, we have like a feature update. So today we have a few people, like anyone want to give some feature update. First, maybe before we go into the issue.

A

Really to this project board.

B

So, patrick, I think you are waiting for your pr to be viewed right and then that's the only thing planning for the feature. You're working on.

C

Yeah, I guess that's that's not up to you or michelle, to give it an lg dm and then I need to get an api reviewer on it.

B

Okay, yeah I'll take a look just to have a lot of meetings today,.

C

And there hasn't been any change in the in the in that pr itself. It was just that when reviewing it for last time we came up, uh we we identified a few areas that might need more tests and that that was done in separate prs and that's now all merged and ready. So I think we can just go ahead with making a ga.

A

Is is it about informal storage.

C

Or generic ephemeral volumes.

B

So who was the api reviewer? That's.

C

That's what that's the problem. I still need to get it into the review queue, but the precondition is that visiq actually agrees to move ahead with it. Oh.

D

B

Not at that stage because it.

C

Yeah have an lgdm and so on, yeah.

B

I was just wondering if it's better to get the same person who reviewed the beta version it'll be easier and that's what I'm trying to say.

C

Perhaps I I can, I can ask tim whether he wants to take that. But okay still the first.

D

C

That we need to get it into lgdm and then set the api change or api review label and and yeah.

B

Okay yeah, so I want to go look later today after night. I just have lots of meetings, but after my meetings I will for sure take a look today.

A

So you have a few items in here in progress so which one you think is the.

C

What I've done I have, I have other smaller prs pending with code cleanup to use, futility functions and those those have been refueled and yeah. I'm not even sure why this one is merged, but there are others that were still open. What I did is I I merged them.

C

Oh, I included them in in that ga pr. So it's now both ji ga promotion and some final cleanup work.

A

So this one is done, it's merged.

C

I think so, yes.

A

And uh so we need to review your.

B

Screen a little bigger somehow it's tiny, okay.

A

So I'm just just want to double check uh this one is the the one we need to review this.

C

Yeah, I think, all well, the others are pinning I I can add two more pr's for smaller ones that are pending. I I right now. My thinking is the people who should be reviewing them are still. I still haven't done it, so I think I'll I'll just try to get that merged through that bigger pr with tim's help, or so that may be the faster way. Overall.

C

B

um Yeah, I remember well with the prg you need a lot of uh approvers like they are kind of many files are affected right. I think that's this vr, it's not even like in the store teams, like all over the place.

C

Yeah, that's why we have so many different owners, because it's okay using.

B

C

Plugins for scheduler plugins for cubelets and that's why it needs so many different approvers, okay,.

B

Have you pinned the people uh from those uh like scheduler and the hublet.

C

Whose owner javi.

B

Pinos I mean just.

C

I don't I don't I don't. I don't really think we need to get them reviewed.

E

But I mean otherwise it's not going to be merged right because.

B

The other yeah.

C

B

That's why that's.

C

Why why we, I will need a root approver like tim, but I can. I can ping him. I think I'm fairly confident that once we have that api review done on my, perhaps it's even done by him, then I can get him to approve it and it will get it going.

A

A

uh uh Okay, so I think um let us know if uh some pr can be reviewed by us.

C

And uh vga one generic fml volume, ga- that is what six storage should be reviewing: okay, okay,.

B

Yeah, I will take a look at that today.

A

Okay sounds good, uh so any other feature. I want to give a quick update or an epr video.

B

So I wonder uh I don't know if get his mic fixed, because we have not oh yeah, I'm here. Okay,.

D

uh Sorry, yes, I'm I'm actually working on the. I have a pr open for um the fs group change policy going ga and actually it's failing some unit test, but I'll fix those, but the pr is out. It should be pretty trivial to fix it. I'm working on the pr for recovery from resize, and I'm mostly done. One of the things that I am kind of was discussing with you on today morning also was like was our cap.

D

Originally we proposed that we will only change external resize controller for recovery flow will not change the entry one and I think uh that's what I want to do for this series, but at the same time it does mean that any of the entry changes in the cubelet should not affect any of the entry drivers for this as far as expansion is is concerned, so I'm just working on that bit actually because that makes that ins that requires making sure that the code path is separate and things like that so yeah.

D

So I have a pr open for this one. I linked it in this thing for the fs group and I'm working. I should have appear for recovery today or tomorrow.

B

A

Okay, so now we can going to discuss the young's issue. I think uh young create a new one. This one.

A

F

Thank you very much, uh so I created a separate issue for one of these issues I found so this one is about.

F

If two pvs have the same volume handle a csi volume handle, then cubelet is not able to start a pod that uses one of these series if the other one runs off the. On the same note, and the reason is that the pv name is on the staging path. It's the only reason.

F

Everything else would work because, like cubot is smart enough to see that they have the same, these two pvs have the same volume handle. So it is the same volume it caused the node stage once, but unfortunately, there are no stage is done into a directory that contains the pv name, so only one of the spots can actually run.

F

So I looked at where this staging path is used, and there is a case when, if qubit cannot find the json file, that is in the staging directory, it parses the pv name from the staging path and tries to read it from the from the api server. So it knows how to call the csi driver to call note on stage.

F

I am not sure if this code is necessary, because the json should always be the rv.

F

I hope we remove it only after successful notes on stage, so I would be willing to remove reading the volume from api server in mount amount device routes code, and uh do you have any opinion about that?

F

If I remove that, then I can remove. I can rename directory and try to make this use case working, but I would need to remove this pv.

B

Removing from where from api, so I'm.

F

Sorry from node from unmount device core.

B

A

Yeah, so I I think uh just want to add a point in tray driver does not suffer. This does not have this problem, because in tree driver stage like a global mount path, we call it right use the unique kind of volume id like from source volume id. It depends on the driver.

A

They will find out that unique id uh somehow in csi we use pv as the part of the global mom pass. Is there a particular reason at that time? This is. This is the reason.

F

To be able to call and mount device, I think initially we didn't have this json file, so I think that was the reason, so the unmount device retrieve the pv from the from the directory name but later on, we introduced the json file in staging directory. So I think we can pretty rely on presence of the json file.

F

Nowadays, I'm thinking.

F

No, that's not possible. I was thinking if somebody's dragging like the same volume amounted by, I don't know kubernetes 1 13 or this kind of stuff and just restarting cubette without draining the note. I don't think that's possible because we require notes to be drained during updates so yeah this. This can't happen.

F

A

F

A

Oh, you mentioned: uh if cubelet restart um what will happen if we don't drink the node.

F

Well, I was thinking well, there are two possibilities we discussed last time: either we updated in the minor version, update, meaning 124.0 and require a draining during update from 123., or I could imagine some code that tries the old path before using the new path. I think it would be kind of error-prone, but it could work.

B

Yeah, but if you don't uh handle those uh no existing cases, then I I just think there'll be a lot of issues right, because this is not only for the. This is a particular file file share uh thing. It's also for everything right all the block volumes. Everything is passed. It's if we don't try the old past. I think we all we just rely on everybody. What's in the node, I mean, I think, a very big assumption.

B

F

Oh well, we already rely on that uh during csi migration.

B

So that was actually another thing I was actually today. I asked uh debian from my team to join us, but he was actually saying yeah, since we are asking that can we kind of uh if we can align those two changes he was asking, but oh.

D

B

This, but this one this is a big change. This is going to go through some stages before we are, if this is a drastic change, we also.

F

B

Update must go through draining that that's I'm just saying that if this one I'm just saying if we want to align those, maybe we cannot line those two, because that one particles.

F

Still, every time you update kubernetes version the minor version you must drain so.

E

Something else.

F

Could break like I don't know something you know.

A

And, and uh also in in case of csm migration- let's say you have entry uh pvpc and after you upgrade and csm migration is enabled. um Will that break this uh in case in this situation that multiple? No no.

F

No because you must reign during dublin, you.

B

Have to join again, basically just searching. Basically that's what I'm worried. It's like with this change. That's the joint before you apply this change right because.

A

Otherwise, I'm just saying without any change, uh just enable cs immigration.

F

Well for entry, yes, but this back is about csi volume itself.

A

F

By csi migration.

A

So I'm just saying like before in tray right, it works, but after you enable csm migration will.

F

Break, I would need to check how increase the ffs works. I don't know.

D

So uh what about going in other direction? And we actually do notice this twice for two different pvs, even if they use same volume, handle.

B

Well, you say: google's tries just keep the same same code, but always do it twice same global path, global path, let's.

F

Break csi in csi uh volume can be staged only once on a node. You can stage it thrice to different directories.

D

um So the fact that the volume is part is encapsulated by a pv does not matter.

B

F

Of view they have the same volume, they have the same, handle for.

B

The for the fact that you try not to do the stage or the stage is the required in our case actually uh for the firewall, and we we don't do stage.

F

Yeah, like I checked, uh I checked nfs, I checked efs. uh They don't do stage.

B

Kind of fired for stephanie; no, no! It's.

F

Not no, no, it's not they. Don't that I mean. Can they also do all.

B

Of the same I'm just asking, I mean: if only does this then run into this problem, maybe they can just change that and follow the driver. Is that.

F

It's pretty impossible to change to remove staging from existing driver, because, again you would. You would need to drain the node during the driver, update.

B

Azure file.

B

Do they have this problem like I was trying to find out.

D

F

Okay, they should.

D

Have same problem.

F

They should they should. I didn't try. They should have the same problem.

B

I mean if we, if we can keep the like, if you record all you know, use the old path and then I want to think that's. uh I.

F

B

Otherwise, I just feel it's so dramatic change.

B

I mean every driver is I mean uh until we move everything to this right, so it's not a problem to keep that path right. Why is that a problem to keep that pass.

F

That's what forever for that's.

B

What backward compatibility is for right? I was just thinking.

F

Again, like users are required well, cluster arguments are required to drain nodes during minor update. This is nothing surprising. This existing requirement is documented. You must drain during update minor update, not patch.

F

This is not surprising change.

A

Yeah, I kind of also feel use pv as a part of the global month pass. It seems uh not like correct.

B

Yeah, I probably want to find out that, like the history like it's a little weird if we are not doing this for entry and only doing this for csi, weird, oh as.

F

Well, first for csi there are not so many options how to do the unique volume name.

B

But the but the warning handle was there since the beginning. Why? But.

F

You know it's long uh and I don't know, but then.

B

Now do we have the same problem now I mean you're right.

F

Yeah we have the same problem.

B

But then what do we do you? So so you're saying I wrote it.

F

B

What is sure, but what is the promise you too long? I mean it's the same length that it's not going to be so, whether we.

F

System the system really starts to complain, for example, I'm sorry so systemd can handle only paths up to 20 250 characters. I believe.

B

Oh and oh you're saying warning handle because it's returned by the driver. We don't know.

F

Yeah, it can be long, it can contain with characters like we can.

B

Escape everything you know what how do we fix this? I thought that your suggestion is to use one another. How do we solve this problem.

B

We have the same problem, isn't it.

F

In the issue like we can use, for example, we can compute sha. Oh.

B

What did we do for the entry but gene you're saying that, like in screwdriver, we just use the one handle, but that's the same. It's. How is that different? The entry drivers handle on the side.

F

Volume handles for you, uniform ids are computed by the volume plugin and they have their own way. How to do it correctly. For example, in aw s we use the volume id in the cloud, which is, I don't know, 20 characters total and we are sure it contains only the right characters. We can use in paths. So every single uh storage backend has something like that, which is usually very simple, just contact concatenation of smart strings, but in csi we have this volume handle which can be wrong and actually in cfs.

F

A

But it's for how about intrigue self driver. They are using different id.

F

uh I would need to check what they use, but I don't know what they use. It.

B

Looks like shouldn't be the same problem if it's the same you're saying it's using one handle for entry right.

F

um uh I haven't seen any issues with injuries fs, but I haven't seen it for a long time because everybody uses csi these days.

B

Only for the same one handle I mean yeah. What's the difference, I'm not getting that? No, no.

F

No, I'm pretty sure they use different volume handle. Oh.

B

It's a different volume handle. That's not the same.

F

Well, they use unique volume and unique volume id really. My id is not a volume handle it's something else.

B

D

Machine of actual state of the world, their unique volume id is not the.

F

um Let me find.

F

A

So we don't have any restrictions on volume handle like how long it can be. uh I think we do, but.

B

We also we did, we also expanded and we did expense something in css back, I don't remember which ones uh I did.

F

And yeah, like uh the volume handle itself, is not enough, because you must press exit with this driver, because two different drivers can agree on the same volume handle.

A

F

Use accidentally the same volume handles so you need to add the driver name which is another. I you know 64 characters, and you know it adds up quickly.

A

Right basically,.

B

A

His driver name.

B

Is that okay, would that be enough? uh Could you be running like multiple drivers on the same you know on the same, not could you be running multiple of them? It's actually possible right.

A

um Inside of us, the same driver right, the id should be unique and you can have different driver and the id could be the same.

F

Like everything, this is written in the issue already, so I am suggesting you computing. Some have some of the of the driver name and the volume angle, and I hope they will be unique enough that they will not conflict like once in a lifetime. It could happen if they conflict, but yeah.

F

We use something similar in volume attachments, we hash the volume, the pd name and node name, and we compute hash, and we expect that they will never ever conflict.

A

Yeah, I think you, you already put all the details in the issue and I tend to agree like we. We want to change to get rid of pv name and uh but the issue we um after, like uh you, make the change right. We can test in what situation in the break and how may we address that? We can yeah so.

D

The breaking will be like you will have to like upgrading the nose will be, existing mounts will be broken, so you'll have to drain each node.

F

If you don't drain, I don't know, what's going to happen, something.

A

Bad yeah, I I don't know yet so that's what we want to test. Maybe we have a way to recover from that. It's not like yeah. We are saying it's totally brick, it's still possible. We can recover.

D

uh Still recover from like undrained, no, then it will have wrong path. How like we'll have to quote some back, like fallback logic, in the cube that you're thinking.

A

I right now- I I don't know so because yeah where it will break right, then we can nail down the possible solution right now. It's just roughly, it probably will break somewhere, but after we know exactly which code right, then we can talk about the possible solutions.

F

Yeah, okay, I can try to write some patch.

F

And we can discuss on the pr.

A

Right uh like uh we'll just make them a bigger change, and then we can test um against that to know what will happen. We can restart kublaid, for example,.

B

So yeah you're, saying the you check the code. Basically it's really just at this one place. We're actually checking this path.

F

B

F

The global the global path really is parked only here. I believe.

B

F

B

Did we check the scene this other.

G

Reconstruction to be do something.

A

During reconstruction, I do remember we do check the global moon pass somewhere.

G

B

Because yeah- because I remember I look at that part, uh where is that, though I forgot, I.

F

Think we reconstruct from reports from the local.

F

Just read the json file from the local volume path.

A

Yeah right for csi, it's probably um different yeah.

B

Okay, so maybe maybe this is something that okay, okay, I think you know there's some measurements in my this other captain grace clean up the global mount directory in the cube, cubicle warning manager.

A

So um yeah, I think young could go ahead with some changes to try out.

F

Okay and I'm not sure, do we have a time to discuss the other issue. I have some idea and but it's not that well uh described, I would say.

F

No, maybe we can skip it and talk about the gray social note shutdown on angry, so.

A

Oh, you mentioned this, something else you you, you have some idea.

F

Well, yeah, I have the other issue.

F

When two pvs use different volume handles, but in the mount table they look the same, they have the same mount source, then uh cubelet outside of csi. It checks when it wants to unmount the device it checks. If it is mount and it's somewhere else- and in this case it will find the other volume mounted and it will not have volume and yeah, uh and it will always complain an error and you can't basically unmount the volume ever.

A

Is this issue? Is the practicum.

B

That's the that was the issue yeah, but.

F

Exactly it's a very similar issue, but yeah uh like.

B

From the same issue from the same implementation causes two issues, I guess.

F

Now we have two pvs with two different volume handles that's the reference, but the rest of the setup is the same. So in the end.

F

In the end, in this case, node and stage is not called because before cube red course not unstaged, it checks if the volume is mounted somewhere else and it finds the other volume on it, because it has the same source in mount table.

F

That's the problem here we have two different volumes, totally different volumes, but they have the same source in mount table so cooper. Does this check? So I was thinking like what, if we remove this check completely, because qubit already checks that all the ports that use the volume are gone and their volumes are, the local volumes are unmounted. We checked it somewhere else, so why is this check actually good for.

B

So basically, just allow cubelet to call it uh as many times as it needs uh if.

B

So in this case it will be called twice basically.

A

Do you have item uh code is here.

F

Oh no! No! No! I don't. I don't think that's about I, but I said it. I don't have it well described it's just in my brain. So in this case.

D

What's happening is that that two pvs they use same volume and when both the pulses are using that are deleted. The node on the stage never gets called.

A

Yeah so yeah well.

F

It doesn't need to mix the same volume. That's the problem like if nfs use, node stage it would hit this issue too, because the sources look the same just.

A

Okay, correct so you mentioned that you want to remove some check on monks right, so I just want.

D

A

A

And I always like feel if possible, we should remove those smart check. You see, we have hit a lot of issues.

G

A

So it's possible to get rid of those logic.

F

It was somewhere in uh operation generator.

B

So the check is to check if the monk points is already gone or something or is it.

F

Yes, it well, it checks that the staging uh mount is the only amount on the system of the volume on the system.

B

Oh it okay, it only goes.

F

B

uh It if it only causes, if it's uh not the only cause, if it is the only only one yeah, it's called two, then it does not call on stage okay.

F

Yes, but it checked before before that it checked already that all the ports are gone and the volumes are mounted this is it was checked. Why.

D

There are two actually left at that point because one was.

D

What was to do two entries left in front like.

B

Two handles pointing to the same volumes stage twice. Basically, so then.

F

There are two different volumes.

C

F

I can link it somewhere or into the document in the meeting notes yeah. Unfortunately, I'm not at the right computer right now, so.

A

You can just post it on the chat here.

F

Well, I'm going to through my phone and this is complicated, but I can't sure yeah I can join. Maybe I can join also on my computer without the audio.

A

Yeah, typically, we do the check, I'm thinking the reason we do, the it's kind of like a month reference check. um Is it because we want to avoid some risk condition? We don't trust um the actual state or we want to kind of verify physically, like there's. No, no longer some months, um hello, where that.

F

I'm just thinking uh if we unstage the volume, then we also remove it from the no status and we like treat it as ready to ready for detach, but if it is mounted somewhere else, I don't know.

F

D

Not that ready.

F

F

So I in the chat I pasted, link to the code that checks the mount references. This is outside of csi. This is the same for all volume plugins. If nfs did staging or mount device, it would hit the same issue I believe, and the same for sev and for most of the ones.

A

So I think this is the get device mount reference.

F

A

If it has mount reference right, so you think this still being used by some part.

F

um By anything on the system, it's not about pawns, it's about really any any mount somewhere else anywhere else it could be. I don't know system, I mean mounting the volume for some. I don't know.

A

So, in that case, every amount, if there's still some mount reference, it will fail.

F

No, no, it will amount to this the stage mount and if it's mounted somewhere else, it will be still mounted somewhere else. That's okay! So.

B

Will there be any problems because it's like, if we remove this check, could we do this on stage too early? I guess that's the. Could there be any potential problem there?

B

That's that's why this yeah right, uh supposedly you are supposed to cause stage only once so now we actually could call that multiple times could that be causing problems for some other drivers that actually only want to on this one to be called. I guess that's something you just need to think, and I don't know so.

F

um I personally don't see too many problems, but I would appreciate any feedback before we remove this check yeah. The only issue I see is then, if the note on stage succeeds, which probably succeeds, they declare the volume to be ready to detach.

B

E

We know that the volume right- oh.

B

I okay, so how does this one work now? Yes, you do that twice, but then I mean like for like, for this is a s driver. They have to call this unstaged choice before they detach. What is that right.

F

Well, they don't they don't implement it. Actually, that's fine yeah.

B

F

B

I'm just thinking I have a problem so.

F

That's what I don't know, yeah.

B

Maybe we need to ask- and just uh you know this behavior: could this be a problem for other drivers? If we go on stage too early, I mean what problem will.

F

B

Be I guess I mean for the.

F

Drivers that they find out- I don't know, um but for the drivers they do at that edge. They, I think, like all of them use block devices and the block device is always unique for each volume.

F

B

F

B

It should be just one that yeah I I was just wondering why this this check is here. I don't know if there's other reason why this chair.

D

I think that there could be like if the pod, if the cubelet crashed- and it was restarted, then the the counters that we maintain like okay, we before we are on one device. I think we count the reference like if this is the only reference left. So I suspect that, uh like yeah, if we just relied on the state stored in cubelet, then we could have like wrong data or something. So that's why we are referring to to mount table to verify.

F

Okay, but uh if it is not in the cubelets states of the world,.

D

Count is wrong. Like the the the reference count, we maintained right.

A

But like uh young, I think it's that's not not in actual state. We. We never call this like uh call this uh operation at all. So that means uh we should have information about the volume actual state.

F

um I'm not sure that we have a counter. I think we have a list of uh spots that depend on their volume, yeah.

D

Yeah but that's like we use that as a counter like when it is.

F

Yeah, but if the list is wrong, then nobody is going ever to unstage the volume, because there is something missing in the list and nobody is going to unmount. That thing.

D

But what if it races with the r like reconstruction and the score races? That's what I think.

A

uh Reconstruction only happened during the accumulated once yeah. It did not periodically run.

D

Right but it the reconciliation runs. It is possible that some code is reconciling, while, while the because I.

A

I feel like it's here. We do the check it may help like, but if we think about the risk it does not have any guarantee because after you check it still my some new month company, so there's no 100 like a percent guarantee.

A

Even we have the check um the the tech. um I don't know I'm trying to find history, but I I yeah.

D

I suspect that's why the check was introduced, like you know, like having this not having a very robust reconstruction mechanism or something but yeah, like your point, is having this.

F

Then we have a csi driver where this check doesn't work.

D

Yeah, I don't know, I think we need to think like how to.

A

D

Yeah, I understand the issue, I understand the request, but I don't know so.

A

Let me try to think if I can find some history about this.

A

F

A

F

The r that adds it, which four.

A

G

F

uh I just sent it to the chat it's four years ago and that's the huge reconstructionary work that you read.

G

The old history, it's my pr, oh.

B

You're is it new.

F

No, no it's four year old, oh for you,.

B

F

Knows what nobody remembers.

B

Maybe I'll take a look at least you have some description and see how you know what how this is supposed to work. Okay, good, at least I will be because I don't know the history.

A

uh I I I can take a look because this is my pr and um yeah. If I can remember anything, it.

D

Has it has a very helpful comment, actually yeah.

B

I think that actually very very good, because if you just read the code, you don't see that it has a comment.

D

Why this is a workaround, for uh this is a workaround in gci cluster. If gci cluster is used, mounted gcm monitor user mounting.

B

This is the first part of the redesign of the warning reconstruction work.

D

Like you open the second link that I posted in chat, it has the comment.

B

A

The second link, this sorry.

D

Another second, the one that I posted before the answer. I didn't post anything the last link.

B

Yeah, the other one has a big comment: yeah a long description, long car description.

D

uh It should take you to this.

B

Pi is the first part of the design right that one no.

D

No, no, no! No, I don't know github sibling is failing. Oh.

B

Oh, oh, oh, oh, oh, oh you you're! Also, okay, okay uh walk around for the month. You should cause she's hammered. Oh.

A

B

A

No, the the this workaround is mean, uh doesn't mean we don't want this month, reference, it's just how this month, reference implemented. I think.

A

So because, at the beginning, the maybe some implementation of month- reference we'll have issue, because there are some months we should not come, but we can't uh so it's just the implementation of how much reference is implemented. That's uh to do. I think, but it's not a reason why we want to check the amount of reference.

A

Oh but yeah I will check yeah. I will check and see. If I can remember. Oh, it looks.

B

Like this is this is your mood.

A

Isn't it uh this is uh no it's it's already there. I think my design does not add that logic. Oh.

G

You basically just.

A

Moved it to some other place, yeah right. Let me yeah. Let me take the details.

A

Okay, uh I think, uh next time uh I would see whether I can find something about this.

A

uh Do you still want to move on.

F

Oh yeah, I think it's I think it's all from me. I took already too much time.

B

uh Yeah, so maybe you can maybe jean you can take a look and uh you know.

A

And tell us: what's.

B

The history behind us yeah.

A

That's what I will do here.

A

uh So wait: do we want to continue.

B

Just to quickly talk about this yeah we cannot, and obviously we cannot finish, then we can continue next meeting, but I just want to talk about this a little bit. Can I share the screen.

A

Yeah yeah, please: I need to stop.

B

Okay, so I need to find the okay. This is the pr uh yes, I just want to go.

B

I have not updated the yet. This is still the same one. uh So I think what I want to do is to remove the part what we're saying we're going to depend on the graceful, no shutter tap, because we're not right. So now, initially, uh we said we we narrowed down. The scope only handle the grid for no shutdown, but now it's actually the opposite.

B

We uh we only handle the case that is not handled by the crystal no shutdown. Basically, uh so I just want to see if this general idea and see you know uh what you guys think, and you know what what else we should try for this. So basically this uh we introduced this uh csi spec field called the safety touch, mainly it's just for csr driver to say if they want to opt in for this feature, obviously meaning you know, css driver know knows this feature and they think this can solve their problems.

B

So this uh we have this flag here and then uh the in the existing logic. What happens is uh if, uh let's say if we have a, uh I think so now, we should actually also consider like node partition case with partition or or graceful or it would like no shutdown but not detected by cubelet, somehow right not detected by their graceful, no shutdown logic.

B

Then it will all falls into this because what happens is like after five minutes uh the uh there is the notice the taint manager will try to delete the parts um and then and then but then the part will be stuck in the terminating state. It's going to you know. I think this is reproducible.

B

If, uh if it's not yeah questions, do you have questions sure.

A

uh Yeah, the idea is mainly like a force to detach how about the pod, so the part will continue stuck in terminating.

B

No delete, we will delete parts so the so this. This is the existing logic right. So the proposal changes that is, we will delete the part. So that's the the proposal logic. This is in the there is a pod copy controller.

B

This controller got a gc port controller, so in here we are adding some logic here, uh just to check to check if uh you know, uh but this is of course some of this will change, because I was initially trying to check if the if this is a great final shot. Actually so right now. Basically it's just uh over here if this is this is a terminating. We basically waited for already. This is like wait already waited for uh five minutes right, maybe five minutes comes to here. Then we will start to.

B

We will check this we'll check the safety catch flag in the seaside driver so that one we introduced if that is set, we know that driver wants to opt into this feature. Then we will try to delete the pod, but we also you know here we are trying to add some uh quarantine paint right so before we delete the part.

B

We add those tint uh just this is trying to make sure that when the part comes up again uh well, no on the node on the node, when the node comes up again, we don't want to. uh We want to clean up before we schedule the part to them. So the so, basically we add 10 on the node first and then we try to delete those parts that are kind of stuck forcefully delete them and then um yeah. So that's what it says here.

B

After applying the taint on those node forces delete the parts that are stuck and then and then there is also some change that we need to do in the attach detach um reconciler.

B

So uh there is, there is a check that basically we need to. We need to kind of skip this there's a particular check there. We have to skip that. So, basically, if there's a safe detach, then we will skip the skip this check so that it will not block us from moving forward to allow the volume attachment to be deleted.

B

So so this will actually then this will go ahead and delete the volume attachments and then- um and then this way, basically the volume unpublished, controller on publish one will happen before um publish or no downstage this. You know when we allow this one to happen. So there's this. This is something that just you know. We allow this one to uh go kind of out of sequence.

B

Because of this uh you know abnormal situation and and then, of course, the the attacher external attacher will detect that and then it cause controller on purpose volume, basically uh and then uh in the and then this one basically just saying that well, the css drive right see the driver is it's opt in and then we'll check whether it's safe to detach or not.

B

So uh you know you see a drawer, doesn't I want detection and it does not detach and then it will it basically retry it easily.

B

A

Right overall, like I think it's uh we probably can handle the shadow case. uh Just um if we think about the network petition, then it's very hard because uh from the network, when the workload's still running and at the same time you detach the volume it will cause some data, corruption.

B

So this is the this is why I think uh here we're saying it's the csa driver will have to check. So that's why I think this solution, initially, I think, was written more for kind of cloud provider like you can actually check your compute node uh go find out whether to save to detach or not yeah. So I think it's kind of like a more narrow cases, because, uh yes, just as you said, the partition case, uh if you see the driver does not know then just have to you know, don't don't detach right.

B

So basically this will be. First of all, this will be like opt-in, opt-in option because otherwise, we'll have to really go. You know there are some, I think, there's actually, um I think huaming have some something in his own ripple, that's kind of old. I think a few, maybe a few years back, he proposed that it's a fencing right that you actually have to get.

B

You have to have to reboot the node, but then that's also very complicated, because uh for every cloud provider or even for any driver, it's a different shutdown command. So it's very complicated. Also, it's like external um controller. To do that, I I you know I just feel kind of uncomfortable to have that incorporated in this approach.

B

I just wonder if we should try this one first, uh I don't know if you have you look at the the fencing thing that he proposed, basically you so basically for every so the so basically adam will have to go configure that uh if it's uh you know for different for different kind of environment, you have to use different command to shut down. Basically,.

A

B

A

Adds or just one thing uh to make it safe, so we can make sure the machine we can get the machine state and uh if we can say the machine we know is shut down.

B

That's a driver right driver's side because it's we're not really. We can't really uh like hear we're not really. We can't really check that.

A

Yeah in the beginning, in the very beginning, like controller side like a cube controller or whatever controller, they can check the machine state as long as.

B

I don't think they can check. What do you mean check? The machine state check notice that but no state already notice node is not ready right, it's not right, no, it isn't already, but so what what you want to check? But what you are asking for. I think it's more like the call provider may be able to go check. Those kind of, if you know that but then but then I think, then I don't think it's. Maybe that's. That would be like cloud cloud provider logic uh we can check it. Can we check?

B

I don't know when I'm checking here. This is the this. Is the paw, the gc controller, so you are thinking another okay who is.

D

There no shutdown reason in the pod. I'm sorry see that again, who is adding this reason in the pod status to be known.

B

Oh sorry, sorry for this for the sorry for the confusion, this was at that time. I was uh trying to incorporate the grace for no shutdown logic into here. That's why so. This will not no longer be there. This was at that time. We can uh frown the grease for no shadow. We can know the reason, but but as we discovered griswold, no shadow can handle it. If you know if they actually detect it's a great one or shadow, so we can't. We cannot rely on that. One.

A

Remember we add a logic in, I think node lifecycle controller somewhere to check the machine and then taint the shutdown state.

B

Yeah, so I think we are actually at the top of that. Maybe I can uh maybe jingle ping offline on that that thing we will. Can we continue this on monday, yeah sure.

D

B

The other thing.

A

D

That there are two things one is the tent was there, and the second thing was that the the case, the race racing, is when the node is shut down, but then and and the cloud provider temporarily. Yes, reports, the node is shut down, but then it gets.

H

D

Back up and what, if that races with whatever we are doing,.

B

Right, so that's why we we need to add the you know the we need to add the we need to add the quarantine 10 first before we actually even shut down right. So that's, uh but we, I think, yeah I actually have to jump on another meeting. um Can we can we talk about this again on monday, yeah.

A

uh Sure then I will send an email saying we still keep the monday meeting.

B

Yeah, I think- maybe just I will say I would suggest- maybe just like on friday afternoon- check if there are anything on the agenda. If there's nothing on the agenda, then we just cancel that meeting. How about that in this way we have you know if we actually have this the time to talk about these issues. I think this is actually helpful.

A

uh Sure so anyway, I sent an email to remind uh like.

G

Make it clear, yeah, okay,.

B

All right sounds good. Thank you.

A

Thank you again. Okay,.

B