Kubernetes Storage SIG Miscellaneous, 28 Oct 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Storage - Volume Health Proposal 20191028

Description

Discuss Volume Health KEP

29 October 2019

A

Okay, video started okay.

B

So I will share screen.

B

B

This one, okay, alright, so.

B

Okay, so, basically, right now in kubernetes, there was no way to monitor the the Peavey's. So that's why we want to add the support to find out if watering modems are unhealthy to find out. If there is any problems, always happened with the volumes either that the volumes doesn't exist anymore or it's not attached, or it's not mounted anymore or in terms of local TVs, and if something happen to the node, then also we should know so that's the little motivation so here so listed for cases.

B

The first one basically just meshes that if the volume some house deleted not not through Q&A, this is outside. So we need to mark the volume and there's of that user knows something is wrong with the volume and also, if attach the volume.

B

Somehow it's not attached anymore and I think this one is some rewarding and if the mounted volume is no longer mounted so we also want to also mark the volume and then so this fourth one I, don't know if this is a use case or basically, this is more like a different way of implement this. This is actually coming from one of the review comment that cap so some sort of systems.

B

They can actually detect these type problems, and so, in that case, maybe they want to report that when it happens and then in this case our house controller can actually detect that right away. So that's something! This is something that we added after that review comment. So it's a donor's that can take a look at the diagram. How that looks is that it's a little different, okay.

C

I had some questions on the use cases: okay, so I kind of see the errors, the types of errors you might hit, I guess kind of split up into well I, guess so: I'm I'm not sure about the attaching and the melting one I guess.

A

That's not necessarily volume health right; those are kind of just communities state, but.

B

It's still warring with house I think it was after someone opened an issue and then Michelle actually said. Oh you're, cheating me and Nick on that issue. So basically they it's basically CSI volumes that God attached and Monday and somehow things happened, no longer amounted. Then we also want to know right because otherwise, what it's it's just another same as before, superstitions you still find part of the the whatever audience that that are used here. I guess.

C

Maybe just to reworded use cases, I. Think, okay, instead of saying like we need to mark the P because marking the PV sort of like enough in detail. Okay, it's more like what do we want to happen? Yeah yeah like for like the attaching one. What we want to happen is if a volume becomes unattached, out-of-band.

B

C

Want it to be reattached mm-hmm, and if, if a volume got unmounted out-of-band, then we want to remount it. Okay,.

B

So that got you that reaction phase, which is listed as the second phase of so it's not really part of this, this cap right now, okay,.

C

Right now, this is like from like a user story. Standpoint like it's it it's easier to sort of see the bigger picture. I guess, okay, like where we want to be because, like here and state, is mark the PV yeah.

B

What do you do we done yeah sure? Does it make sense to you list some detailed cases of what happened like network errors, or you know like there are all kinds of errors live.

C

Those yeah like really list out a lot of the different types of errors: okay, I.

D

C

I think that would be good, okay, but I think yeah. The other important thing is to also sort of enumerate like from a user's stance what they might want to get out of it. hmm Okay,.

B

C

Okay and Mike in terms of implementation details, we can definitely split up the implementation parts. Okay, it's okay, I know from like a top-level thing. It would be good to have the big picture. Yeah.

B

That makes sense, okay, sure, okay! No! So this is the.

A

Case is just right off the back seam. It's not what I was expecting when I kind of saw the volume health ah is the the topic. I was anticipating something along the lines of my volume is on healthy because of some. You know.

C

Okay, slow I, Oh or something bottlenecks; okay,.

B

Yeah, so that's it! Yes, some of those mentioned later on here, but yes, part good to listen.

C

Here it might be good to like sort of split up the types of errors since the two times of things like control, plane, errors and I own. A data plane errors, just those kinds of issues that users might encounter during runtime. Yes, I.

A

Think that's a very good clarification, because this.

B

A

I expect to focus on the data plane issues, because we do have a lot of metrics and insights and all the work that we've done so far has been focused on the control plane, mm-hmm.

C

A

C

From a user standpoint, I think I feel like they would more likely hit data path issues once it's melted versus, like things got detached.

A

Track that as a separate kind of cat, or does it make sense to.

C

A

C

Might be good just to track here but like split it up into categories, and then we can Mike. Potentially a look at prioritizing and picking a couple to focus on first or something seems reasonable.

B

Okay, so I think we kind of thinking all of those type of errors will be detected. If you know, if the storage system can't detect them and then it's just, we didn't really categorize them because yeah it just there'll, be a lot of everyone's and the user can.

D

Clear, whatever message.

B

They they want in the you, know the annotation and actually look at yeah yeah, but definitely we should that this, this part of them. We should add more to this far so I think the we do even tend to capture all of those type of errors. Just we got a lot of comments. Actually, when people are reviewing this cat, they listed all kinds of arrows, so fun, it's kind of hard to categorize them. That's why I know just say: okay right now, we just still have this.

B

You can just put whatever message you want, because there are so many, it's always it okay. This is the format you just.

C

Feed here, yeah so I think I mean it might be good to like a numerators and then.

D

C

Can see based off of all of those types of errors? Can we categorize them into specific things? Yeah.

B

C

I'm like specific types for different types of errors: okay,.

B

Yeah that makes sense. Okay did that, so this basically just shows how we're going to implement this. So we'll have a external controller, a house controller that will be deployed on the master node. So this will be detecting the health problems for both network storage and local storage. So now we combine the two I think initially in the first draft, they are kind of completely separate so and then for the for the for the network storage.

B

Then we're going to have this CSI controller, a PC to go check the conditions of those volumes, whether they are, whether they are still there, whether they delete or and whether they are still attached and then for the local Sergey I think this is no controller or PC for logo story. So this is just that you check the know the condition just to see if a node actually failed and then actually, this node failure can that we can also have those for the network storage as well.

B

Just those would be events so, and for so this is the so I think this is the the use case for who mentioned earlier. So this is basically saying: okay, if some sort of system they have this, they have a way to find out, detect those errors early. They want to report them. Then we want to be able to receive them, and so this way it's like a quicker. We can collect.

C

B

C

With this, so, like network storage is already kind of separated out from local storage, could we sort of generalize network storage and CSI? It.

B

Is CSI yeah so basically we're actually okay, so what happens is maybe if you we go further, you will see so and if you look at here, I think Nick actually added some notes here. So when when the CSI driver for local soil is ready, then we're going to use that, but otherwise for local storage. You know it doesn't have the CSI and right so yeah, so this actually means, if you, if you see here, we measure this control RPC, those are actually what we CSI so for local storage.

B

If the driver is available, then what you see I say otherwise, just but.

C

What try to make the interface the same? Okay, so for the external controller, are we saying this is like a sidecar SCSI sidecar yeah it'll be.

B

A sidecar just.

C

Calls and will happen, add some my call to the CSI spec, okay.

B

C

B

Yeah so now we're thinking. Do we think you were taking this ability? I don't know. Maybe we can discuss about that later. I, don't know if this one makes sense to you. Maybe it doesn't right. This is just to check in I. Think.

C

It was the only purpose of this controller is just a query: the driver, yeah yeah, thatis, yeah and.

B

As you record,.

C

It I think I can all could be contained within a site. The.

B

Same yeah that.

C

B

It doesn't seem like there's.

C

Much common like actions or anything like that: yeah recording health, yeah.

B

Yeah, it's more like yeah getting those but not really yeah, not really a.

A

Position this new controller: when does it figure out which volumes to track or how does it figure that out well,.

B

It's my you watch everything. That's its yeah.

D

B

Yeah, it's going to be very so that's why I actually think of that way. Then the second model may be more appealing, so this is more like okay. This will be kind of waiting here, yeah and then whatever never something happens, and then it's going to send out some message and then this the sort of the controller will detect that do.

C

We need a are you're saying, so this third model is like a push model instead of a pull model. Yes,.

B

But this way it's not no longer the CSI and anymore I. Don't think this fits into CSS. Yes, it doesn't have this type of thing. So that's why you put it this. This separate box here, like I'm, not.

C

Sure like, why do we actually need this third thing, or can we just say it's implemented in c in the CSI spec yeah.

B

We can say that the reason we added is because there's someone added a commenter they are saying hey. Is that a maybe a more efficient model? If you do this way? That's that's why we have it up here. So we have a question mark yeah, so we could elicit that. Maybe we can remove it if I'm here in this city and like alternative, saying, hey, there's some mentioning, but the system doesn't really fit into the CSI model. I want to you know Jeremy, so you can.

C

Support bi-directional things: I, don't know so that a CSI driver can actually push events I.

B

Can look into that? Oh it's just.

C

B

I, don't know if that's possible can look into that. Hi can I make a quick point here, yeah sure well,.

D

Just going back to your earlier, you know thing.

B

D

A health expect to the CSI spec right in additional method: uh-huh, isn't that ugly, a get that can be used for the purpose I mean if you.

C

Like CSI has a list volumes call yeah, we could potentially add some stuff there. Yeah.

B

So that's well at your last time, I think when David proposed them. What does that get? Some notes, basically get the published notes. So I was actually thinking. Oh great. Actually, if we add more information there, then yeah we need to introduce this by I was thinking that might.

C

Be like the most ideal place, to put this kind of information.

B

Yeah that that's probably fine yeah so, but that still will be a separate two billion right. I think when David use that it's a separate. It's just about.

D

Together, yeah.

B

That's fine yeah, that's totally fine, but I think the one that he added doesn't really allow us to add more error, message or something: it's I think it's a like a list or something I forgot, but we can look at the into that later. Yeah! um Okay, thank you. Okay, thanks so I think.

C

B

C

Interesting but I wonder if it would just be easier if, if, like storage, vendors, really wanted to write an external controller to push stuff, they just directly do that they don't.

B

Need they need this.

C

B

C

Controller or sidecar wanna find the status for them. That's true, yeah.

B

They can actually go add an annotation in the TV themselves right, it's funded and that's true about you, okay, okay, all right! So we'll just leave that one out for now and.

B

So so this one would be like periodic periodically kind of uh gonna do the check? Yes, it's a yeah, good man, heavy okay, I guess this would.

C

Live so okay, so this would live as some external thing, either as a psyched, CSI sidecar or like just some external controller, yeah.

B

But we also have an old component right, so that has to be on the node every node actually wrestle, because this is actually check the mount points and stuff right right, so I think especially enough a local store. Definitely you need that right, even for remote for never store use. You also need that you need to check if.

C

And also that for CSI, everything has to go through cubelet, so.

B

Yeah, so that's the thing, because I thought we are not really planning to add this thing into part of cube later, but I mean, if that's the case, because also we so try to add it there. But if this is outside, then it's going to be you kind of enjoy agent every node and then but then, if you look at at the interface of I, was trying to design there.

B

It's even needs a lot of information, definitely so whatever, if you believe like the or the past, it needs to have those in order to you find out exactly if the you know, the mount point is still the.

C

Same as before,.

B

C

We have the I/o metrics stuff like how many bytes available and use and stuff, and that's also kind of done directly by cubelet. Okay,.

B

So, okay, so you think that actually, this is possible to add this as part of Keyblade. Okay,.

C

B

Look into that yeah.

C

We can look into it, but I also know at the same time they were trying to insignia was trying to sort of we architect the whole metrics collection thing to be like outside of cubelet Oh. Okay. So we might need to sync up with sig note on this. Okay.

B

Okay: okay, but.

C

Like there's a bit of a conflict, okay,.

B

Sure so, whatever yeah sig no decide, we can decided. Where is the right place for this.

C

For like, even if they move the current metrics collection out of cubelet like we would have the same problems than that like, we would need to generate to see all the CSI arguments and stuff which is usually generated by cubelet. So yeah.

B

So that's a tricky yeah so because I was looking so maybe okay, we can actually jump to the later part, which is the CSI setups looking at it. This is the this is the well this controller, but look at no rice. It needs all of this. Basically the same that you need for node like this stage and publish holding costs.

B

If you want to check not only that it is mounted and also mounted the same way as before, then you need potential need all of this, because it's if you just check this amount, a then yeah. We don't need all of this, but if you want to check if it's really the same as before, then so all of this I think most of this we can get in from TV for awarding attachment and from the the PvP VCC itself, but I think the past party will have to assume.

B

We know how cue blade construct the path we just constructed the same way. I don't know if that's a good way to do it, but otherwise we don't know the those pathway mm-hmm. So that's that's the same that we need to decide. I guess that so basically for for node check volume request basically I have you know everything that I needed this.

C

Is the signature similar to the node get volume stats.

B

C

One is the one where elects used and available bytes.

B

Okay, I need to look at that right, yeah having to pay much change that. So let me take a look up that one damn.

D

B

Not have this much because that probably assume needs already bounty to something. Isn't it I.

C

B

C

No identifier, okay, it passed in the args there's a volume identifier, I wonder if we can simplify it. Just by saying like we give the volume identifier in the args and then they return, and then the driver returns the status.

B

So I, okay, so I think that is possible. If you don't really want to know whether it's Monty that exactly the same way as before I guess, that's definitely possible. So if you're just just kind of saying whether just to check whether it is still not it or not, right otherwise, I I will definitely take a look of that. I was just thinking that one doesn't have all of this information, then probably it you can give you some definitely give you some information back, but it won't be able to check.

B

I got all the details which wipers.

C

Don't really keep a mapping to call you my path.

B

Right so it's mostly like you have to give.

C

All the inflammation yeah.

B

If you just yeah so, like I, think our driver, if you give a leave, if you give a volume context then, and with voting ID, we can figure out something but I guess every drive is different right, so some draw may need more. So that's why I thought? Okay, probably missed you, need to keep all of this information. Yeah, cuz I, don't know what the driver deal with this. They probably just don't save anything. Yeah I.

C

Would be curious as to what volume get stats is returning? Okay, because that that's kind of in the same boat right, it's kind of cold out of band periodic, and it needs to tell the driver. Hey, give me information about this volume. Okay,.

B

I was just thinking that Carly assumes it sort of mounted; it's probably not really checking if it's mounted, because that is the same way as before. That's you know only thing, that's probably, if maybe that isn't enough for the first face, maybe we'll try this out and I know, okay with the opportunity to cheat me more than maybe what kinda more than maybe that's a good if rest face make it single initially so yeah I'll take a look at the bottom stats, and this idea.

A

Here would be to call this call periodically mmm-hmm.

B

C

Mean literally, it's how we call get get Iost a today.

B

Yes, so if actually, if we only want to see if it's attached, then the the one that Devi added already has.

C

B

This one, basically just added a few more things if you want to check if it's exactly attached the same way as before, basically we'll need all of those parameters, but if we want it just to I, don't know if we want to say: okay, maybe it first face, we just make it simple, just to check its a pat or not assume. It's still saying, yeah I think just work that David's doing will cover the attache.

C

B

Think that's case, then we don't need the this. The only thing is you wanted to check if say like. Oh.

C

The the met yeah.

B

The problem is that other the error messages are not covered there, so we- if that's the case we can so we can't really expand that one anymore, because that one does not allow it.

C

B

Like what would.

C

You do with the error message, so the work that David's doing is that if he finds that he's basically adding the logic to reconcile so, okay, yes, if he finds, if he creates a driver, then finds that it's not attached and it should be, then he will go call attach so.

B

If that's the case, if he's doing that, and is this to mediate it because, if he's with, we are, if we are periodic checking if it's a cache and then try to you, reconcile yeah.

C

B

Is not needed: okay, okay, it's probably out yeah, maybe she's, just sync up with him: it's his work will be in window, 17, hope so, okay, okay, all right! So if that's case, we can actually just remove this one. So we will just need to. The other use case you mentioned was if volume got deleted, yeah yeah, so that one yeah so that we still need so. Okay. So here is the response.

B

So the response here we rich so basically check if it exists so attached, we don't need any more now and also and of this one. We also have the capacity how much space is available. Does.

C

That need to be controller side because we already have load node capacity, but.

B

Of this yeah, but this is from the node capacity- that's not really showing you how much he used. Isn't it it's.

C

B

C

Gives you used and available? Okay.

B

So we part, we may still need this for the block, but I will take a look of that one, maybe yeah. Maybe that can definitely maybe solve some of the problems. Maybe it's not and then.

C

For volume exists, I, wonder if we can handle that through the current list volumes. Do we when we call list volumes, do we pass in a list of volumes? We expect or is it just list volumes and it returns everything list.

B

Of loading that doesn't have a theater and remember it is a snapshot, it's natural has it, but mister volumes that remember does not have a filter, but maybe that's something we can add at like a volume ID so that well cuz. If we just enhanced list volume to.

C

Say linked list volumes on this all of these volumes that we expect to exist and then Liz Williams call returns like not found or something like that. Mm-Hmm yeah.

B

We could we could probably yeah I think that's actually a good idea. Yeah. Then we don't need this. Yeah only reason that we proposed this is because we usually want to hand over the attachment as well.

B

I, probably want to take a look of the nerve as well the issue somebody opens that I want to see to see if it's all cover here. So so the house condition that's something that the existing interface does not have. So we need to somehow have a wait. She provides some messages. That's the note annotation that we want to add to those pvt in case. Something is wrong, so I think this can be part of this can also be in Katt Williams. You can add us there, but then we wouldn't need like for each volume.

B

We will need a node ID. Who is this? Health condition is Lee Rainie, so otherwise we don't know where it's attached, because otherwise we don't know where it's attached right. If you don't have the note.

C

There, oh the list volumes response should have nodes now, yeah.

B

Yeah but I'm just saying the the one David added: that's like a list of nodes, I, there's no way for me to add a messages in there. That's one.

C

Yeah yeah yeah.

B

Yeah but just need to correspond to it because that's like a real array, so either I need to expand that thing or I need to add another field, so that I can correlate a node and the negative.

C

From like a controller standpoint, though, does it actually matter which node that error is on cuz, the node, the then the node health stuff should cover the house on the node on a specific, no.

B

It looks like it looks like neat has joined Nick.

E

C

Okay enough sleep because I think the problem is is like what happens if you have like an NFS yeah I get smelted like a hundred nodes, yeah.

B

But what but I mean every attach is different. You do want to find out. You know you attach to this node, but not the other node and what's wrong right. Otherwise, then it's not really helpful. I mean it just says so say it on this node and then some people, some error, don't really know which. Why is a problem right? Then? You still I think you still need to have those.

A

If we have one stand-alone call those all of Health, what is the input to? It must be as large as basically all those other inputs that we have, so it must be able to handle the input that we pass from out that you put that past for attach and I.

C

We can at least split it up into node and controller calls right.

A

And we could, you know Dhin controller.

C

That's true cuz list volumes is currently a controller call yeah.

A

And then we would end up with one status call on each side. Then one on controller, one on server or a Sarah want to note. In one controller, mm-hmm.

A

And the assumption would be that it's never called unless it's completely published, except for the new one.

C

Except for the node was we have.

A

C

That it's actually still published.

A

A

A

A

Right go ahead.

B

Okay, so okay I think we were just talking about this condition. I think I think we maybe need to add another fear there in list volumes kind of like a so basically more like a map kind of have a saying: you have a node ID, corresponding to some messages. Something like that.

B

So so this way we wouldn't know which, which one cannot be attached and why so I.

C

Mean so we're not really, we said that we don't need to handle it. The attachments you're right because David's uh well.

B

That's true: if even there was a, if actually, uh if there is already reactions happening, then meaning they will will never have a voting. That is, we thought it's urban I thought it's a tight and it's not attached, because the communities will keep trying to success so basically meaning we don't have any who shouldn't have any problems yet. So, if that's the case, maybe yeah it's true, okay. So let's.

E

Handle the story dramas which is support, mounting attached, Modi.

B

Attached, what is that that are now sharing? That's.

C

Why that's why the nodes field in the list volumes call is an array of node IDs because of the multi attach types.

E

Yeah we used to try to find a way used to try to put it at a distance and attach checking to the know Deacon to the node option. Obviously interface bug, but since we have the controller RTC's, so I put it here. I.

B

Think we still need a node. Obviously we still need the check node condition, but this controller one we maybe can just enhance the uses, seen list volumes, maybe just to add a volume ID there so that it can filter. You know what volumes we want to check.

B

So we don't need this is attached. We don't need this. You don't need the house condition. So basically, we just want to check whether it is still it's still there and then given the capacity where there is a mr. Sandra's, actually capacity, information on the notes side, so I'm going to take a look and see see if that one is enough to cover this.

B

So this mainly is just to check if the warning is still there.

E

If the story they I first had, mmm could we get the right capacity on the notice, ID yeah.

C

I think that it's up to the driver to implement the capacity the CSI call, but if it's a file system type, then you could just well, you can get both. You can get the block device capacity and you can get the filesystem capacity, although I think currently CSI.

C

The CSI call expects you to return the filesystem capacity. If you its filesystem type,.

B

Okay, so maybe that's something can look into and see if we need to enhance that one.

B

B

Okay, so for this one just to make this: oh this one, we need to sync up with no, no Tina see whether whether we should make this a cubelet call or outside so and then we may not need all of this all of this parameters in the beginning, we'll see if the I think the node gets. That's right that one take a look at that one.

B

B

Yeah, so we probably don't need all of the supreme interest.

B

Okay, so let's go back to the CSI yeah, that's me so, basically they're two, and if what they're alpha or the node for the note check volumes, so we also want to other than to find out whether it is among data. We also can find out other informations. Like the you know, file system, corruption, those type of information. We can also get from the note check volume, so yeah.

C

That sounds good to me. I guess one question in general for, like the API point of view, hmm do we walk.

C

Define specific types of error, codes, yeah.

B

We can, we can maybe try you define some common ones. The.

C

Other thing at the end, if people want to have a custom error, okay,.

B

Yeah, because there are a lot of those and probably can't cover all of those yeah, we can yeah. We can identify some because.

C

It's not the type of error like we might be able to reconcile.

B

So in the this reconcile that David is working on, basically it's going to try, no matter what arrow is right: yeah.

C

B

Okay, so maybe that one you know you're, for we can figure out what kind of arrow that is a reconcilable. Maybe you know we could some of those. Maybe it doesn't make sense to keep your I guess. That's, maybe in the future, not for right now.

E

Actually, the detailed app design interfaces is implemented by different story, drama corners. So it's hard fast to define all them message errors, but.

C

I guess it would be, at least from the CSI side. We can define some error codes based off of like CSI existing CSI calls, so we could say, like yeah, publish our stage error.

E

Yeah we're gonna define some common errors. Yeah.

C

And then maybe like I, don't know io errors or something or.

B

Yeah we can yeah, we can cut the the existing CSI our code and you know, come up with some common ones so from the kubernetes api point of view, since we're going to use annotation, so there won't be any new api's for at least for the first version, and then this will be based on whatever error message returned by those house check our pcs annotations just for alpha yeah yeah, because I think I think I mentioned this. You have the six code calls right.

B

Initially we thought we can use the the taint what in taints, but that one is kind of evolving. So we don't want to depend on something that is still I, think it.

C

Will be good to sort of prototype using annotations?

C

Eventually, when we do add, when we want to add some sort of like reconciling thing to it, then we probably need to make it like part of the API yeah.

B

Yeah so I think, because the devoting 10 days also have other use cases like the data populate or so maybe that will come up when we discuss about the other populate is. This will be another use case as part of it mm-hmm yeah. So we definitely will still think about that one. It's just me if we depend on that mhmm who pretty can't started working on the season controller part.

E

The tens is editing the opportunity- oh section yeah,.

B

Yeah we moved that yeah. We did more with that one there's a law we mentioned that we could use that when that ready, yeah I think.

C

Eternia, so so this out of tree like if we can, if we made the first phase sort of all out of tree, then I think I think some annotations is okay mmm-hmm. Once we have entry stuff depending on it, it needs to be API. Yeah.

B

So if the so, if we decide that we, the node Hardwick, will be caught by the cubelet, if that's case, then we probably won't need cuz tainting. Okay. He you.

C

Know the other challenging thing about the cubelet stuff is today shoot, let's not allow to modify PVC or PV objects. Oh okay,.

B

But what even that even this type of a tank is okay, so it's like you need to add a new, oh wait.

C

Actually for the file system resize, it needs to update the PVC right.

B

Yeah the control, but it's the yeah, actually does yeah the node resize. Actually. Well, yes, updated the PVC. Yes,.

C

The I know in the past security team has been pretty strict about not allowing cubelet to modify objects. hmm Okay, that might be another hurdle for the node part of things. Okay,.

B

C

B

Can we can we can make a minute external agent if you can make it in there? It's just that. You said that there are 20 more the magics part out right. That's why we want you could try to make it in there and put it won't happen. Anything the.

A

Attached air I saw that you had.

B

Attached yeah, but this is since we said we, we maybe don't need this anymore. This is just an example but like, for example, this one. Those are just some examples. Decay machines can be anything oh. This is the this is the taint basically yeah, but if we're annotation right now, we don't really have a fixed format. I think it's a it's like very general, and then the driver can put whatever miss you yeah they have there, but if once we defy some common ira code there, maybe we can well steal the rmsd part.

B

I think they probably want you have their own error message ray, but just the error code maybe can be set a notation to have some error code like.

C

I think for the attach volume attachment and the snapshot are stuck. We we define like a specific type for the snapshot, errors right, yeah.

B

It's like a CRT, just watering snapshot yeah, so we don't find.

C

A special I CRT.

B

For that, okay or something sure sure, okay, but then we will be adding those information to the relation.

B

Where are we going to keep this.

C

I guess for like the initial phase, if we want to use annotations, it's kind of hard to encode stuff types, yeah.

C

Maybe I don't know, but.

B

It can still have what we can make. It is like a certain format right, so the annotation will have a certain format so.

C

Like, like that,.

B

Maybe you can't hear the.

C

Key is the or loss of that, like this value is also an app with key and message. Yeah.

E

Hello Vera is adjacent stream yeah. It's.

B

Just because there.

C

Could be lot of arrows mm-hmm so.

B

Like just first.

C

Part, the first key like attach error those could, we could define some sort of well-known types: some code, some error code, yeah.

B

A

You're trying to think of the case where we're going to differentiate based on user versus.

C

B

Different arrows have no.

C

I mean I was thinking more like in the future phase, when we can actually build things to reconcile these errors. It would be good to be able to group types of there's things and like if it's like an I/o error. Maybe you could run some like I, don't know file system, fixer tool or something if it's a provision error? Oh I, don't know if we want to reconcile provision but I hit create a new one and you here's your old volume, but here is a brand new one.

B

Toleration, oh that's, that is for the for the teens. Actually, that's a yeah.

C

I think that was like something like you can as an application. You could say I tolerate this error, like I think the original idea is was, if you tainted the PVC, it would prevent the pod from getting scheduled to it and as a pod, you might say: oh I, don't care about this error go ahead and schedule me anyway. Okay,.

A

Find a way to do that in the back yeah.

C

There's I think what they've done in the past is like. They have an admission controller that adds like a blanket toleration to everything.

E

Only for money to pad, but do we really need the toleration if the PPP's aren't heresy clearly can tolerate the error.

C

Depends, if maybe to fix the error they want to deploy a pod to fix the error.

B

At least I think for the initial use case for the wadding taneous because of their data popular right, so that the there will be like a data. Probably the pod will still be able to attach, and the cop you know copy data over, but a deposit should not be using that according something like that, I think. Initially, that's.

C

Mm-Hmm I think the other thing with tapes is that you can also specify if you want the pod to be evicted or not. So maybe you can buy a toleration that sounds like even though my disk has IO errors. Now, don't evict me I still want to. You know, continue running and just get like I/o errors in my application and handle it there. Maybe hypothetically.

B

Okay, I think we have cancer.

B

The whole document.

B

Yeah so basically Yoda the main thing is a capturing in here and then it goes through other. The CSI are pcs, so we'll make those changes. This.

A

Would be a good feature to try out the new alpha beta GA policy.

B

A

Mistake that we made with the resize okay.

A

There's a bunch of problems with it and it's hard to fix it with that in the backwards compatible way. Okay,.

B

I need to take a look and not sure how to make that in. Let's take a look at your your.

B

Okay, okay sure take a look yeah. We couldn't.

C

And also uh maybe like just to summarize the main action items prior I think the controller side.

C

We should investigate the CSI list volumes call and we can potentially just extend that mmm-hmm for the node side, we're going to have to investigate a lot of stuff the whole like metrics collection model and.

C

Permissions to modify for cubelet to modify objects.

C

B

That product we need to find out frantic node. What is the direction because yeah I don't know it make sense, was to try to add this when they are trying to move everything out right.

C

That's right so.

B

C

Yeah and then just on the like high-level use cases part just me moving more bigger picture. Yes sure.

B

Yeah I, actually initially thought I was gonna put together. Something I then forgot about that. Okay, so yeah welcome that so for the CSI. So okay, so once we update this, we for the scenes that part we can probably go to the CSI community meeting to propose at least enhancements in list volumes. That's.

A

B

Could at least get that started right? Yes,.

A

We have kept approval here, yeah.

C

Yeah, can we, if we want to enhance list volumes, can dad go through alpha beta GA I? Can we alpha afield? Oh I,.

A

Think there was room for that: I think it was just a decorator tag, but I got to double-check: okay,.

B

Yeah for the node definitely notice, and you have to see them definitely done. Yeah.

C

Or maybe even no get volume stats, yeah.

B

That will be using the exist. One.

B

Yeah I'll try to use the existing one.

C

Okay, cool thanks for putting this together, hey.

B

Thank you all right, bye.