Kubernetes Storage SIG Miscellaneous, 9 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: CSI Community Meeting 20201209

Description

CSI Community Meeting - 09 December 2020

Meeting Notes/Agenda: -

Find out more about CSI here:
https://github.com/container-storage-interface/spec

Moderator: Michelle Au (Google)

A

All right start.

B

B

All right today is december, 9th 2020.. This is the container storage interface community meeting. um So we'll just go down our agenda. For today I guess haman. Would you like to start is hamat here.

A

B

Okay, maybe um then ben we can go down your items first and we'll see if hamach comes back.

C

Okay, yeah, I imagine he'll he'll turn up a few minutes late, um so uh my action item from the meeting from last week was uh to create a two-hour tracking issue, so I've done that. um I just started throwing stuff in there that I was able to think of.

C

As I outlined and at the top message there, we don't actually plan on a csi, 2.0 spec, but if one would ever happen, we have a bunch of ideas that we'd want to put put into that version. So this is a place to collect them so that you know, if our future selves decide to do a 2.0 version.

C

We have a laundry list of things to go. Do at the same time.

B

It sounds good, do you want to actually go through and discuss these no.

C

No, I I wanted people to be aware that there is a tracking issue now, if you have ideas, just stick them in here. um I don't know. If we'll have a lot of discussion but um yeah. The point is just to have a place to write them down. So they're not forgotten, because because we do have a tendency to be like oh we'd,.

D

C

That in 2.0, but then we forget- uh and we forget, to write that down and we'll probably forget about it. When 2.0 comes around.

A

So basically, all the issues will be added to this. One issue.

C

Yeah, the idea is to use that one issue as just a laundry list um and then it.

E

Would be better to use like a project board with that look.

C

Well so so we had talked about, I don't I don't know if you were there last week, shane but.

F

Yeah, probably I forgot.

C

That idea came up and, and the thing is we don't want to give people the impression that we're actually planning to do a 2.0 version, because the whole idea is we'd rather avoid it, but if, if it ever becomes unavoidable, then we'll have a list of things that we would like to knock off. At the same time,.

A

Okay, yeah: that's fine.

B

All right sounds good thanks for starting that um next you have the bi-directional secret handling.

C

And I'm sorry assad isn't here for this one, because I wanted his opinion in particular, but uh all this for everyone else's benefit. um The way secrets are done in csi.

C

Is that, like the the co holds all of the secrets and passes them down to the sp in every case there there is no existing rpcs in csi, where the the sp generates a secret and then hands it back to the co, and I was thinking that might be useful, especially at volume creation time to have a way for sps, to generate secrets and inform the co of them in some return value and then have the co store them and- and I the thing is I I wasn't around for, like maybe the first six months of the csi api design.

C

So I don't. I didn't, hear the discussions that happened, and so I was wondering if anyone is here who remembers why secrets only ever flow from the co to the sp and never back in the other direction.

A

One of the things that we talked about early on was to try to avoid turning the co into a database for plugins right um and this you know, and so we we went back and forth a lot. You know context and attributes fields, and um you know trying to find a happy medium, and where do you draw the line? As far as you know, data the co is going to be guaranteed to store on you know on behalf of a plug-in so.

C

A

The two, the two.

C

Places where we do use the co as a database is for the the volume context and the published context, which is information that the sp passes back. But none of that is treated as secret right right.

A

Yeah- and I think we at one point we had talked about- maybe putting sensitive information like if we had to put sensitive information somewhere, you know, would it go there? Would it go somewhere else and we thought well, you know in the spirit of avoiding you know, over engineering, um let's just build what we need right and if there's a use case or a set of use cases that strongly argues for secret information um in those fields or adjacent to those fields.

A

Then maybe we tackle it then so it sounds like it sounds like there might be a use case or maybe.

C

Yes, regulating that there.

A

Are use cases no.

C

No um so like the reason this came to my attention is because we were doing a security audit of our own csi plugin and people discovered that we were generating chap secrets in the plug-in, passing them back to the co in the volume context, and they were getting persisted on kubernetes unencrypted and they were getting logged to the logs. Like the secrets were people were like well, this isn't: okay right.

D

C

We were like well, okay, so if we want to generate the secrets in the sp like there's no way to get them stored anywhere safely, they're going to get logged they're going to get stored in unencrypted forms.

C

So the only solution is, you have to generate it on the the co side and then push them in and that's limiting um because, like who's, you know it's that's work for an admin to do or for a user to do, and you can't just make it happen magically um without, like going outside the csi paradigm and like having your csi plug-in talk directly to kubernetes, to set up secrets which we should not be encouraging. So like yeah.

C

If there was a way to as part of the volume context or part of the published context, also pass back secret pieces like secret volume, context and secret published context, then we would be able to solve these kinds of problems.

B

I guess um you know if we want to consider adding this to the spec. Do we know that all cos are gonna? Have the ability to store uh secure, sensitive data.

C

um Well, we have ceos passing secrets in which is of course, an optional capability. The ceo doesn't have to avail itself of that, but if it is, then presumably it's keeping that information secure somehow.

C

I mean I I don't know if we have generalized requirements on ceo's ability to actually keep secret data secret. I thought that the whole point was at the csi spec level. We're only saying this is this is something that should be kept secret, but how you do that is, is unspecified you could do it very poorly and still meet the spec.

C

But, like the things like you know, the published context just getting logged to a log file like that. That's a reasonable thing to do, because there's not supposed to be sensitive information in the published context, but you know, if you want to have sensitive information in there, then you have a problem.

C

So I I don't know I just I wanted to find out about the history and then and ask like if we did have a proposal to return secrets from create volume and return secrets from controller publish volume. Alongside that volume context and that published context, would there be any strong opposition or would we just have? Is it just a matter of designing it and working through all the details.

A

Yeah, I'm going to see if I can find. I think there was a pr where there was some discussion on that, so I'm just going to spend a few minutes kind of digging around for that, because I I.

C

Did notice that, like down at the the grpc machinery level there there is the ability to have secrets in return fields and the the grpc client does scrub secrets from logs um in both directions, and so I somebody at some point was thinking about secrets being bi-directional, but then we only ended up implementing the one direction um so yeah I just if it wasn't going to be. You know too big of a burden to just extend the specs, so you can also return secrets.

C

um Then I I'm not sure that I will propose something, but um what we're going to consider it.

C

I mean there would be a lot of work on the kubernetes side because then you'd have to you know in the volume attachment you'd have to have a volume, attachment secrets or something- and you know extra yuckiness.

A

Is this anything that's been discussed um by sig storage, or is this the first place it's coming up? I.

C

I'm bringing it up here for the first time, as far as I know, um just.

B

C

Because this is, this is a csi sort of design idea, and I didn't have a history.

B

So, are you thinking uh volume just volume create, or are you also considering other calls.

C

Create volume in the controller publish would be the two pla. Those are the two places you can currently return data, so the proposal could could be you. You may also return secret data in those two places and then, in addition, you know so in great volume, kubernetes shoves, that into the pv, so you might have to also shove information into a secret at you know, after create volume and then controller publish generates the volume attachment object and you might also have to have a secret there.

C

B

Guess do you know of a reason why a secret would be generated at the attach time and not the just the create time.

G

C

Perhaps if the secret was node specific, because at create time you don't know which nodes it's going to be published to and if you want to generate a different secret for every node so that you can fence nodes that get compromised.

C

You might want to be able to generate per node secrets at a published time so that you can revoke those secrets on a per node basis. When there's a security.

C

B

All right cool, I mean it sounds like um james you'll. Look at the past issues um to see if there were any major issue issues that we discussed in the past about this and then um I guess ben you could work look into if you actually need.

C

This and work, if there's openness to this idea, I might you know, go to the next level and actually design it and make a more concrete proposal. I just wanted to test the waters.

B

First yeah at least it sounds like so so far. It was mostly just because we didn't have the use case, for it.

H

I

C

I have a use case very similar to ben's, so definitely plus one who.

H

I

C

Nicholas nicholas okay good to know.

B

All right um looks like haman is still not here. um Was anyone able to ping him.

C

Up come on: where are you I'm pinging him right now.

B

B

C

Slack says he's online, but no media response.

B

All right, um I guess, were there any other topics that anyone wanted to also discuss.

B

Did we um conclude on all the uh previous meeting topics.

C

Except for the ones that are in haman's name, yes, yeah.

E

C

Think I think those two that are in juan's name were carryovers.

B

uh Oh, what about yawn, restoring.

B

I guess we kind of concluded that we need to push this down to the plugin.

C

Yeah, this is one of those things that could be made better in a 2.0, but given the way it was written in 1.0, there's no good way to to fix this. It's just.

C

And I captured that in the 2.0 tracking issue, I mentioned that if there's ever 2.0, we should change the way that we do restores from smaller size to to have a clear capability.

B

And okay, um let's see from two meetings ago, did we conclude on this 128 bytes issue.

A

uh Yeah, I think so.

B

Okay, so then we just have to go, um send out a pr to change. The spec is that right.

C

What was it just for some one specific field? You were going to change it, or were we going to try to change it in general,.

D

So there is a conclusion.

D

C

Paths node id yeah either the danger is, if we, if we change the spec uh cos that are talking to older sps, may attempt to send a longer string and the older sp might be unable to deal with it because it conforms to the older spec and could get an error.

C

So it's not that we want to prevent sps from accepting longer strings. It's just that cos can't depend on sps, supporting longer.

I

I

B

C

I mean I I I would be very surprised if, if there were sps out there that like barf, if you send them 129 byte string because that's a weird thing to do, but um but that's what the spec says. So you never.

C

B

B

um I guess if uh if haman is not here and we don't have any more topics, then we can end the meeting. I guess we.

C

Just replied on slack: he just replied on slack. He forgot about the meeting. Oh okay, maybe he'll, join yeah.

B

We'll give him maybe a minute, but I think there were definitely um some of the topics that hama wanted to discuss. I think, could impact windows potentially so? Oh.

C

B

C

The group stuff yeah.

B

It's good that deep is here.

C

um While we're waiting for him to join, should we talk about the timing of the next meeting.

B

C

It's supposed to be in four weeks right, that put us comfortably after the holidays, with no conflicts.

B

I believe so. um The next meeting I see is january 6th.

C

B

There might be some folks still maybe on holiday, but.

C

You made it him on it.

D

C

Welcome yeah, we were just waiting for you because we finished my topics.

D

Right so uh we didn't finish, uh I know I was supposed to open uh at least a draft of a pr for gid handling and I didn't get a chance.

D

And uh the last time we talked about our uh just pull up the our conclusion was uh that uh we can add it as an optional capability, but we we have to kind of discuss the we didn't have conclude quite yet. Actually.

A

D

Okay, so new capability for driver to advertise, it suppose efficient application of gid, but and if available uh cubelet will defer to storage provider and so provider must handle. Now the question that we had was like: uh how do we present it into the? How do we represent this? In the rpc, like node, publish volume, node stage volume? How do we represent that data like? Should it be string, or should it be something else.

C

Well, there was, there was the question of like if it's windows, it's not going to be an integer right, it's going to be a string of some kind right and there was like there was a question of do. We want to have like some sort of a union struct where you know you get something on linux and you get something else on windows and we don't support anything other than linux and windows, because one of the nice aspects of the current csi spec is there is nothing that is windows specific in it.

C

It just kind of works on windows, which means that you, you could, in theory, add support for a third operating system without changing the spec.

C

But the moment we start putting like os specific union structs in there, then you lose that property.

D

Yes, that's true, but we don't have anything linux specific also, so.

D

B

I guess, though, is there like someone is at some point down the stack. Someone is going to have to parse these strings and make it linux or windows.

D

Specific yeah like read this uh gids and make it as a mount option string like the whole option string. However, the the storage provider supports that I mean it's, not linux specific. It is how the how the storage provider implements it. Like two service providers may may compose different strings actually when they actually apply that as a mount.

C

Option well that what matters is the the client that will ultimately be accessing. The storage right like a windows client will be representing groups as windows groups, which have names and sids, and a linux client will be representing groups as a list of.

C

Integers and yeah the storage system may or may not be compatible with with one or both or may not care.

D

Yeah yeah, but but I was just saying that the the string formation like how it applies the mount option and when the mount command is issued, it depends on the both the operating system and the storage.

D

D

D

Our options are, it seems to me that, like we could- uh and we also wanted to to keep the possibility open for passing, uids pass it now or like pass it later. But it's something you want to leave it out, but but if we have two different fields, optional field, like say, let's say string like uid and gid. Definitely that works or we can hide it behind.

D

uh I think michelle suggested the idea of using any uh like one off and put a in the type so that, but that that leaves out the problem that I think there was a problem that ben was talking about. If we use one off, then yeah.

C

If we use a one of them, we have to specifically enumerate the linux way of doing it in the windows way of doing it. If we ever want a third way of doing it, we need to modify the spec.

B

Like I, I think we we kind of have to enumerate it right, because if we just give an opaque string, the plugin needs to know what format that string is going to be in to be able to parse. It.

C

Well, I mean I don't agree like the way we handle published paths and target paths is that it's defined to be a path, that's meaningful on the system where you're running and we know that on windows paths look different and that's not a big deal right. You just say: oh it's a windows path, so I'm going to treat it like a windows path, but.

B

I guess with like use the same tricks.

C

And say it's a group id and you know that on windows group ids are strings. That look. You know like windows, group, ids and uh and just sort of leave it. You know a little vague enough that you could bring in another operating system without having to rewrite the spec.

B

I think the the issue is saying that some storage providers may not um use the like gid directly like they may have some way of translating gid into something else.

C

Yeah yeah, so this is the point I was going to raise was that I I really want to see us leave enough flexibility that like if, if better technology comes along, that we can solve the underlying problem in a better way. So, like kubernetes or not kubernetes, any co should be able to express its intent clearly so that if the sp can do it better, you get that better result.

C

But if it can't kubernetes can use the hacks that it uses today um and then, if a if in the future, we have a better hack that we can still use that and and in particular I'm thinking of, like you know, uid shifting mounts or something along those lines coming along that we will want to use instead.

D

So the the the gid like, for example, the what I was referring to the gid. If you, if you pass it a string, the source provider will want to compose the mount option like hyphen gid, equal to the whatever the string that was there and some other story provider will compose that differently. Like hyphen group.

A

D

Or something else they may not call it gid or they may call it something else and uh yeah.

A

D

That on the mount option, so uh so not necessarily interpret the jit differently, but it's like how they use it is is this could be different like like ben also, that also leaves us a possibility that a storage provider may itself change the permission.

D

If suppose it doesn't uh support like applying gid's amount option, then it can maybe change recursively ch on chmod file and apply it, which cubelet does it currently. But if you pass gid to the storage provider, there's nothing stopping storage provided from from you know like.

C

Yeah and the storage provider may have a way more efficient way of doing that. Recursive challenge right, like they could do it server-side instead of client-side, um which could be a lot faster. I I don't know I I guess I guess what what my hope would be here. Is that what we get the way I see this is like as an optimization and like it. We should allow the sps to advertise that they are able to optimize. This particular uh or opt, have an optimal solution to this problem and then cos can use it.

C

But then I want to make sure that like when, when there's an even better solution that we can just stop using this and switch to the even better thing so like let sp say that, yes, I am able to do something better and then let cos leverage that if they want to, but then, if we stop wanting to because we have an entirely better way of doing this, we can just stop using the capability and go to another way. So so we want to keep the existing way working.

C

We want to add a new optional, better way and leave the door open for something even better later on, if it, if it ever comes to the linux kernel, yeah.

D

Okay, but I just want to emphasize that it's it's not an optimization for some drivers like like, for example, for azure file, as we talked about the driver is broken without like if it cannot pass gid to to note uh note stage or not not published, then you cannot simply see h on chmod files. Actually, cubelet cannot do that because it doesn't support these unix extensions and the files uh the volume will be unreadable and writeable by the part. Actually, so it's not just an optimization for these this, this class of driver.

D

It's like a necessity.

C

Well, yeah, but that that suggests that, like any co that is coded against the 1.3 spec will is going to be, the driver will be broken for that co right. Basically, this the sp depends on some new feature and that feature will only be present in csi, 1.4 and later, and so older cos simply won't work with that driver like is that, is that an accurate statement.

C

Like we're basically saying that, like some drivers are unable to function unless we enhance the spec and there's no way to do that back in a backwards compatible way, so these new drivers will just only be compatible with versions of the spec that include this change. Well,.

D

They they are comfortable like as long as you're running your container as root.

C

D

They work as long as you're running as root, but the moment you choose not to run this container as root you run into these problems.

A

Okay, so I'm I'm looking at the um I'm looking at the api and we've got mount flags kind of nested under volume, capability and volume capability is used throughout the spec in a whole. Bunch of calls right controller calls no calls.

A

It is it's part of like the get capacity request call and what it seems to me like what's being discussed here, is that we're somehow going to augment we're gonna we're gonna first class, um some kind of credential right or identifier- maybe maybe identifier is the better word. I don't know for a user or a group or both, and I'm I'm wondering how that fits this model, um because you don't really know right like a uid or gid or something in the context where all these calls are possibly being made right. So well.

C

I I agree with you first of all, but like we don't need to put it down at that level. We could put it only at the level where we do know it, and would that.

A

Be adequate to say that, like the like something like supplemental amount flags, where some things are first-class fields like I'm just trying to get my head around what this looks.

C

Like we don't need to think of it as a mound flag, we need to think of it as like. Here is the uid and gid that needs access when you do the node stage and when you do the node publish and if you accomplish that through mount flags great, if you accomplish that through a different mechanism, that's all that's! Also. Okay.

C

We happen to know that the existing drivers would do this through mound flags. But, as hemant mentioned like that, you could just do an efficient recursive chun on their storage back end and also achieve achieve the same effect.

C

D

And coming back to the volume capability thing, the the the reason we didn't want to uh put into the volume capability is volume. Capabilities are decided at currently at creation time and they are basically fixed for a volume correct, but the the gid and uid actually changes based on the workload.

D

So those mount options. Those flags are not not fixed and that's where the problem comes with that volume capability, as it exists, is, is not suited to represent these.

A

I wonder I guess I wonder if you've got workloads showing volumes that have you know kind of conflicting credentials.

A

You know how, whether it's the specs job to handle that or maybe I don't know I I haven't thought about this problem in a while, there's been a there was a bunch of discussion about ids some number of years ago.

C

Yeah one of the one of the basic problems is just like: if a workload just moves from one node to another, due to the way the container subsystem works, like the uid that that container gets can be random, right, yeah yeah, and so it's not the user's fault. You know it's just the way that the co works.

A

Yeah well, something would be to translate right, like I think julian at one point had mentioned some kind of nfs shim driver that they were working on. That was actually translating uids or something so.

C

Yeah and there's work at the lineage kernel layer to do like shifting bind mounts where you know the bind mount performs. Uid gid shifts if you access it through that mount, but I don't think that stuff is ever merged to the kernel. um So, like.

D

C

Are better solutions to this problem, but, given that we don't have them like, we have to do something.

D

um Ben there's also that uh the bind mount solution may or may not work for something like uh like like, for example, you cannot. uh The uh like the like, for is your file or samba. That doesn't have unique extensions, and things like that. So we don't know like how how it's going to work in future. So, but we need a solution.

C

Why I thought that the the one of the kernel proposals was like a bind mound that layers on top of your other mount and like performs the shifting in the vfs layer before it gets down to the actual mount, and so it would work with something like nfs but again, given that it hasn't merged. You can't really play with it and and see it that easily.

D

Okay um looks like james is dropping, so so I just wanted to like the week. I know sad and and both were like thinking that if we could enhance volume capability, but volume capabilities are fundamentally like unsuitable to in represent this actually because of their uh they're fixed once basically, volumes are created, so that leaves us with uh with like that feels on either fields on and not either, but but pretty much like feels on no stage and and and not published.

D

For me, I think I thought of different rpc call, but then the problem becomes like is because the mounting has to happen at the time. The note stage is called different. Rpc call doesn't make sense, because the remount many mount options are not applied on remote, so remote is not guaranteed to apply the gid.

D

Yeah, so only first mount can apply the gid we had. The similar problem with, I think, sl and x, where jan founder, that are playing on the first mod can apply ac linux options.

C

So so I want to get back to the the basic problem where we have drivers now, for which the existing csi spec doesn't provide a mechanism for cos to give enough information to sps so that they can get access or so that they can provide access to a container. And this is a windows. Specific problem. Is that not.

D

C

D

It's a right now, it's a like one. Specific driver is as your file running being used on linux boxes uh or our linux server. So as your file traditionally doesn't support unix extensions and as a result like ch own chmod, doesn't work and oh.

C

Oh great- and you said um it's a it's basically samba right, yeah, okay, okay! So so that's why I think I have windows on the brain. So um so it's samba and you're saying you can't implement any samba based driver correctly, because the channel isn't going to work and you need to have some sort of gid information communicated.

F

C

Yeah, so so this is, I guess we just never thought about samba when designing the csi protocol and and um to make it work at all yeah. You have to say like okay, your this is the the ui dng id of the process that that needs access. um uh I don't know this. This doesn't feel like a very backwards compatible. I mean I I I agree. We need to do something but, like I don't know how to do this in a backwards compatible way.

B

What do you mean by backwards compatible.

C

Well, let's say we let's so so what haman is saying is like, if you don't do this, it's not going to work so like let's say we we make some change and we we roll 1.4 version of the spec and releases a new driver that works with azure file or whoever's doing azure file driver that and it relies on this new thing. But if cos like, don't do the right thing?

C

It's it's just not going to work so like we have this new required field basically, and that and new required fields aren't backwards, compatible right like by default. It should be optional, so so we're going to have a situation where, like even after you upgrade like, unless you do some work, it's still still going to fail.

B

I mean in terms of backwards compatibility the the driver will still. The driver is not like breaking in behavior right. The driver doesn't work today. It will continue to not work. If the ceo doesn't support it.

C

Yeah, well, that's why I'm saying like it feels like there's a there's, a fundamental gap in in csi that we never addressed this and and the only way to address it, is to make a new required field.

D

um But is that so because then, then the not all drivers might want to support that or they don't necessarily need it.

C

Oh, no, it's the cos. This is information. The ceo must supply for this particular driver to work yeah. So.

D

C

Like and so if there, if there's any co that doesn't supply it, then the driver doesn't work, and then you have a weird, basically a driver that depends on us on the co operating implementing an optional feature which isn't something that we that we like. Is it.

D

I I don't think, there's a precedence for something like that, actually that the ceo must support something like this. This must supply this field. To, for this driver to work.

C

That's why I'm saying it doesn't feel back it's compatible because yeah this is a case of this driver, will not work with older cos period.

D

No but like people have been running as containers as roots, so.

C

Right without using this hack right like either the hack is acceptable or it isn't, and if it isn't.

D

B

Think, generally, when I, when I think about backwards, compatibility, I think of like something that used to work and then no longer works rather than something that I mean. If something just doesn't work and it continues to not work, I don't think that's a backwards. Compatibility issue.

D

Yeah yeah yeah, it never worked properly. So it's it's not like, strictly speaking we're gonna break it. It's just that we are gonna, introduce an optional field behind that is behind optional capability. We are going to say that if you support this capability, you must supply this field. Otherwise it's an option.

D

D

So I think that should be fine ben. What do you think like did? Let the driver define a capability and if it did defines it it implemented carefully, then it must support.

C

The comma supported.

D

Support yeah, the sp must accept that field and.

C

Yeah but but then the question is: what does the sp do when faced with a co that doesn't support the capability, because there's no concept of a co capability right? It's it's all sp stuff.

D

C

If the ceo doesn't doesn't know how to speak that language, it's just not going to work. Even if you do have the capability, the co will ignore it and you know it'll just just break. I'm.

D

Just trying to look into the read only thing, which is also then behind a capability actually.

C

But that that that is specific to the you know. If the ceo wants to get read-only semantics, it has to use that interface to to not to publish the volume. If this is like, if you want it to work at all, you have to do something unusual.

C

Okay, so it's not the co opting into a behavior, it's the ceo, expecting ordinary behavior and not getting it unless it uses an optional feature, but I mean I I don't it's not like. I have a better answer. I think we're just kind of in a bad situation here and we have to do something awkward, but I just wanted to make sure that we had a understood the nature of the problem. This is really uh like.

C

This type of driver cannot work without this additional information coming from the co, and so we just have to figure out how to supply it when possible, and maybe ad verbiage, to the spec that strongly recommends supplying this or something you know anything short of a.

C

Must specifically to address this, this kind of situation.

C

And then we still have to address you know what what do we do on if it's windows or if it's you know linux? I still don't have an answer for that, but.

C

Yeah, I I do think arguments to no stage node publisher, the right place to supply this information.

D

So for for the the controller published request, we say that just to indicate that sp must publish the volume and read only mod, so you must set this field false. If sp does not have the publish read only controller capability, this is a required field.

D

So, as um so, the field is required, but it's required in all cases it just. The ceo must set it to false. If it's not there either. If the capital.

C

That read only flag has been around from the beginning.

D

C

It's not something we added later.

E

D

E

In the kubernetes context, uh will this typically come from the storage class at the time of creation or.

D

No, it comes from the part, the the gid right yeah, it comes from the part. A gid like okay, the the jd could have.

E

Default right amount.

D

Yeah, when the part tries to mount it and the jrd could change, depending on which part is trying to use the volume.

E

I think in the azure file case, it's probably used to be documented that the storage class can also specify it, um but maybe yeah. The pod makes a lot more sense.

C

What would we do if there were two different pods accessing a multi writer volume and they were had two different gids.

D

Apparently, as it works with the with samba, actually so you know with.

C

Only if you use the mount option, not if you try to do the recursive channel yeah, because.

D

The regressive.

C

Channel will obviously clobber the first guy the.

D

Recursive charan, I think the ch on siege mod throws errors. Actually, if you try to do it as far as your file it just. I.

C

Mean if you try to do it for any sort of multi-writer volume, if the second guy comes in and starts shouting everything, while the first guy is still using it you're going to have big problems.

D

That is a different thing. The better one is trying to say that the ch on ch mode does not work. It throws errors.

D

C

For some, but I get that, but I I guess yeah the reason I'm bringing this up is: we have to be careful about how we how we mention what this field is supposed to be used for, because if we imply that it's acceptable for the bsp to do a recursive town in response to sting and gid at node stage time, then in in a multi writer case where you have two different writers with two different gids.

C

That's not acceptable.

D

Yeah yeah, then the yeah that would a that'll, be you shouldn't the sp shouldn't use ch own if it's a multi writer and it doesn't support like mount options. The mount options case works, but.

B

I mean even this is a problem in a single writer case right.

D

Yeah, it's a problem right now. Also right, like two parts are trying to use the same volume, single writer or on the I think.

D

Yeah two parts are trying to use the same volume and because we do see on on at the mount volume call not mount device call. That means that it's it will try to ch on the files, so they will.

A

D

Will either of the the part will be locked out of the volume.

C

You know all this suggests to me that we're missing something fundamental in sort of the the design of csi and how we support. You know: pods, gaining access to storage um when you think about these cases of like multiple users with multiple identities, sharing access to a volume like you, it implies the need to do some sort of a uid, gid remapping in the middle.

C

Otherwise no solution is going to be.

C

C

So we have all these work around so like. Well, if it's just a single pod, we can do this recursive channel and well, if it's multiple pods, everyone can just run with the same identity and all these different hacks to to to work around the practical limitations that we face. But we don't have a comprehensive solution that says: yeah. It's actually, okay, for multiple pods, on multiple nodes, with different identities, to share access to a particular volume and just expect it to work, because that's kind of a hard thing to do.

D

Yeah I mean multi-writer volumes. If you start seeing files, then it won't work like and the single writer. Obviously they don't work right now.

C

Yeah and so so, the amount option actually solves this problem but, like I feel like we kind of have to define the way that this flag works in such a way that the only valid option is to do something like a mount option or a shifting mount that that uh that results in the pod getting effective access without affecting any other pods. That could also have access simultaneously.

C

D

So we talked about that the efficient. If the driver has us efficient, we are for providing that access like we want to discourage users. uh Sorry discourage storage providers from implementing their own ch own chmod thing like yeah yeah.

D

I agree to that and we can add some wording into the spec for that actually and that's something we can like carry on the discussion in the pr. If, when I get a chance to open it so but uh yeah, that's us, that's fine, but like if we have to say that okay, if we had to, I think the the so the first thing that we agree is that we need a capability.

D

Is that correct or we just can just introduce optional field without a capability.

C

C

Under what circumstances would you not want to supply it? I mean this is sort of putting a burden on the cos. The ceo has to be ready to supply these this information, and once the co has the information, why would it not want to supply it.

B

How do we know if a sp actually took action on it.

C

um That's where the capability would come in yeah.

C

If you need a signal from the sp back to the co that that the the sp is using those fields to to grant access as the spec suggests, then the co can disable it whatever workaround it was using.

C

But if, but if this sp didn't act on those fields, then the ceo might continue to want to use whatever workarounds it has.

D

Right so, okay, so.

C

That's what the capability is for, but but the cos should still supply the values in all cases, there's no reason not to pass them in.

D

So the thing is that if co already did see a ch on ch mode, then why should c or supply that field? uh We don't want stress providers to mistakenly, like you know, like again, double recursives do do the same work twice.

C

uh Wouldn't it be the case that once once the sp started asserting the new capability, that kubernetes would never use that hack again, if it's all the capability was, was there and would just leave it to that particular driver. So for a particular volume, it's always owned by a particular csi driver and the capability should be unchanging.

D

So if the community is not going to check it's, the only thing that co will do is like okay, if it will supply always, but whether to use the hack or not depends on the presence of the capability like.

I

D

After the node stage are not published.

I

A

C

I

D

Okay and uh okay, and how do you want the format to be that's the second thing we could not agree on like.

C

I'm comfortable with a string and some documentation that explains that it's going to be os.

B

B

I think to decide on this: we we need to actually think about the fields that we need for both linux and windows and see if we can make them a common field or not, if they're different enough.

D

I don't know anything about windows actually and uh deep. You there right. So we don't. We are not we're running everything as root on windows. Right now correct.

D

So in future, like I think, ben was also suggesting there's something called what was it called sip or seed? I forgot.

E

Serbs yeah the ideas.

D

Yeah, okay, so do you want them to be supplied here? As as as like to note publish note stage.

E

Yeah I mean having that option would be great like as an opec string, as was being discussed.

C

Well, it wouldn't be an opaque string. It would be a string to be interpreted by the sp, but with an understanding that it's should be interpreted based on the os where it's running.

E

Right and then yeah passes the string that the publisher, the that the note agent for windows can just translate and pass it in whatever it is. Yeah.

D

So at some point we'll need like, like part spec, to have that field or somewhere like so that part can know. The co can know what's said to use to to supply it to the to the sp.

D

But parts don't have any feel like that right now, right so.

C

Wouldn't it be part of the status.

D

It should be in spec. Ideally that's where the g id is. Is the group id if this group is inspect not in status, so.

E

uh Yeah, so the the field, that's there for windows is under like security context. Just like you can have the run as uid. There.

D

E

A similar run as username.

D

E

So but yeah there's nothing specifically for the group to be honest,.

C

But if you have like run as any then then the actual sid or gid or uid that it ends up running, as is something that only cubelet knows right.

C

So but that's okay right. If, if you have run as any in the spec and then cubelet picks one, that's the one that you want to pass down to the plugin. So you need to have access to it at the point. You're calling node stage no publish, but presumably cubelet can do that because it's the one making those calls.

B

Right, I think um so we are at the top of the hour. I think the actual format of the fields, I think we're gonna- have to need some more discussion on this. um Maybe maybe.

F

We need actual prototypes.

B

Yeah, maybe we can actually follow up um in the pr or in the issue, on the exact format we want.

D

uh One quick thing before we we just stop. So do you want this to be an alpha, thingy or like it goes as.

B

I think we should make it alpha.

D

Okay right, so that when I open the pr I have some okay yeah all right alpha is probably.

B

B

All right sounds good, um so uh hamad you have one more thing about adding secret to node expand. um Is it does it seem pretty straightforward, we're just making it like all the other calls.

D

Yep and there's already a pr opened by someone so yeah, I personally don't have use case for it and maybe other, but but there are, there have been use cases where people want to pass secrets to note, expand volume. Rpc call- and it seems it's optional- seems straightforward change. Actually, unless yeah.

B

Okay sounds good, um so I think the last the next csi meeting is going to be um on january, 6th.

B

Does that seem? Okay? Are there any um other urgent topics? We want to discuss to happen meeting before then.

D

I really like to get a solution to the azure file so that we can.

B

Yeah sounds good, so maybe if you can open a pr and then we can just start looking at that and discussing that, I think that would be the best way to move forward. Awesome. Okay, all right great, then I will end this meeting and start the next meeting thanks everyone. Thank you.