Kubernetes Storage SIG, 14 Mar 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Storage 20190314

Description

Kubernetes Storage Special-Interest-Group (SIG) Meeting - 14 March 2019

Meeting Notes/Agenda: https://docs.google.com/document/d/1-8KEG8AjAgKznS9NFm3qWqkGyCHmvU6HVl0sk5hwoAE/edit#heading=h.d9k1eyhevswq

Find out more about the Storage SIG here: https://github.com/kubernetes/community/tree/master/sig-storage

Moderator: Saad Ali (Google)

Chat Log:

09:12:31 From Srinivas Brahmaroutu : https://github.com/kubernetes/community/pull/3361/
10:01:12 From Hayley Swimelar : I do, but my mic isn't working

A

This is the kubernetes storage special interest group meeting today is March 14 2019. As a reminder, this meeting is public recorded and posted on YouTube. Today on the agenda. First, we're going to review the planning spreadsheet to go over the items that we've been working on for 1.14.

A

The 1.14 code freeze was last week on Thursday and.

A

Then, if you have any PRS that need attention designs that need review, please feel free to add to the agenda or, if there's anything else, that you'd like to discuss, feel free to add it to the agenda under miscellaneous as well. So, let's get started with zigge storage planning spreadsheet and go over that very quickly. First, we have the migration engine for CSI deeply able to give an update on this.

B

114, so mountain mountain attach already there's a final piece remaining in the external provision and that should be very pretty soon and.

C

B

Have started planning for the things for 150.

A

In this inline volume, migration is still ongoing, or is that for next quarter? It's.

B

A

Awesome well great work there guys. Thank you. Next up is the entry. Sorry inline volume support for suicide blood. Yes,.

D

So that APR, eventually got landed in 140 and got merged so and I was able to work with a early application in the host bath to test that. So I have a couple lingering issues open yesterday due to fixed couple bugs, but as far as the main parts, it's in awesome.

A

Great glad, thank you. Thank you. Next up is block volume, actually I'm gonna go ahead and start working. Some of these is completed block volume, support to beta for CSI misaki, or been any updates on that. This.

E

Is the the feature gate, murder, I, think the doc changed merged I, don't know about the PR five six. Five! Let me go look at that. One, no way, that's the enhancements patch I, don't know! If is any work to do there? Okay,.

E

That one's still open so I, don't know whose job it is to close enhancements. The.

A

Enhancement issue yeah.

E

A

Think the release team can help with that. Okay.

E

A

E

We're basically done here, Oh.

A

Awesome Thank You Ben.

A

Next up is adding support for online offline resizing to CSI a month, yeah.

F

F

That has necessary features as well, so I'm working on Doc's I have a PR, open and yeah working on and working on implementing. We have the support for X Files in the mock driver, but I'm working on adding support for the real drivers and just.

A

Awesome Thank You month. Do you want me to mark this is complete or leave it open for now, I think.

F

A

On next item was mostly design: lifecycle manager for CSI yon sent out an email about this about a week or so ago. To give an update you on. Do you want to talk about this.

A

Looks like yawns not on with us today is anyone else able to give an update on his behalf.

A

Okay, we'll skip over this and get an update next time moving. Our next item is moving topology to beta deep, any updates on this. This is topology for CSI right.

B

So I believe King's work pretty much handled all of this and Michelle moved it to beta sweet.

C

Yeah, so the all the entry work is done, there's still some out of tree work that still needs to be completed and.

A

Does that look like it's on track for getting completed in the next couple weeks? Yes,.

C

A

Cool so we'll leave that as started. Next item is alpha: CR DS, 2 beta sync.

G

So right now there was a bug fixed I'm. Still, looking think should should get a dunce and says you trying to figure out a few issues is the integration tests, okay,.

A

Do you want me to mark this as complete or leave it open so.

G

Let me show it: what do you think is that this can't be say we mark this one to be done and then bug fix what he Chuck's it pretty Lord. What do you? What do you think.

C

Yeah I think we can truck track the bug fix separately. Okay, all.

A

Right great work there next up is a design item secured volumes. I'm not sure Jing is able to give an update here. I, don't know if she's on the line, if not Michelle, do you want to give an update sure.

C

I think we're still exploring options that we want to pursue at this point, so we don't really have any in concrete at the moment. Okay,.

A

Next up is scalability testing, so the cap was close to merging last time. Any updates on that yeah.

C

The cap is merged, so we just need to start executing on it. I know: Patrick has been mostly focused on getting CI testing set up in for the CSI side, cars, so I'm, not I, don't think he's had a chance to start working on this. Yet, okay,.

A

This may be something that slips into 115 I mentioned, possibly.

A

Sounds like solid progress. Next item is redesigning the keys for per node volume attachment limits. This was a design only it's something that is required for CSI migration a month. Any updates on this so.

F

We have a design, but I need to write it down and I'll work on like getting it implemented in 1.15, so I just.

H

Have a designer.

F

And on next week, sometime.

A

Awesome, thank you come on next up is the pluggable in 10 test framework.

A

The most changes were in was the last update and the only thing left was documentation and examples, and he updates on that Michelle yeah.

C

So all of Patrick's changes are in and he's actually testing it out with the CSI host path driver as part of his CI set up work, I, know Matt and masseuse, and have also been testing this out with a CSI, NFS driver, I, believe so I, don't know it I think we're close to being able to like publish, published some example of how to actually use this.

A

Okay, but the docks other than the example are already done. I.

C

Think that the example is the docks. Okay.

A

That makes sense.

A

Cool next item is kubernetes conformance.

I

Pr's outstanding, but I don't expect any of them going into 114. Yet there is a king of test weights over the last two weeks, I think and we'll have to work using that we opened a PR in community 33, 61 I pasted it into the chat that is for promoting validations weights if that PR gets merged. That will give us ability to start working on the validation straight. So oh.

A

So I think that's okay. We made good progress here and hopefully we'll carry on this work for 115. Thank you. Three next item is local TV 2g, a montmichel yeah.

C

I think all the work has been done pretty much. We will we're planning on releasing a new static provisioner, but it's it's kind of orthogonal to hold G a thing so I think that could be tracked separately.

A

So I'll go ahead and mark this as complete, then.

A

That is awesome and glad to hear next item clean up storage, feature gate handling that was completed, no need to go over that new non-recursive volume ownership designed for this Matt Smith any uh any updates on this.

F

Yeah we'll take this up in on this quarter to come up with a design.

A

Okay sounds good. The next item is revisiting UID GID handling with volumes Brad. Any updates on that. Yes,.

J

I've actually got the design doc started for this, but were I haven't even fish it around internally, yet so the community, oh.

A

Cool next item is volume health.

A

And this was also a design for this quarter. I believe Nick, Ren and Shang were helping drive. This last update was designed focused on local storage, neat ad network storage CSI to the design. Any updates on that.

G

So I'm looking at is a from the CSI side. What are the things we want to detect and did you see if we want to detect those so what are currently available in a CSI interface and what we might want to add there so.

A

Yeah I think this will be an interesting design for next quarter. I have some ideas for it and some other potential uses for it. It might be as simple as potentially introducing a you know. Volume is healthy call and CSI.

G

Yeah I'm not sure, that's enough, because there's a lot of things missing if we really want to find out right whether the body is deleted or not. What is this your touch, but it's a Monte seems like there are a lot of things we need to find out so I, don't know you forgive me just to make one just add one interface or giving you can add multiple things that I'm not quite sure yet yeah.

A

Okay, so it sounds like good questions we'll carry on this design for 115, okay,.

G

A

Right next item is CSI entry, post g8 asks Andrew started. Working on these v1 was completed, the read-only stuff was moved to 115. Is there any other work that was remaining here.

A

C

Don't think there was anything remaining besides to read only stuff and some of the cleanup work.

A

Okay sounds good to me.

A

Next item is CSI out of tree automatic, automate, the side, car release and test process, so the cap was under review framework for consistently building and pushing to Quay it was done manually. This would simplify that any updates on that yeah.

C

So all the side, cars are now using the same, build and release tools. The only thing left is to set up CI across all of the side. Cars.

C

In order to set up the e2e tests.

A

Cool, and is that something that is planned to be completed in the next couple weeks? Are we bumping that to 115 I.

C

I hope to have it completed by then. Okay.

A

So we'll keep we'll keep you here. Oh great next item is refactoring cubelet driver registrar to.

A

Cubelet driver registration to the reconcile model, so this is there's a PR in progress. It did not make it into 1.15, but there's been there's a good progress on it and hopefully we'll get it in for 1.15 who's.

D

Working on that side is that you know.

A

There's a somebody in the community I think Tara something working on a medical review that.

A

A

Mean if you're interested in helping review that, let me know. Okay next item- is volume snapshotting. Any updates on this Shing.

G

Yes, so the capsule is still being reviewed. We are addressing the comments. Jean wants to set a meeting with API reviewers and to see if we could reach some consensus, because some comments are for different. Reviewers are pointing to different directions. Yeah.

A

I actually have a.

G

A

This yesterday, okay.

G

A

Comments are actually coming from trying to make sure that the design that we come up with the lines with the design that say gaps wants to pursue. Okay,.

G

A

Cohesive story kind of intend for that and I think in the next few either the meeting on Monday or the meeting after that. That Jing has four snapshots. She'll discuss that more at length, okay, so but yeah good progress. There next item is CSI library, moving the mount library to an external common repo that can be consumed by multiple places. So there was a large peer that got split up the first PR merged any other progress on this Travis yeah.

K

It's part of splitting it up into smaller ones. I did open a new one, but it didn't get in before code.

G

K

Since it's just refactoring anymore, until the freeze is lifted, but once that comes up, I've got a bunch of things queued up. Okay,.

A

All right, Thank, You Travis- this is good progress. Next item is provisioning capacity reporting for generic topology, which is also required for local volume, dynamic provisioning. We had no owner for this last time. It sounds like we're unlikely to have found one right before code. Freeze of that true Michelle yeah.

C

There's still no owner for this okay.

A

So we'll put this to 115 and then the sub path work. There is a PR merged. There is a node in ten test failures that were under investigation, they've.

C

Been fixed, so this can be done.

A

Next is post GA in tree topology work yeah.

B

I haven't had a chance to look at it, we'll take a look at it. Okay,.

A

That make sense. Next item is CSIRO tree moving NFS driver to separate repo last update, the initial PR was merged and there was outstanding work to add, update to CSI ad deployment scripts at CI CD and a bunch of other work, Mithu sonore anybody else, who's been working on this any updates for the NFS side of things.

A

Okay, I'll assume that is still outstanding, John. Any updates for I scuzzy.

A

Okay and then finally Fiber Channel, and you updates on that Patrick.

L

Made a PRG ever these tools and I have to see I see. Let's get that in first, okay,.

A

So PR is out and under review, need someone familiar with fiber channel to help review it, so it sounds like same status as last time has anybody are able to help review the fiber channel PR.

A

Okay, in that case, what we can do is Michelle or I can help review it and just kind of do a sanity check and go ahead and get that initial merge in there and say you know it's a prototype, and if anybody wants to use it, they find issues with it. Hopefully they can revise and fix things. We're gonna have to merge it without having any experts review. It then all right thanks! Medusa next item is moving online resizing to beta. This was designed for this quarter.

A

The kept was going to be presented to say, gaps or was presented to say gaps. They determined it not to be a blocker any updates on this month, yeah.

F

I think you're just going to be the stuff in the way the default mode in 1.1 15 and that's the design will be going it. There hasn't been any major objection from sig apps. So thus the design will be for saying.

A

Okay and I hope people in this say gun Durst and the implications here, and if anybody is familiar with the storage system for which this will not work, now is the time to speak up before this becomes the default. If you don't understand the implications of it, please reach out to her month or follow up on now one of the related issues. Maybe a month, you can post a link in the chat for folks who want more context. Yeah.

A

That'd be great and yeah if anybody has issues with it, now's the time to speak up.

A

All right next item is auditing, see it's the IPV PVC code, which I believe we really somebody already taking a look at any updates on this mission.

C

Yeah, so on the issue, there's like a detailed analysis of the code paths and stuff, we decided that the corner cases were pretty rare, at least for local PV, but we might need to come back to this for CSI inline, ephemeral volumes.

A

So maybe we'll carry on this work for 115 and do the same review for CSI.

A

All right, thank you and then. Finally, last item is volume. Expansion for stateful sets. This was blocked on how volume support offline resizing any updates on this a month, a.

A

Month are either.

A

Right have lost him. Okay, we'll get an update on this next time.

F

A

Worries: okay, so.

F

Yeah, the update is that we had to solve, or at least propose a solution for the offline resizing, just adjust the volume tap that only support offline resizing. That cab does not addresses that. So that's what is blogged on we'll work with see gaps, to unlock it and move it in 10 1.15. That's the plan.

A

Perfect all right, thank you so much for that update. Alright, let's go back to the agenda next up. There is an announcement from Brad Brad. Do you wanna talk about this sure.

J

So I'm for the cube con tu in Barcelona, there's a contributor summit and every well for the past few contributors, some it's a vest that cigs to either present offer presentations or just do general participation and I.

J

You know I know there is side view submissions for story later topics just for the overall cube con, I I, don't know how many you actually got accepted, I, think I, I, think notices actually went out last week or something, but for those people that didn't get accepted to but still planted on, attending cube continent, especially the contributor summit. I was looking at doing like some six storage presentations. So if people have ideas or an already plan, I'm being there, please send me, you know an abstract.

J

Can we kind of work together to maybe throw some Genda together for your summit? Presentations is.

C

This gonna be different from the storage day. That's also going on. Yes,.

J

This will be contributor oriented, so good stuff would be like CSI driver development operator development, but probably not I hyper-converged storage. That would not how we needed talking for this in.

A

The past Brad, the contributor sumit, has been kind of all the kubernetes contributors in a single room, the first day like on a Monday before Yukon, and then folks will go up and give presentation. So it sounds like you're, organizing, the sig storage platform, correct, okay, it's cool! So that's the context for folks. This is usually a meeting. That's held right before the day before cube con.

A

It's all the kubernetes contributors, it's actually fairly difficult to get in, because there's a lot of people that want to get involved, but they're, you know not ever there's not enough space. So if you really want to be involved, it'd be a good way to get involved.

A

And when you think about these topics, think about how they would be more widely applicable to beyond just sig storage to the wider kubernetes community. What are the things that we want? Everybody else is working on: kubernetes who's, helping develop kubernetes to be aware of I think there are a lot of good topics. So thank you for organizing this Brad.

J

A

G

That do you know how, because there are a lot of other six present here as well right. Do you know how much time we have her six storage, we.

J

Could negotiate that just depending on how many people were interested in presenting so I yeah? But of course we can't take all the time. But if we have a couple presentations we might be able to get him in.

A

All right and then the next announcement was the next meeting, for this sake is gonna, be March 28th and we're going to use that to plan for 1.15 so be prepared for that I'll try and send out an email before that to start generating list of topics. I'll create a new, a new sheet on this spreadsheet and you can start adding items there for things that we should be working on for 1.15 and we'll help review those at our next meeting. So prepare for that next item is is PRS to discuss that need attention.

A

Simon looks like you: have a PR, adding expansion for storage OS, and do you want to talk about that? Yeah.

M

I think it should be fairly straightforward. Just adding expansion to our entry driver, okay,.

A

And a month, would you be able to help review this yeah.

F

I'll take a look at it great.

A

All right, thank you and next doesn't look like there's any designs that need review. There's a couple of items: miscellaneous items from Sri: do you want to talk about these yeah.

I

Actually I think the himand esterday about online volume- resize- that's not that's in alpha stable, so I assume that we can push forward to beta in 115, and the second second issue is a separate one that essentially some of both of these things. We have customers who are interested in we are using Aakash is also on I, believe is available, so PVC annotations that are used today to set certain parameters for creation.

I

So there is a workaround for provided there, but there is no alter the Box experience for this, so it requires a design or a cap to if there are any ideas, yeah.

A

Hi, so this is something that is a little bit of a pet peeve of mine I.

A

What I want to underscore here is that the PVC object is supposed to be all about portability portability across clusters across implementations of different types of storage. So what we don't want to do is have the PVC object, decorated with fields annotations labels that are specific to any given storage provider. The PVC is a weight, a generic way to represent to ask for storage.

A

The things that are specific for a storage system should be on a PV object. They should be on a storage class that said, I do understand and realize that there are cases where a storage classes, opaque parameters are insufficient, and so what I've been asking folks to do is instead of poking a hole in the API and saying hey, let's just you know, expose per annotations directly to the storage system from the PV C. Let's talk about the individual use cases that you have and folks have highlighted a couple of useful use cases to me.

A

One was around storage system topology. That was a good use case and Mishelle enough about a design and we'll we'll get something out there. Another interesting use case was around the ability to tag volumes and I've got a couple of different ideas, for that one is if you want to just tag it with PVC information. If you look at the name field on the create volume called that already includes the PVC UID, so that should be sufficient. If that's the only information you want, the type of tagging that you want is more advanced.

A

So, for example, you want the ability to tag your volume with workload, information or you want to be able to tag it with, for example, information about billing, or something like that. My recommendation would be to create a small, simple controller that looks at your provision, PBS and kind of rectifies and applies whatever needs to be done. The benefit there is that you're gonna get the customization that you want, as well as the fact that it'll actually be continuously updated, rather than applied once and potentially get stale over time.

A

But basically, my point is that I'd like to focus on individual use cases annotations themselves are not a use case, they're a way to be able to unblock some use cases. So let's talk the use cases.

N

Hi so I wanted to highlight one of the use cases which one used: here's what I can bring up from our customers is. They would usually provide the kubernetes zones and the regions from the PVC, so they decide anomaly where they want to provision the PVC, and today they send it why the PVC is where the zone is sent by the customer. So that is one of the year's biggest we had where we needed this to be passed on, driver sure.

A

And Michelle, do you want to address that use case a little bit.

C

You mean for being able to spread, follow, spread volumes across topology that is independent of kubernetes. No.

A

This is topology that is kubernetes specific, but they want users to be able to control it, so they want to be able to specify it on the PVC.

C

A

The the pod yeah.

C

So I think you can we you can do that today, with the volume scheduling feature. Basically, you turn on late binding in the storage class and then basically, we passed through the scheduler decision for pods to the volume creation and that way, all the all the information is taken from all the constraints that the pod has specified.

C

So if you want your pod to be spread across zones, you just need to put pod anti affinity on zones in the pod and then let the scheduler schedule the pods across zones and and when provisioning comes around its going to know, it's gonna have a hint as to the nodes that the scheduler chose and you can use that node to figure out what zone it's in.

N

Okay yeah the topology aware she didn't right so so that is what you're suggesting. So we add the antiphon affinity rules for the pause and then based on that we can try to support our use case of dynamic zone selection. Yeah.

C

So feel free, you can ping me on slack. We can discuss this in more detail, I. Think for that particular use case. We should already have that solved.

N

Okay sure I'll check this check this out and I'll. Thank you. Thanks all.

A

Right, thank you any other use cases or anything else. You want to talk about in this space.

O

Yeah I had one.

O

We are our storage product. Has you know a concept of tenancy and it aligns really well with kubernetes concept of namespaces and we'd like to be able to do our provisioning within you. You know essentially align these two things: kubernetes, namespaces and tenancy, but with the current CSI model, tenancy has only passed in you know. If you turn on this particular CRD object and it's only passed in with a like two calls and I think their node calls in order for us to support the tenancy model that kubernetes has.

O

It would need to be the namespace information we need to be passed in with every call made to the backend, because our the storage, essentially once you provision within a tenant, it's sequestered to that tenant and it might as well not exist to the other tenants.

O

So the we would need that namespace information and at least currently I, don't see any way other than maybe do an annotation like this to to provide that information to the storage plug-in. Okay,.

A

I think that's a legitimate use case. Could you open an issue to track that and we can brainstorm some ideas? My initial reactions are one: can we make this more generic like? That is one way to do tendency? Are there other ways are there? You know? Oh, let's see if we think more generic, if not honestly, I'm not passing through namespace information is not a huge deal, because you can't use that alone to violate portability.

A

It's not that bad.

E

I see a problem, though, in the future, where, if we ever had a way to move like a PVC across namespaces like because those have no namespace right and you can, you can delete a PVC and then create a new PVC, a different name. Space, that's bound to the exact same PV. True buddies will allow this and.

O

I believe that the generic way to solve that would be to create the. If say, you didn't have a way of moving between tenancy models in your storage back-end. You would essentially do a slow transfer host assisted transfer to the next one like fit. You would create in the new tenant and then transfer between know.

E

That the problem is, is the PV object, doesn't have the name space, it's the PVC, but the PV is the thing that's tied to the to the CSI volume right. So if the PVC goes away that the PV hangs around now neck, no idea what names face that PV should be associated with.

A

O

That there are no issues here, just that that I think that this is something worth exploring sure.

A

Yeah feel free to open up an issue to discuss it. I think the points that been pointed out are legitimate and areas that we definitely need to think about all.

O

Right so we're which, on kubernetes kubernetes, you.

A

Can start on kubernetes kubernetes assigned sink storage, that'd be a good place to start the conversation, and then we can go from there and figure out what needs to be done. Okay,.

O

N

This is a little specifically for three fuse community.

N

The characteristic there is various cases for every volume we have a possibility that there can be multiple users using a single new space and you'll also have multiple new spaces in such a way. So, basically, each user will use their own secret to provision the volume I know.

N

Today we have the facility where we can give the secrets in the storage classes, I mean with the special key fears, but the problem here is due to a number of users: it's creating so many storage classes for each user, for namespace to lot of storage class to be created for particular storage.

N

So so this is something one more is case. We're passing the secret names for each volume. Dynamically is something that we are looking at.

A

Sure so there is a solution here as well Michelle, do you want to talk about provisioning, secret, templating, yeah.

C

Actually, I think we just discussed this. We ironed out some more details on it yesterday, but I think so right now we support secret temple in George class, but right for all the operations except for provisioning, and then we just worked out some details on how to support templating for provisioning yesterday and I I think it can work. There's an issue open in the external provisioner I would take a look at it and with our proposal and see if that works for you. So.

A

C

A

A very high level is that in your storage class, when you define a secret, we you instead of saying I, want a specific name or namespace.

A

You can actually have templates that you specify as the value of that name or namespace and then at provision time or at a later time, kubernetes will automatically replace that template with the name of the PBC, for example, that is being used to provision, and what that allows you to do is that you have a single storage class and you could have multiple different secrets for that single storage class based on the volume that's being provisioned and the secret could live in the namespace of the user. It could live in another namespace.

A

This is very flexible currently for basically the attach and mount calls secrets that are acquired on CSI for attach and mount on the pub on the create and delete volume side. It's not very flexible right now. It only allows templating of pv name and we're looking into making that more flexible right now. So that's the issue that michelle just mentioned.

N

So I looked at this template, so I just take a look at it, create and delete things now. The thing is, it is taking the pv name. Now, if you see as a template, I'm referring to more like a dynamic, provisional cases where the user will just create where the PVC and there's not aware of the pv name is dynamically assigned right, so creating secret with that name is changed with every PVC. It's automatically assign right.

A

So we have an issue opened to make this use the PVC name instead of the pv name.

N

Okay, yeah yeah- this makes sense. Yeah PVC name would be more better yep.

A

And then I think it'll become more flexible to satisfy your use case. So take a look at that PR and or this issue, and if your, if you want this faster, you know you can help us implement it.

N

Yeah sure yeah, okay, I'll, take a look at decision, cool.

A

Thank you any other questions. Concerns use cases folks want to talk about yeah.

M

Hi this is Sam, so we found the same thing in this when you have multiple configuration parameters for that you concluded in a volume level, as opposed to at a pool level. We see this explosion of storage classes.

M

Another issue is stored. Classes are typically only administered by the cluster administrator where, as we'd like users to be able to to specify how many replicas they want, whether they might encryption and things like that. So you know like the storage class, this does work, but we just end up with loads of them. So we we found that by adding a passing through the labels, that's the storage system, we're able to remove that.

A

Yeah I understand the attraction of being able to use annotations on a PVC to be able to reduce that combinatorial explosion of storage classes. That happens. I. Don't think that that's a good pattern to move towards in general, when we're violating a basically portability of a kubernetes portable API object, we need to have a very, very high justification, so I do recommend looking at it on a case-by-case basis of what are the things that are resulting in that combinatorial explosion and see if we can come up with a way to reduce that.

A

Are there first-class fields that we can add are there you know? Is there first-class functionality that we can add to make kubernetes more aware and able to allow automatic selection beyond all of that, there was an option that Michelle and I started talking about which would allow users to potentially application developers to be able to override storage class parameter defaults and that we've kind of tabled after discussing with Jordan Liggett, it becomes very, very complicated, very quickly and also begins to kind of I.

A

It moves the problem of portability to a different place at what we were proposing was having a config map that you can have a user defined. That would allow you to override the parameters from a storage class and have the user reference that config map in their PVC. But then you know we need to define edge cases like is it required? What happens if it's not specified or what happens if it's specified and those parameters don't match up with what the storage class provides?

A

How do you specify which parameters within the storage class can be overruled, so I think Jordans biggest questions were around security. The storage class exposes parameters that are very powerful and not all of them should be exposed to application developers or end-users, and then at that point, if we get to a point where we're saying you know what is going to have a default value, it's not going to have a default value. What can be overridden? What can't be overridden we're essentially basically defining a very complicated schema within the storage class.

A

So at this point, I think the recommendation is to live with that combinatorial explosion and right tooling, on top of it to simplify it for your end users, so that they provide one two three parameters and you figure out what the class is on their behalf, and we can continue the discussion about these potential workarounds I'd say, let's start start with a case-by-case basis and then, if that doesn't make sense at all, then we can consider these alternatives that are much more complicated.

P

I know that one location, I sure.

P

Since setting the THAAD system type in the PVC level could be useful, I wonder if this is a use case other this.

A

Was a mistake in the API when it was built very early honestly uh file system should not have been a a per volume field. It should have been a PVC level field.

A

Unfortunately, that wasn't the way that it was implemented and block didn't exist at that time and for a number of reasons it wasn't implemented that way, so we have to kind of live with that, but I guess what you're asking for is a way to be able to have users be able to specify the file system we're sort of having it once per storage class. Yes,.

F

Is it like there is no? The reason we did not do is because, like file system using a file system requires you like tools to be installed on the operating system like XFS frogs or you cannot.

H

F

Excel face on gke or or like similar things. Actually, so those were the motivations that we wanted to leave, the control of which file systems users can use to it in the hands of Questor atman yeah.

A

But I mean we could have exposed the field in a generic PVC field and let that pee, let the storage plug-in figure out if it could support it or not. Yeah. The fact that we put it in the individual volume fields was a little bit of a mistake, but.

C

But also just stepping back like how many different file systems would you actually want to support in your system? Are there really that many.

P

There could be at least a two three four, maybe more. That's.

C

Not a that's, not a lot of storage classes.

P

To open for storage classes for each our system type, it could be too much.

O

Especially when you add in things like replicas and you know all the other.

G

O

Potential storage class parameters in there because you have to do that for each file system, type, yeah,.

L

A

Interesting thing is it's a difficult challenge to solve I'm open to ideas, because some volume plugins chose to define a file system, some plug volume plugins chose not to so. There is no easy answer to say: hey we're going to expose a new parameter at the PVC layer.

A

That is the file system, and if you already have one like, we need to define what all those edge cases are and if this is an important use case, I'd suggest kind of putting out a cap and proposing something now that handles all the weird edge cases and backward compatibility and all that.

Q

If I may make a nice suggestion, this is a Rivera. What about simply telling vendors to implement a controller and Ciardi to abstract away the storage class combinatorial explosion so like you'd in and so you'd be stuck effectively, not working with raw PVC with straight-up PVCs, the final CRD of or volume type. That then, is handed off to a controller that then takes care of finding the appropriate storage class based on the parameters you give it, and then that created the PVCs. So it's a level of abstraction on top of PVCs yeah.

A

That's a great suggestion: it's similar to what I was proposing, which was put some sort of UI between the user and the storage class. If you have this combinatorial explosion, that would be a perfectly good way of doing it. I.

R

Mean oh I mean also. If if we could have a PVC bind, you know a request to a to a store or to cool or our to a storage class based on labels. Then you would have this easy way to dynamically, select, a storage class and then, having you know, a dozen or a couple, dozen source classes wouldn't be so bad.

Q

Well, it does well, I will admit. The suggestion does feel, like a sort of you know, go away and deal with it on your own scenario.

Q

The I can't imagine that many storage solutions will share enough of will share enough permutations of the same of similar variables that it would be worth it to implement a generic solution so implementing custom logic on top of the explosive general solution might be the might be viable and relatively frictionless solution.

A

Yeah and I mean in general, it's not such a horrible pattern in kubernetes in kubernetes. What we try to do is expose the lower level primitives that enable a wide variety of use cases and then have folks basically be able to extend and build on top of that to look at any sort of very complicated workload in kubernetes you don't actually deploy those directly by creating you know, stateful sets and replica sets, and quads and PVCs manually in general.

A

You have exactly what Jose described, which is CR DS in a controller, an operator to be able to deploy a specific type of complicated application. So, presumably end-users are not working on this stuff manually or directly. They have some sort of nicer abstraction between them that minimizes the complexity a little bit if it exists at that lower layer.

J

H

Main issues portability because yeah.

L

H

We have the series each plugins gonna have their own series and everything we're talking about having a generic PVC to enable application mobility. All that goes away. Yeah right right, I mean.

J

If you're, if you're putting every property out to your user and having this combinatorial explosion like I, think you're kind of doing your users dirty rightly, but an application developer should care about. Maybe the backups, but not necessarily every little parameter of me and by you know, specifying that you've loses the portability I.

O

If this CR D route is going to be the the suggested way of doing this, can we have maybe an example so like just something that we can? You know jump-start as a jumping-off point.

E

There there's a sample controller with the sample CID that shows you how to write controllers. It's really easy to grok. Okay,.

O

Good I just haven't come across it yet.

Q

If anything, it sounds like it might be worthwhile to have sample storage controller that you know maybe more directly showcases how how to write a controller that takes in storage parameters, converts them into either creating or searching for storage classes and generates PVCs that way, yeah.

A

But I I think I want to reiterate Brad's point as well really think about what are the parameters that you really really want to expose to your end users, the application developers right the kubernetes kind of wants to get to a point where application developers have a very small set of choices available to them. They shouldn't really have to think too much about storage beyond, like I want slow medium fast. Something like that.

A

If you find yourself exposing a lot more details, there is probably something that we can do either within kubernetes or somewhere else, programmatically to try to reduce that complexity on their behalf.

A

All right any comments about this we're a minute over time.

A

Okay, good topic, three feel free to move the items that we didn't get to to the next meeting and then we'll talk about it. There thanks all right. Thank you. Everyone for the great discussion and if you have any questions, concerns, feel free to follow up offline on the sink storage mailing list and we'll see you in two weeks. Thank you very much.