Kubernetes Storage SIG, 6 Dec 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Storage 20181206

Description

Kubernetes Storage Special-Interest-Group (SIG) Meeting - 06 December 2018

Meeting Notes/Agenda: https://docs.google.com/document/d/1-8KEG8AjAgKznS9NFm3qWqkGyCHmvU6HVl0sk5hwoAE/edit#heading=h.floajrnz1d8l

Find out more about the Storage SIG here: https://github.com/kubernetes/community/tree/master/sig-storage

Moderator: Saad Ali (Google)

Chat Log:

09:12:00 From BKkvkgY47EVe+2A5qybNHDOEmY3oN22gMx8LVc4hAuoKH2SpT0ZBc5yHSmCr9AgcAsyl234QvZt6LzSUBT94fw== : there is already a PR for snapshot support in EBS CSI driver - https://github.com/kubernetes-sigs/aws-ebs-csi-driver/pull/131
09:12:32 From cheng pan : yep, haven’t got the time to review yet
09:31:50 From Xing Yang : https://github.com/kubernetes/enhancements/pull/643
09:32:10 From Xing Yang : https://github.com/kubernetes/enhancements/pull/642

A

All right today is December 6 2018. This is the meeting of the crew brunetti CSI implement. Sorry, this is the feeding of the kubernetes 6 storage group. So today we're gonna go over the queue for tasks that we've committed to and get an end of quarter status on them and then we're gonna review PRS that need attention, designs that need attention any open questions and discuss Q con Seattle, which is going on next week.

A

If there's anything that you want to talk about in any of these categories or anything else, please feel free to add to the agenda as we go. First of all, a huge congratulations to the group for working so hard to get CSI to GA this quarter, we were able to successfully make that in to get that into 1.13 and then block volume support also moved to g2 to beta. So a huge congratulations to everyone who work really hard to get that done.

A

Let's get a more specific update for each of the tasks that we committed to. Let's start off with this, the CSI driver for VMware. We wanted to get an end of status. Update on this is glad on the call by any chance.

A

Or is anybody else from VMware? You might be able to give an update on this driver.

A

Okay, we'll leave that then and we'll get a status up. They did some other point. Next up is the kubernetes conformance process established processes test documentation?

A

Any updates on this.

A

Michelle, do you happen to know where this ended up.

B

A

Well, mark this as not started, Yeah.

B

A

So we'll pick that up next quarter next up is CSI driver updating the ice, cozy library to use the lid. But before that, let's talk about the CSI I scuzzy common library to alpha that was completed and then John were you able to also get the ice. Cozy driver updated to use the library.

A

Looks like John might not be on the line. Does anyone else have an update on this? Maybe Ben I.

C

Haven't been following it since I reviewed the code, okay,.

D

C

Think the last the last place it was, it was a needed, a new home and github. Did you ever get that new home I.

A

Think the repo got created, but I haven't seen any PRS on it. I can take a quick look just to verify.

A

That's the library, that's the driver, Oh is.

C

There any commits in.

A

The library, let's take a look.

A

So the library looks like it is good. There is the library there so I guess the library is complete, but the updating of the driver hasn't been done yet so, okay, as not started, and we can pick that up next quarter, so good progress. It was a short quarter, so there's gonna be small things that we didn't end up finishing. Oh this.

C

Is still trekking, q4, okay, I'd say yes,.

A

This is last quarter or this past quarter.

A

Alright, next up is a dynamic, max volume, dynamic max capacity per plug-in per node count a month. Are you on the line yeah.

E

So there's a open division, get up like for CSI, but I think mean you. Take it up, make a CSI chance to be with how to take this up in next quarter. If it didn't make much progress in this quarter,.

A

Sounds good and next up we have CSI adding support for volume expansion. Yes,.

E

Pair ghost, but we have two and there's already some bit of resize controller done, but we have to get a new release of CSI as well, so cool.

A

So we can follow up on that next quarter, maybe get a CSI zero point at one point: zero one or one point: zero. One I'm, not sure what the next version is gonna be called yeah.

F

A

Point one yeah: okay,.

A

That looks good next, up is AWS, EBS, CSI driver, I, know Nisha, and her team were pushing hard on this. This quarter and I think Chang might have been working on this as well as Fabio. Is anyone able to give an update on end of quarter status here? Yeah.

G

I have sense, so we did a alpha release for the CSI driver.

G

And it's tracking light ah si si zero points right now. It.

A

Is 0.3 yeah.

G

A

H

By the way, do you help lens change mine snapshots in this river yeah.

G

We're planning doing like our next corner: okay,.

A

Great thanks awesome. Thank you, Chang. That is super exciting, so we'll go ahead and Mark that as done and then next quarter will pick it up and hopefully move it to beta and pick up CSI 1:04, azure, disk and azure file. Csi drivers I think we ended up dropping those because we had nobody working on them for the quarter, so we're gonna skip over those begin implementing the migration of the entry plugins to CSI. So again, I think Chang, David and deep were working on this. Can anyone if you give an update on this.

I

Yeah I think it's a work in progress. At the moment we had some initial PRS done, but basically it if the Alpha is scheduled for the next corner, so pretty much the ps4 and hold there were any shell around the reviews done, but nothing got merged. Last one: okay,.

A

So we'll keep this as started and then we'll get updates on this later. There are a bunch of items that are missing from my spreadsheet. Oh I think it's collapsed.

A

It was a trick to I'm doing that turn off filters. Okay, alright, next up is beginning implementation. We talked about that CSI external.

B

B

A

At the top okay, we can review these again number one is CSI entry block volume support moved to beta. We ended up not doing this I think it was pretty close, but we were pretty tight with the CSI GA, so we ended up putting Soph Ben. Do you have anything else to add to that.

C

Well, we still need to get auch support into the hosts path driver. That was the missing piece. Okay,.

G

C

Think I think we didn't know if there was a needy test and I thought that there wasn't, but it turned out that there was, but we couldn't run it because none of the none of the gate test drivers had blocked support on which it could be run so either the gcpd driver or the yeah the host patch drivers, one of those drivers. If we get blocked support into it, then we can just run the user some tests and say yes,.

A

Okay, that seems reasonable, so we'll go ahead and move beta to next quarter, for this looks like we were pretty close, so I'll keep it marked I started next up is we talked about the VMware vSphere driver? Already next up is the library for moving the mount library to external common.

A

Travis any updates on this.

B

J

I was gonna, say I'm, not sure if Travis is on long I know Lester last I heard from him. He was working on this, but I have no idea where it's at okay.

B

Yeah I haven't seen any updates on this for a while. Okay.

A

Did we do we know if the library actually got put into an external repo? No.

B

No, it needs we need to do this. First. Okay,.

A

B

Is we need to refactor the library first before we can pull it out, got.

A

It okay, so we'll keep this as Markus started. Vlad I just heard you on the call. Can you talk about the VM yeah driver.

J

The CSI driver work is currently going on going right now and I believe we're shooting before before early, really, not in early means before an alpha. This quarter so we'll see where, where it lands, but it's it's being actively developed.

A

End-Of-Year: okay, cool we'll mark that as started.

A

Pluggable end-to-end test framework.

A

This was, let's see, refactored intend to use outside partially done. Refactor volume tests some work remaining to make it useful outside of KS, so Patrick or Michelle. Any updates on that.

B

Yeah so this quarter, we made a lot of good progress on this. It's still not completely there yet, but we're getting close I think, probably by I, think definitely in 114. We should be able to have something usable.

A

Well, I do want to call this complete for the quarter or started. I guess started. Make sense here.

B

Yeah started as hind: okay.

A

We already discussed the conformance stuff provisioning capacity report reporting for generic topology.

B

So this has pretty much stalled and I think we need to find a new owner for it. So if anyone is interested in this, please let me know this is needed in order to be able to do local storage, dynamic, provisioning.

A

Surf anyone's depending on local storage and is interested in dynamic provisioning. This is probably something you should look into helping out within the next quarter.

A

Next, up si si external containers move leader election to lease object, Matt, any updates on that.

A

Okay, so I think what ended up happening was on the side, car containers we disabled leader, election by default, and so, if you opt into leader election, it requires the kubernetes service object and you are required to update your are back rules to ensure that you have access to that object, but by default leader election is disabled. So you don't have to do that. But as far as I can tell, this work was not completed so we'll keep this as mark just started and move it to the next quarter.

A

Next up is sub path, expansion.

B

So the cap has been submitted for this and we're starting work for one fourteen cool.

A

And I guess the disk pool here was designed so seems, like that's, been accomplished, update the I scuzzy driver. We talked about that. We talked about these begin implementing migration, CSI external provision or topology support for kubernetes to beta that ended up. Not actually did we end up doing that, so it's still yeah.

I

A

Do we know what's remaining.

B

We need to figure out what to do about the admission webhook, how to get that installed and also we need the e to me. Tests.

A

Okay cool, so it looks like there was a lot of good work done here, but we'll keep that in alpha this quarter and move it next quarter. A CSI driver for fibre channel and updated to use the library and updated to do 0.3 changes.

A

Medusa yeah.

K

The fibre channel library cut merged to the repo and but I think the driver would need because right now it's in my own repo, no.

L

K

Think we need a separate home second.

A

Okay, so I think if we look in kubernetes CSI, there's already a repo there, that you should be able to use, take a look, so there should be SC, so there's the library and right now it should be empty but feel free to send out a PR to that. Oh looks like they're. Oh, this is the library okay. So this is the one that you created: yeah, okay. So now, let's take a look at FC. Oh it's like we don't have a driver for SC, hmm okay,.

A

Reach out to me we'll get that created, and then you can proceed from there. Okay,.

A

We can do that next quarter, no big deal. Okay,.

A

How to test Fiber, Channel, I, scuzzy, NFS, libraries, so I think there was a weekly call set up Brad. Do you wanna give an update on this sure.

M

So alright, we've we've been doing calls from status on this and it's been making pretty good progress. I, don't we're not complete yet I I! You know my plan was to kind of meet informally at UConn and go over what we've been working on and then kind of come up with the next steps. This might be like another or a good topic for the informal thing and.

A

I'd be perfect, all right cool thanks bread will market I started. Then.

A

Next up is flex resize support discount merged. There was a known issue that was discovered afterwards, but as far as I understand this is complete home on. Do you have anything that or Anika so.

E

Be good I think it got done and there's like there's a the flex start wondered that the spec has to be like and the documentation has to be updated is.

A

E

A

Tracking that on it.

E

There's a shoe tracking, it I hope someone from IBM takes a look at it, but yeah they open the issue. So yeah.

E

A

Thank you. Next up is improving node shutdown handling. Unfortunately, we didn't make too much progress on this jingshu created a proposal for how to handle this more holistically. That proposal requires cooperation from sig node, because a lot of the work there is from sig note. So the next steps here is to share that proposal. Would sig node I'm, not sure if Jing is on the line or if she wants to add anything else, I.

E

Think she already sent it in front of signal. Okay,.

A

Do you know how that went over I was.

E

Not in the car but but I spoke with Derek Carr afterwards and there's like some concerns, and there are some comments like in the in the proposal itself. Actually so the summit's I have added in the endo sinks document contains document.

A

Okay, so we'll keep that Marcus started and move that to the next quarter, moving topology to GA Michelle, and you updates on that yeah.

B

A

Continue working on snapshots, shame. Do you wanna give an update on this yeah.

H

So we added deletion, protection for a wooden snapshot and one snapshot content, and also we have a change in policy for what in snapshot contents. Those.

A

H

Detention policy for the content, I'll read: I'm, sorry retention, let's see lardy clean call, Z and yeah. So that's it! Oh and.

A

uh And there were some breaking changes in CSI and on yeah as your lease yeah.

H

A

Fixed I, just as an FYI to the rest of the community, okay,.

H

A

If you implemented an early version of the snapshot, alpha know that this quarter, a snapshot still remains a now we're breaking changes made to 0.3 in 1.0, and the 4-volume snapshot also had breaking you're using a older driver. That was there's.

J

A background noise, it's making it hard to hear stuff well,.

A

Let me see if I could meet someone.

J

It got better right.

A

So if you had a driver that you already have in producing, it should continue to work. But when you upgrade to CSI 1.0, your driver will need to be updated and you need to update to the new external snapshot or sidecar container, which contains the new CRD.

A

And then these two features are just part of kubernetes 1.13.

A

So we'll call that donor.

A

Next up is automated end-to-end testing of sidecar containers, both major and minor versions.

A

I think we are kind of doing this now with the intent, testing and kubernetes, but there still needs to be work done to add n 10 tests into the sidecar container, repos I, don't know if Patrick was able to make any progress on this, but I haven't heard anything recently.

A

And I think this is work that will spill over into the next quarter.

A

Icsi speck moved to 1.0, so that was completed. Csi is now 1-0.

A

Csi entry moves, CSI, persistent volume, source and volume attachment objects to be won. This was completed. The caveat here is that the callers must be afraid it updated in next release. So, even though the API was updated, apparently you're not allowed to update the callers in the same release, so we're going to have to do this in a follow-up release which will be next release: 1.14 cubelet device registration mechanism to GA. This was a lot of work, but it also ended up getting done thanks to blad for all his hard work on that.

A

And next up is the CR d installation mechanism. We have a temporary workaround, which is.

A

The cube add-on that we added, which will automatically install the CR D and until we have a better mechanism, we're going to continue to use that next up is moving CSI driver and CSI node info objects from alpha to beta. We ended up not doing that or not completing it, at least so. We're going to punt that to the next quarter,.

A

So the node info object, which is alpha, changed a lot between one point: twelve to one point: thirteen, if you're looking at that object, heads up next step, is moving our back role: definitions from kubernetes kubernetes to external repos.

A

This is on track, I, think glad you're working on or is it Patrick who's been working on it Oh.

J

Which one is this I'm sorry.

A

Once our back world definitions that.

B

Should be done, okay,.

A

Cool, so for those of you who are not aware, there's a set aside car containers that CSI uses and those side car containers used to expect their role, definitions to be defined in kubernetes kubernetes we're moving away from that model to having the are back definitions defined inside the sidecar containers as llamó files that need to be deployed. So just a heads up, if you do any sort of CSI volume, development.

A

Next up is implementing CS iPod inline volume support. We started on this, but did not complete it. So we're going to push this to next quarter.

A

Okay, next up is PVC namespace transfer design, which is required for cloning. Does anyone have an update on this looks like a design. Doc was created.

H

Muda to the there was a cab moved it under enhancements.

F

A

F

You can hear me yes,.

A

F

Yeah yeah I'm just want to add the the proposal is already like there and we start to review it and kind of relevant to this and data population. We have a readiness proposal and discuss the ins snapshot meeting and we all like also put it on the committee repo got.

A

It so we'll do we want to mark it as completed for this quarter or started no start and there's not completed no I mean in terms of design. So it sounds like the design started. Yeah.

F

So it's kind of in progress or you review yeah, okay,.

A

Sounds good and then you mentioned the work for data population as well sounds like yeah. The.

F

Main part is the readiness feature like or technician and also relevant to snapshot and yeah clone.

F

But by the way, I'm, not I, don't think her stem cell can be say done. There's still a few things we probably need to finish before can can see. There is done done.

A

For this quarter, is there any remaining for this quarter? Oh this.

F

Caught her? No, no okay, that's fine yeah.

A

That's definitely far from complete okay cool, so that gives us an update on data. Populate er is both of which were design CSI sidecar container splitting up driver registrar in two separate repos. As far as I understand, this is pretty much complete. I think. The only thing that's left is deprecating the old repo and transferring over the issues luis any update on us.

A

Michelle, do you know how far along we are in that I think.

B

The the PRS have been merged I, but I. Think they're still like repo maintenance does still needs to be done. I think.

A

We're pretty close, and this should be completed in the next couple of weeks. I'm gonna preemptively mark this, as done so it's easier to track, support mount options in CSI, external provisioner was completed and allow mount options for CSI volume in Cuba. It was completed and we did not end up working on this.

A

Alright, thank you. Everyone and with that we will create a sheet for 1.14, feel free to get started on adding items to the 1.14 for the next meeting, which will be in two weeks, and this is planning for the first quarter of next year, I'm going to go ahead and delete all the items in 1.14 and we can start with start fresh and feel free to add items where is delete.

A

I

Do we have any high-level idea of dates for the next quarter, like.

A

I'm not entirely sure but I can't I.

B

Think all the dates should be on the cig release repository so.

A

Let's see if there's a 1.14 schedule published, looks like they don't have a schedule published yet, but it'll show up here when it's ready. Okay, typically it's a code freeze in the second month on near the end of February is my guess and then a release in the third month, so release would probably be about March thanks. All right. Next up is PRS that need attention. So we have number seven one. Four one, six added here by PO neat, yah Raja is this person on the call?

A

A

A

Having a little bit of difficulty hearing.

A

N

A

N

Is this actually a sick.

A

E

Yeah, this is I. Looked at it briefly, it's nothing to do with six stories. Such a thing so generate thing that nothing particularly about six storage. Okay,.

A

So maybe we can discuss this in the appropriate sig. Would sig node be more appropriate or sig GCP a.

E

Sick note: okay.

A

Does that seem? Okay to you.

N

A

I think this would be definitely a sig note issue I start there and they might redirect you to sig GCP, but I would start with sig node.

A

In fact, you could probably all helping a couple of signal folks for, for you.

B

Hey sod, my understanding of this PR is that it's actually just adding debug statements to help debug the test case. I'm, not sure if there's actually any like anything to review. Okay,.

A

So what was the concern that you had.

A

So this this test, after being a refactored, is failing in the gke noted, node environment.

N

A

So and you're trying to figure out why it's failing on gke, node, okay, I, think yeah I think.

N

A

You've already understood what the problem is.

N

But we got done with.

A

What's the difference.

N

Differences, we are expecting.

A

The test more robust is the new line actually like bad, or is this a test issue, or is this a a environment issue.

A

Okay, can the test just accommodate for the new line like stripped off the trailing and post empty white spaces.

N

A

Not the right thing to do.

A

Any one of them.

N

A

Let's, let's take this offline I, don't think we need to have the rest of storage sake involved with this Michelle and I'll follow up with you to help sort this out and see. If we need to redirect you to a signal, please reach out to us on slack and we'll take a look at this offline.

A

K

A

Next: up is storage, OS attached device, bug yeah.

L

This is Simon. Can you hear me yes yeah great, so this is a bug in the storage OS native driver where, if a volume was mounted outside of coronaries control, were unable to walk kubernetes from attempting to mount the volume itself. So this adds a call in an attached device call to our API, which will allow us to fail on the Athiya at the attached stage.

L

So storage west volumes are attached to all nodes by default, but this just gives us the opportunity to block command. Ok,.

A

That seems reasonable, we'll take a look at this afterwards and go ahead and approve it I'm just curious: do you guys have any plans to create a CSI driver? Yes,.

L

So we already support 0.3 and 1.0 and hopefully I'll be cutting some of you guys next week about how we can migrate. Our native driver 2, to remove in CSI whoo one.

E

Quickly, feedback, can you keep like the volume expansion also is included in the same peer can be like splitting.

L

That yeah, that was a bit of a mistake and I, can definitely get that splitter. It shouldn't have been any bug fixed, ok,.

K

A

I'll make it easier to cherry-pick around if it doesn't have any feature code. Yeah.

L

Yeah I mean we'd really like to get the bug fix into all the versions. If that's possible, ok,.

A

I, do you want to send up a follow up just to remove that commit and then I'll go ahead and approve it ah perfect, yeah, I'm thankful. Thank you.

A

Right no design reviews there is an outstanding question from Shang. How do you write a CSI plug-in that works with both kubernetes 113 and 112? So so.

D

I recently discovered that so after we change the two CS I went out of 113 so seems 113 can works with CSI v-0 I found a one. Last minute check came from started right. Yes,.

L

D

So, but for our storage winders, so how do we say suppose once again, 112, but also of course, we won't have still incorrect incorporated the CSI we want instead of this arrow, but James is just either one of this, but not cannot. We cannot not do both so.

A

The way that I would recommend it is create a 0.3 driver and create a 1.0 driver, so a driver that understands the action.

D

So that's that's the things we don't we're, trying not to maintain true code bases, so I, yeah and I know that will work. So, basically you have to branch out the code and as also means we have to do the Cherokees ending things.

D

First, if there are any fixes, they have to go to youtube cases. It's a second. We do have automatic detection of what the user versions, but we we.

D

We then need you to deploy the different version of the code which can print arrow, so I just say if they are I mean one anyway to say just with one code base, we can provide post endpoint. We want technically.

A

It is possible, strictly speaking, you could write a driver that speaks or basically has both the zero point three and a zero one point: zero support. The challenge is going to be that the sidecar containers that we provide are either going to speak. 1.0 or they're gonna speak to zero point X, and so, if you want to do this, you're going to have to kind of do manual work on these side, car containers to merge the zero point, X support into the 1.0 side, car containers, you.

D

Know on a cycle partway of fine because driver for the sidecar we can just choose that deploying the different image automatically we do have same says protobuf was not seems not version, so the data structure definition is different for 112 and, in fact, I'd be generally. Why something like a warning I give is IB, yes attribute song. The.

A

One point: zero point: X to 1.0 is not backwards, compatible the spec breaking and the only way to make it work is, if you basically have a driver that will advertise both versions and support, both versions. Yeah, so callers, select, basically based on whatever version is the driver support, so the older versions will accept the older.

A

You know zero point.

D

A

D

What I have in mind, but the what's was lying before, is that the protocol is I, think you just convert you to the one, so they don't even have Weezer doing the same grapples. Oh.

A

Yeah, so what you can do is look at, and this is what we did when we added 0.3 support for into kubernetes. If you look at the spec, there is a separate release: branch for 0.3 that you can look at.

A

You do not hear there's a separate release that we did over here. So if you look at the the 0.30 tag, this is going to contain the spec of the generated protobufs for the 0.3 client, and so you can use this version to have a 0.3 client and you can use the 1.0 version for the 1.0 client, okay,.

D

So yeah yeah yeah.

A

The only challenge you're gonna have is in terms of vendor if you've a.

D

Commission Devendra have the same version which I'm saying yeah and.

A

The way that we got around that was, we just checked in a hard coded version of the 0.3 client into kubernetes kubernetes. So we have this one vendored, and then we have this one, just hard checked into a home directory. It's a not a great hack, but it gets the job done that way. You can have two different clients. So if you look at that, PR that I sent out for on the kubernetes kubernetes side that'll give you ideas on how to handle well.

D

Yeah yeah I won't okay to that cool.

A

No problem reach out to me on slack, if, if you get stuck sure thank.

A

Ok last item on the agenda we have today is Q. Khan Seattle is next week. I know a number of folks are going to be. There.

A

Diamante has some space that they set up Wednesday morning from 9 a.m. to 12:00 for use by SIG's storage, and we were considering holding a mini face-to-face there so feel free to add agenda items there. My availability on Wednesday morning is still unclear, so I will once I have clarity on that I'll. Let you know, but I think other folks from my team will definitely be there, so Michelle or David, and one of them can cover for me if I'm not able to attend and I believe Brad is also planning to attend.

A

This is that right, Brad yeah.

M

A

If I'm not able to attend, Brad can probably leave this in terms of calling it an official face-to-face. What do folks think about I think there is sufficient number of people showing up that I think we have enough for a quorum and, let's just call it an official face-to-face.

A

O

We're on our space, yeah, so hey this is somebody area faces up right now. The room is for about forty people, but they said like. If there are more people, they can extend it to 80, also, okay, yeah but I. Don't know if I have to confirm the 80 thing right away or that that is cool.

J

To your convention, yeah.

O

It is a radiance hotel, they.

J

Know that hotel, okay.

O

A

There, somewhere in particular in South, LA, where it is.

O

Renaissance or exile update in very soon I mean.

A

All right and then in terms of the zoom link, we can probably use the same zoom link as this this meeting folks, who are not able to attend in person and that's it so if there's anything that you want to discuss or in person feel free to add it to the agenda.

A

This is a half day, so it's not we're not going to be able to cover as much as we do in a typical face-to-face which tends to be a two day two days long, but we'll get done as much as we can and thank you for hosting this yeah.

O

Sure yet they said that they can even and go up to one one o'clock, because the other thing is yeah nine to one else of me,.

A

Okay sounds good anything else. Anybody was.

J

There, like in the agenda to that document, sorry I. It's.

A

Right here, alright.

J

Okay, thank you.

J

Thank you. Cool.

A

Anything else anybody wants to talk about.

A

Okay, I. Think last thing is talks at cube con. um If you have a storage related talk, please throw it in here so that anyone who's interested cannot attend if they're attending cube con all right. Thank you very much for your time and we'll see you in two weeks we're gonna do planning for q1. So remember. If there's that you want us to work on for q1, add it to the planning spreadsheet under the 1.14.

J

A

All right take care of folks Thanks.

A