Kubernetes Storage SIG Volume Snapshot Workgroup, 25 Jul 2019

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Storage - Volume Snapshot Workgroup 20190725

Description

Meeting of Kubernetes Storage Special-Interest-Group (SIG) Volume Snapshot Workgroup - 25 July 2019

Find out more about the Storage SIG here: https://github.com/kubernetes/community/tree/master/sig-storage

Moderator: Jing Xu (Google)

A

A

So can you sorry it's my I guess he is stuck? Oh yes, so in the Monday more whatever snapshots meeting, we discussed this kind of change related to external snapshot or controller, but I'm not sure, maybe because knots, oh, the people joined no one like ask question or have any concerns. Oh nothing so, but so far as before, we move into the like working on this I.

A

Think all of us want to kind of make sure we're on the same page, and we are like very sure, positive about this change because, after like looking at a code, I think there's still quite some work. I think something I also talked about this, and so we want to make sure this work bring us enough benefits.

A

So I think is, for now external controller right act more like if we consider the PV controller and concern no curvature on a driver part, it's kind of combine everything together. So current external sector includes a controller act very much like PV controller, which finds volumes national and snapshots contents, that's by logic, and also it will trigger the CSI cause, consider the driver, and so now we are considering like separate them and deploy them separately. So the the logic for binding bottom snapshot and snapshot.

A

Contents will be you nice, a cube system namespace and deploy as a stateful, said or deployment as a separate controller, and that the rest, the driver part, will deploy it. As let's say, a static are and we list some prawns and cons. I want to see everyone like in to see whether you have more key buyer concerns, whether we we list. We have a complete list, so the back.

B

A

Oh anyone says of me: yeah.

C

A

C

Just to be clear, I was in the I, acknowledge that is its property, the right way of moving controllers out and put into coop system, so that kubernetes has more control ways. I'm just evaluating the time now, I know we're here for you, 1.16 I actually took a deep look into the code. They are currently five list. The issues under this project right, the controller needs to separate out. There should be metrics.

C

We need to flip the page for the version and the feature gap gates time out this time out, issue for snap shooting what we do with that in the controller and the dozen testing listening. All this wire together makes me believe that is not probably doable within the 1.16 frame. I'm, not sure how you tell you look at this, but at least from my perspective, it looks like there's a whole bunch of work over there. So.

A

Yeah, just to kind of clear, like the timeout issue, I think we discussed and I have a conclusion and the change is quite small and these, as far as we are like currently the way we want to walk yeah.

C

Yeah, but that the heavy lifting power is the controller and the metrics and testing yes.

A

C

Pace course, the metrics also depends on the controller change as well, because if you want to expose anything, you need to wait until the controller is moved out. So the the those kind of explicit dependencies makes me believe that it's probably hard to achieve within this contract. I just want to see what do you guys think about this and also what its drawbacks?

C

If we stay as it is at this point of time and go to beta, and there.

D

C

Go back and revisit.

D

The the main thing is I think backwards, compatibility if we go to beta. We ordered like agreed to maintain this model according to the beta policy, which is six months so once we go to beta, it's gonna be harder to change the higher-level architecture of this.

C

But those storage vendors they have their own snap, shoulder which is implement the the way it is in open-source. At this point of time, or rather or rather are the indirect direction stay preferred, did you guys get any collect any feedback on them? Yeah.

D

I think the main feedback I've gotten from vendors is they think the sidecars are too complicated right. They don't actually want to buy cars. They want kubernetes to own all the complicated controller logic and make that the responsibility of kubernetes, but the way that it's currently implemented. It puts all the burden on the vendors to understand the intricate design of snapshot or controller.

E

But I don't sing to the splitter, well make that the well, because we still have a sidecar. They actually favor general concert on all the Santa Clause. Yes,.

D

Yes, but I'm saying like it will help right. It makes a sidecar lighter the.

C

C

Sorry, if they're open, if they're using the open-source one, why would be there any concern? The only thing they need to do is really about deployment of the sidecar right. I.

F

Think it's more along support. If something goes wrong, they are on the hook for making sure it's. Okay, it's separation of concerns and responsibilities. Really so because.

D

They because, like the version of the controller that runs now, depends on the version that the vendor has included right so, and you know if the vendor might end up, deploying incompatible versions of the snapshot or with the kubernetes version or with different side, cars or I, sorry or with different CSI plugins I.

E

Actually, don't think this will make it better I think this can get that more complicated, because now we'll have, we still have a sidecar controller, and then we have this common controller and you still have all the other components so actually I think the versioning thing will be actually to be more complicated in just in this part. It may.

D

Be we have to make sure once we move the API to beta, that we maintain strict backwards compatibility with the API.

C

That how to separate in the controller out help backward compatibility in this sense, even though you separate this controller out in the next release of kubernetes, they still won't change anything like as long as they just change the mod or actually change nothing, because because worried we'll be providing is the from the CSI repository you're going to provide two things. Right is the controller that should be deployed on the coop system and the other is the sidecar which just issued the CS icon, monitor sums natural object and that's it. They don't know defined it.

C

In this case, the version is still not sold in this manner. Is that is minus standing right.

D

Well, the main the main thing here is that you don't have two plugins installed that have conflicting logic in their snapshot: controllers.

C

You don't have two plugins in store that has conscious freaking conversion in this lecture.

E

So the common controller register implement just leads to one version, because the plugins were not to be in store. The common controller part anymore.

C

I, don't know about that. So how do you guys, even in the future, who will be responsible for installing the common controller? Then it's.

D

The kubernetes cluster manager, there are the ones responsible for installing common infrastructure components, so.

C

Whoever manages the cluster needs to install the corresponding control if the functionality is needed. Yes,.

A

So they should be useful for install pose controller and Driver no.

D

No, so it's like think of, like cube controller manager today, right yeah,.

A

Yeah, the cross term manager, I say you start the controller, but to the driver you mean who insulted or not the user vendor or.

D

The the admin cluster admin which might be different than the cluster manager.

F

C

F

Driver here is to separate the responsibilities of kubernetes and the storage vendor into two distinct categories, and by doing so, it will allow each of them to kind of revved independently and decide whether they want to support this functionality or not, and then, in addition to that, the most important thing is be able to monitor it right. If it's a component that I am deploying I am managing I can monitor it in the way that is important to me.

A

So I see, the main thing here is also it is hmm if that means yes, I will be multiple, actually common code, I'm talking total deployed, oh the others, maybe like I I, feel minor I. Think.

D

The observability is a major partner.

F

Ability is huge right.

D

Because if, if it's the responsibility of the plug-in to have this controller, then we can't put in common monitoring because we don't know how or where or what plugins are going to be used. So we can't set up monitoring pipelines to it. Yes,.

C

If it's a cluster administrator, they shouldn't know that way.

F

If somebody comes in on a gke cluster or manage cluster and all the driver that they got off the shelf somewhere,.

C

Well, the same thing, because unless the schedule the controller on the cube system, you still won't be able to more into that like by default right exactly.

F

And so you, if, if this is the scenario that we care about which it will be, we as a managed solution can then deploy the component that we care about and that we need to do the monitoring and therefore we'll have metrics, regardless of the off-the-shelf component, that customer chooses to deploy.

C

You separate the controller out. What I would be assuming is that the common controller will be responsible for four of any a snapshot stuff right. So, regardless which storage wonder it is, and one thing we we I, don't think. There's too many too much discuss shown around a there is the performance right now we at this model is like okay, a storage vendor. You deploy the series by itself, so every single storage window we have your own snaps or etcetera, etc.

C

Is my listening right right, yeah and if you move to the comment, there's a different story, because this common one would be responsible for for all the storage windows. That's one thing, and the other thing is that how do we do with the backup in the future? Are the way two ways? One group: let's.

F

Take it one step at a time, so first concern is around performance and I. Imagine it's kind of intend latency of operation yep yep. So if you look at what the controller is doing today, it's not really going to change from what it's doing today right the sidecar container monitors objects, specifically, the volume snapshot object to, and the volume snapshot class object to figure out if it needs to issue a call to do a create snapshot.

F

That logic is going to remain in that sidecar container, and once that call completes successfully, it do the same thing it does today, which is create the volume it snapshot. Content object, no.

A

It's I thought it would change a little bit. It's similar to PV controller and it's a visionary right right now or I, say the snapshot of controller first to watch this user crania snapshots kind of voting, snapshots, 80 objects that is we're marching in today and then a parent he calls the SI driver. You know to create there's no other nice day overhead right. If we split that means the snapshot.

A

Controller watched volume snapshots, if you have jumped epic object, created I, think I need to update some fear that I say similar speed controller is update some annotation so that the driver side is also has a very simple controller took.

F

A person I think it sounds like there's one extra right, operation involved and right on the water and made one extra right and one X, probably possibly a handful of extra reeds. Yes and.

A

Driver side will watch and then create snapshots. You know I'm.

C

Less concerned about backup on this type of long operations about was snapshot everything about this application layer that.

D

C

You acquiesce right so I I, don't honestly I, don't know how much latency this thing will increase, but something at least I. Don't think, there's too much it so.

D

Today, do we have an idea of how long a snapshot takes right from.

A

My GCP kind of experience is the snapshot. Cut is very quick, I think waiting a second I didn't measure like this, maybe seconds or something it's kind of amazing second, and then the uploading phase, depending on your disco size, its might take kind of minutes. Okay, so.

C

Like DCP teaser, it's really a different story. If other I test this some of the SPS people taking a snapshot release is like a mere seconds level right. It's me most of the most of the consumes with the controller where is tries to update the status of a snapshot in the status of natural content.

A

Either end to end just like when you create a key object, and then we update the Avatar to indicate it is being cut visitor, it's updated timestamp. So the total time like we have to include it object, creation and updates time exactly.

C

And that actually makes me question a sync thing: CSI Drago for the snapshot in peace, because, if you think about the application mango, you need to actually already stop the application for some of the applications right to start record application for from function in and here the snapshot of the warning has been done now. If the, if the whole finding and finding process is next day, let's say, take 99.99% of the whole latency time, it doesn't make any sense anymore.

C

But if you say yeah by saying that, if you separate the controller out, there will make the performance tuning the future less likely. That's. Basically what I was saying. The whole conversation about performance.

A

So, with current way, we haven't heard like any customer like complain, but it caused by the operation or ahead yeah. It's great.

C

Because there's enough alright, so many complains about a lot of stuff is aggregating. The experience.

A

I was just thinking.

C

A

Yeah, that's what we put in there like constant performance in C rewrites, involve a few more like API object, read/write equation.

C

So that what I was saying that using the current model, where the snapshot controller and the CSI called literally sitting in one task in one binary, making it possible in the future to do the CSI call CSI call in a much in a Melissa near synchronize. The way right you can check, you can do the binding in a much more agile manner. If.

F

If the, if the application queue essing coexists with this external snapshot, which it will not.

C

hmm Say that so.

F

I think if I understand what you're saying you're saying that, if all the logic exists in a single binary, it becomes easier to optimize the intend latency, because you don't have to worry about network halls and things like that. Don't.

C

Ya, you don't need to worry too much about finding because I look into them into the controller implementation. It's basically a true.

F

Piece that you're concerned about is the one where you have to Q s and unn q s an application yeah.

C

F

Now, the mechanism that has been proposed to do Q s and on cue s, the executing hooks API.

F

It is going to have completely independent set of controllers of.

C

Course of current.

F

C

This is a little it's a piece of it. That's what I'm saying I think.

F

That's where it becomes I think we would need to figure out what percentage it is if it like. Is it if it's problematic, let's figure out what percentage of that intent I'm this is adding to, and at that point we'll have kind of better data to make a judgment. I agree, always nice to kind of optimize where we can, and you know if you, if you have two equally good options and they're both equally easy to implement, go with the one that's more optimal. Obviously, but in this case there are trade-offs.

F

We are introducing an extra right operation and potentially a few more read operations which could affect which will affect the intend latency, but how bad will it affect that intend latency and I think that's the that's the key. If this is still a concern, I agree.

C

With you, let me ask you you guys a question because I haven't wrote anything about the controller, it's more about about chin and chin over there. So so, when we go through the the Montessori, when the controller actually checks the snapshot, objects and other natural content objects, how frequently do the chicken is any like time interval.

B

Yeah is what sorry its event based. It's.

C

Based on the UN okay, it's.

F

Not pulling its push push notifications, essentially no.

B

D

The cache is updated on an event based thing and then you queue the results whenever the event comes in I, see.

C

So the usually events, the.

F

C

Okay, I see your.

A

Watched like snapshots at snapshot delete. You went so.

F

We have this one-hour meeting or 30-minute meeting 30 minutes. Okay, so we got two minutes left is. Am I hearing correctly that there is concern about whether we should do this or not, or is the concern more around the timing to both okay? So it's both then I think we need to have another meeting to discuss this because.

A

F

I have another meeting right now that I need to drop off soon.

C

Anyhow, this is actually just from me right again: I'm wearing you through this community I believe in you guys so yeah.

F

No I mean good, raise the concerns now because I'm sure, if you have them, then others are gonna, raise them as well. So we should all be on the same page. Before we go forward. One thing I, recommend is and and Michelle brought this up was a lot of this work is gonna end up rolling into the CSI workgroup I. Think Shane. You attend that meeting already, but maybe Jing and shiny. You can start attending that meeting as well and have this discussion as part of that meeting. Sure.

A

Sure it's just like I want to make sure it won't take too much time of that meeting, and we have to like a few of those okay.

F

I mean, if that's the case, then maybe that's the normal plan and we set up another one of these two one-off. Two to finish this discussion. That's okay! To.

C

Get involved in those meetings, Michelle.

F

C

A

And so we also want to hear like other people's feedback in that meeting.

C

E

Monday yeah, the rest of them are pretty quick, pretty quiet when we were talking about this.

A

F

Think one thing to think about, as we have this discussion is, if you don't believe that this is the right way of doing it, let's think about how we could do metrics in a way that is sane and workable, because that is an important use case. Well,.

C

C

Talk to us as well about that, okay,.

C

Is there any place? I can get more feedback from storage vendors with regarding to the current architecture of doing snapshot and maybe in the future, for backup.

D

Probably six storage mailing list yep.

F

Feel free message: the six storage mailing list and saying: hey guys, here's what I'm thinking! What do you guys think? Oh.

C