Kubernetes Storage SIG COSI, 17 Feb 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Storage - Object Bucket API Design Meeting 2022-02-17

Description

Kubernetes Storage Special-Interest-Group (SIG) Object Bucket API Design Meeting - 17 February 2022

Meeting Notes/Agenda: -

Find out more about the Storage SIG here: https://github.com/kubernetes/community/tree/master/sig-storage

Moderator: Sidhartha Mani (Minio)

A

So good morning, everyone uh jiffin just brought up that the cozy website is uh outdated.

A

This is the one yeah there, it is jeff and I should remove all this okay. So what we'll do.

A

Yeah, I think this yeah.

B

Let's see pages.

A

Just turn it off is that right. This site is published here.

C

Yeah, at least for some time or something like that,.

A

Yeah, I'm just wondering how to undo this there should be. There should be an option right.

A

Does anyone know.

A

So github pages is currently the website um re-archive this. For now, let's find out um stop publishing website who knows, does anyone know no unpublishing? Okay, there you go.

A

Important operation section several click pages on the data pages source drop down menu and select; none, okay, there's a non branch great there you go.

A

And it was the other option.

A

Main page is request to delete the branch using but rate the entire repository from them. You're enabled your dns settings. I see.

D

A

All right, so the only option we have is to delete it. I actually don't mind deleting it we can. Since everything has changed, I think it's fine to restart from scratch and I have a copy locally.

A

Let's, uh let me make sure of that.

A

A

All right, um let's go bring it to master.

A

Yep so latest comment is here: um everyone: okay, with that getting rid of this website, we'll create a new one, all the way from scratch when we're ready again.

E

uh What do you mean you're deleting the whole repository? What is that we have.

A

Oh, it's like a yeah. This is not.

D

A

We have actually, we have quite a bit of information here, um so this is just um a lot of information about how cosy was designed before how to.

F

A

I think yeah, okay, so now that I think about it, I'm glad you went over this um so.

E

You want to just mark something like deprecating. Instead of deleting the whole thing, maybe you can, then you can. I don't know what is the easy way? I I didn't know what I don't know, how how you publish this once, I'm not sure, what's involved.

A

It's just on github, so you just create a repository by this name. It'll automatically get published uh but see the way grant set up. uh Cozy is by following these stocks. Oh.

E

So that's actually as you're useful, so yeah.

A

E

Maybe you don't want to delete the whole thing right, then people don't have anything to.

A

Follow yeah, I think, you're right, so so so different. How about this right now? This is, you know what is here is is what is reflected in the code base, so right, yeah, yeah, yeah,.

C

That's kind of.

A

Kind of valid, in a sense I.

C

Mean there are some parts.

A

Which are not, but that's, okay, I think.

E

Maybe there are like a certain chapter, certain section that are outdated. You can just add some deprecated at a note next to it, so people are aware of that or something.

A

E

Any particular section that are totally updated.

A

Yeah yeah like this is outdated. I think yeah or I think we can just update it. The only reason we're not doing it is because we have to get back some information back from the review.

E

Right, I know yeah I'll just yeah there anything that is, you know you think it's just too updated that you just want to. I don't know it. At least I don't know if that's helpful, but if the this whole thing is helpful, uh like grant actually was able to set up something following this, and would you keep it.

D

Right yeah, I used this a lot when I was working on it.

A

Okay, I'm happy to hear that so uh yeah. We are a little behind uh on updating this website, mostly because uh our design is still kind of in flux, so yeah, but I'm glad it helped you grant. That was the intention.

A

All right, so so so for those of you joining just now or maybe after I was talking about the review process, uh our next step is really getting our review.

A

um We haven't heard back from the reviewers from two weeks ago, after after the last after the meeting two weeks ago, uh where michelle was here, um I wasn't sure if she'll be able to join today, I was hoping she would, um because I think that seems to speed up the review process.

A

um I've been reaching out to her um twice this week.

A

I've done it, um I think she's very busy, uh but but uh I think yesterday she she said that she should be able to find some time to take a look at this, but as of right now, unless someone has something to bring up they're exactly where we were last week um at least next week uh we have, we will have the opportunity to bring up cozy during the sixth division meeting and and have others take a look, um but but as of right now we you know not much has changed.

A

um So you know what I was planning was. We will, you know we'll reconvene next week when the review is done um and, and I'm fairly positive it'll be, you know we'll have we have we'll? Have someone take a look at it um and and next week at least it's there's something actionable there'll, be a six storage meeting where I can bring it up.

A

So, if not for the review, is there anything else anyone wants to bring up any any questions about the design we can spend this time to, you know, explain any of the parts of the design that that you, you you're wondering about you're, trying to see why it was built the way it was.

A

Any suggestions, it's a good time for all that now.

A

Okay, all right um so shane is there anything we should uh you. You want us to discuss other than doesn't the review part.

E

I don't have anything really just uh that's the most important things. You've got to review down that capyan.

A

Yeah, um how about ben or jeffen any of the so so uh so I believe the piero is merged for, for uh uh the grpc uh namespace reservation is that right.

E

A

Grants he's pr.

E

Yes, that's, I think, that's merged.

A

Right that is, that is, that is good news.

D

Yeah, it should be in there.

A

That's good all right, um yeah and I saw that uh yeah, and I saw that you, you auto, generated the files and rest of the stuff, so yeah.

F

I was wondering if.

A

If I had these docs in place already, but luckily it was there.

D

Yeah it took some changing on my local machine to get it working, um but uh yeah. This stuff's mostly worked. I think it's just something that was up with my machine. Okay, good.

A

To know all right, um so okay, so there is another thing we can discuss, which is so you know we are all agreeing on this design. More or less um I mean really. The reviewers are not going to have this fundamentally change.

A

If anything, there are going to be minor changes here and there I think it's okay to start developing and writing code already, even though you know review isn't done. um I think I think that's okay. What are your thoughts on that.

F

A

All right, let's do it then happy to hear that all right, yeah.

C

Like at least, we can remove the things right, which is not valid anymore, like.

A

C

Those things can be removed right. The.

F

C

Yeah, which are pretty sure like it, will be removed. Those can be, and.

E

C

We can update the docs talks as well right.

A

Right, I I agree with you entirely: okay, so here's the here's, the uh things that need to move in locks type when changing the api. So if you're changing the api- um and you know we could be changing the api but not changing the grpc spec.

A

So if you're not changing the jrpc spec, it's easier, but here's the api repository um yeah, we can go ahead and change the api, but once we do that we'll have to go ahead and update every other project, because every other project depends on this. um Otherwise, anyone who pulls code, um I think goget by default, tries to get the latest one. It will fail.

A

I think it also does some kind of constraint matching nowadays. So I think we should be okay on that front too, uh but but I would say once we once we change the api we should. We should also immediately update uh the rest of the rest of the repositories. um So so I like what jiffy was saying so jeffen can I can I rely on you to remove whatever is not being used, so we're not using bucket requests anymore, we're not using bucket access requests anymore.

A

C

A

C

Yeah, like bucket request, we have renaming to market, claim right right, so yeah I'll, look on the bucket taxes request and I'll try to remove it from the code. Let's see.

A

And the other thing is bucket: bucket access is defined as a non-name space object um in in the in the old definition.

A

So this needs to be changed to a namespace one, uh and there are fields that have changed. It's not the same as what it was before.

A

So we need to make sure that is done correctly as well. um Yeah, that's about it. I think apis classes are pretty much the same. Nothing has changed on the class front, um but there are some changes in in how the bucket claim and bucket access work.

A

So you know just make sure those things are taken care of carefully. I think I think the most important thing there is just that.

A

You know we pay some attention to detail and and that's about it yeah um and then, and then we we, we had a generic controller pre-built so that anyone could just implement the controller rather than having to use the operator framework or anything like that. um I don't think any changes are needed here. I'm not sure just make sure this is also taken care of um if you're updating the api.

A

Yeah and then, and then we have the controller repository, so I'm just going over the code um so that so that people wow it's two years old, uh I'm just going over the code so that people can you know whoever wants to contribute, can get started quickly everywhere. We follow the regular.

A

You know the standard kubernetes way of writing code, as in the way we distribute the responsibility into packages, so you know command package. It all means the exact same thing that that you know you would expect it to mean um yeah. So so, once once the api is updated, we'll have to update uh the controllers. Here.

A

I think we can move the specific controllers under package controllers. uh We have bucket request and bucket access request here, um yeah, so so this needs to be updated. So what will be the responsibility of the controller? Now that we have? We don't have a bucket request or a bucket access request, as in bucket claim, will lead to the creation of a bucket.

A

So that part is pretty straightforward, so the controller will be responsible, listening to bucket claim events and creating bucket objects in response or creating the actual bucket, and it's its corresponding bucket object in response to that bucket claim.

A

C

Yeah, like attaching.

A

C

Claim to the bucket yes or not like you can do like uh the people can create bucket and then create the bucket claim right like it can be otherwise.

A

Yeah I was getting to that yeah good point. uh So what's the what's the what's the latest decision on that? What did we decide? It escapes my memory right now, but let's take a look.

A

So that would be existing buckets, oh yeah, so so for existing buckets. We said we will point in the bucket claim. We have our existing bucket name and we point to that.

A

So, on the controller we'll have a bucket claim sitting on the control area. On the controller, we have a bucket claim uh listener, which has two things it does. One is the new bucket goes and creates the intermediate bucket object uh if it's, if it's an existing bucket it just binds uh and while binding.

A

Oh here is a here's, a tricky thing so, given that you know we're always, this is what this is. What we wanted right, we're always using a bucket claim to access a bucket.

A

That means bucket access, always just can point to a bucket claim and bucket claim points to the bucket. So, if you're going to use the bucket from multiple name spaces, you're going to end up having multiple references to the bucket.

A

So knowing who the accessors are.

A

How do we go about that?

A

So let's say we want to take the bucket. We need to make sure nobody's using the bucket when it's deleted, correct.

F

The the question of deletion is it's a policy one right. You can make the policy decision. However, you want okay, you.

A

Decide to delete it out from.

F

Other people, you can try not to delete it out from other people, either way. There's downsides right, if you, if you can't delete it as long as someone's accessing it, then a malicious user can prevent you from deleting a bucket or if you can delete it, while someone's accessing it. Then a malicious user can delete your buckets either way. Malicious users can do bad things.

A

Sure it was coming more from an implementation perspective. How do we implement this situation where we want to make sure we don't delete it if someone's using it? I mean there are many solutions, but we need to kind of round in on one.

F

Well, it depends what you mean by using. If, if it's the mere existence of a bucket access, you should be able to search.

C

F

Accesses that that referred to it and you put a finalizer on it and then only remove the finalizer when you've confirmed that there's no bucket accesses uh referring to it. And that would.

A

So for each bucket access we have a finalizer on the bucket correct.

F

Not for each one, just just one finalizer that says you know bucket, is accessed yeah and.

A

Then, how do we, how do we figure out the list of you scan.

F

All the accessors in the whole cluster, it's the only way. That's.

A

All the name spaces right.

F

Yeah I mean the the controller would have a have an in-memory cache of all of the bucket accesses in a whole cluster. That would be continually updated by an informer, and so you would just do a scan of that in-memory data structure and.

D

F

You could even potentially uh have like a an indexed. You know hash table to rapidly scan it. I think.

A

We can just use the existing indexer right.

F

I don't know I mean there's various ways, but the point is it's: the responsibility of whatever controller puts the finalizer on and takes finalizer off to to be able to determine when to take it on and off.

A

Yeah yeah, I think I think that's the best way, um all right, but.

F

Then you also need to be careful about. You know, never allowing a bucket access to refer to a bucket that is being deleted right.

A

Bucket x, yeah, as soon as a bucket is marked with the deletion time stamp. I don't think uh we're gonna love any new uh bucket claims to buy into it right or uh bucket access to access. It.

F

But you have to implement that somewhere right. The the thing that the thing that handles bucket access is has to look at the deletion timestamp and refuse to do anything if it's set.

A

That will be a part of the controller too the controller if it sees that the bucket.

F

There's two there's gonna be two or possibly even three controllers right: there's the the main controller that deals with bucket claims and buckets, and then the sidecar that deals with only buckets and then the and then a potentially other side card that deals with bucket accesses.

F

That would definitely be a side car because right, the.

A

Oh okay, so when I said controller, I just meant like the binary, not like individual listeners.

F

A

F

More than one binary.

A

This this is two actually so so what was the third one? um Well.

F

It could be as little as two it depends how you want to structure your side cards like within the csi world. We decided to separate out. You know provisioning from attaching those two super side cards to do those things and there's even a third and a fourth side card that do resizing and snapshotting, and you know that's how they've decided to do it. On the csi side, we could have a one giant side card that does all the things or split it out. Similarly,.

A

Right right, so I think it's okay to start with one sidecar right now, I'm just saying that, because we don't, I don't see uh them as having different responsibilities.

A

um The only thing our site car does is well this. These two two things it does one is create the bucket or manage buckets. The other is manage access, um but they're all for the same driver.

A

um Isn't it. Why are the moving pieces.

F

I I think in the case of csi, it was because there do exist, some drivers that can only attach and detach and can't, create and delete when there's some that can't snapshot and something that can't resize and so each sidecar sort of represents optional functionality.

F

One could imagine a cozy driver that only knows how to deal with bucket accesses and then doing attaches and detaches and doesn't know how to create or delete buckets. So it it's just an attaching side or just an attaching driver and not a provisioning driver.

F

If we, if we wanted to enable something like that, then it would make sense to split the functionality out.

A

Yeah, but why two services you could do that in the co in one binary itself.

F

I could I'm just yeah, I'm just I'm just telling how.

A

F

A

Why do you know why they did it like? Why don't they just have it in the.

F

Giant finally, simplicity: modularity, like you know a side card that just does. One thing is still thousands of lines of code yeah, it's it's. None of this stuff is easy and so to try to cram multiple things into one binary makes it that much harder to wrap your mind around, but if we can manage to wrap everything, cozy does into one side card, which I think we can. Then then that's great.

A

Yeah, I I buy all these arguments um yeah. um The other way is also not so bad having everything in one one binary is also not terrible. That's why I was curious uh anyway, so so then, I think I think we have a clear approach on how we're gonna handle deletions, which is uh we're.

A

Gonna, have a an index informer, so a local cache in memory cache of the current state of all the all the bucket accesses in the system, and whenever a bucket deletion starts up, we make sure that no in no no bucket access is pointing to that the bucket, which is you know, going to be deleted, and if not- and we wait until that's true so so, how do we wait until that's true? So do we just kind of uh respond to every uh bucket access update event.

F

Every creation or deletion of a bucket access should trigger a check that the finalizer did, that you know the related buckets finalizer might need to be removed or created all right.

A

That works yeah, okay, so there's the complexity. I mean it's not super simple anymore.

F

um It's I guarantee you, it's gonna be more complex than you could possibly imagine, based on based on all the code. I've seen in csi, like it's really really simple things, take a thousand lines of code, yeah.

E

A

Yeah yeah I'm uh yeah.

F

A

So uh getting back so so all right, who can I, who can I ask uh to to uh work on this, so I'm also going to be working on all of this just to be clear, um but but I I do encourage everyone else to contribute um grant. I know you made some prs recently.

A

Would you be interested in helping me out on the controller side of things.

D

uh Yeah I can help. I helped with the snapshot controller in the past as well, so I can sweep um thanks okay, uh so.

A

We have more people here, so I'm going to be signing up more people um all right. So then, once that is done, we have the sidecar. uh What did we call? It.

A

uh Cozy sidecar adapter.

D

A

Provisional sidecar yeah there you go.

A

All right the sidecast job. uh Now I mean it's remained the same, uh which is to provision buckets, go talk to the back end and create the actual bucket, and the other one is to provision access, talk to the backend and create access credentials and so on and then revoke it. So this was even back then not fully done. uh The creation and deletion of buckets was was in good shape, but bucket access creation and deletion wasn't really anywhere good. I think um that's the best. I remember um so.

A

We had a bunch of issues with the with the life cycle of bucket access, um oh right and and also we didn't have a concept of uh im based access back then.

A

So so yeah, so that also needs to be taken care here.

A

So so this also needs an update. uh The the functions are not like early said: it's not what I earlier said that that it's going to remain pretty much. The same is not true. There are changes needed here quite a bit, so so so on this side um we need to deal with iam, based authentication and also we need to make sure we update it to reflect whatever changes are going to be made in the api.

A

So I mean that's at the high level. What needs to be done here and I can see that blaine is here and blaine has always contributed to us so blaine um can I can I ask you to help you.

D

um I'm yeah, I think I have too many other tasks on my bodies.

A

Yeah, no, that's why I didn't ask ben either up front generally ben is very, very busy but ben. How about you.

F

I'm sorry I was gonna say I uh I I can't.

A

It's it's okay, um this is uh uh yeah, I didn't uh uh you know I can I can I can cover it. You know I can call for anyone anything that's needed. It was more of a making sure we had. We had.

A

We had things planned and then whoever can contribute. um It's it's easier, if I just ask directly, rather than asking people to sign up only because uh the code is kind of stale and uh we kind of have to figure things out all over again. Nobody has this code in context or in mind. Yet at this point so yeah.

D

That's why I went about this approach of.

A

Asking directly um and it's totally okay to say no so anyways, okay, that's the state of things, um I'm glad we went over the court today. So, let's, let's continue uh waiting for the review and next week uh you know everyone. I I encourage you to show up at the sixth storage meeting and and I'll definitely be bringing up this review for cozy um and and yeah.

A

Let's, let's try to push over there, and- and I was also wondering uh this- is a question for you shane- uh should we should we bring saad to take a look? Maybe if michelle is, I.

B

E

Think you should just just just yeah, because since michelle has already gone very deep with this okay, uh but if uh in next you know in the sixth georgia meeting by that time, if she still have not reviewed it. Definitely sad is also in that meeting at that time, but maybe he will uh maybe he will have more time now.

B

E

But but I mean the problem: is that every time you view it, they will have their own questions right. So if you switch a reviewer does not mean it's going to go faster, you know what I mean they will uh so I think yeah. I think michelle already getting. I think, just getting a last set of questions. I.

F

E

A

Yeah, that's what it seems like.

E

A

Okay, all right so yeah. Let's wait for michelle to review them.

E

Yeah and also they probably, they may also have internal sync as well, and you know I know, like you know sad michelle. You know they. They do have internal sync about six search things as well, so maybe sad knows uh yeah but cereal. I think normally every person has had their own concerns.

E

So even if a salary, because you still need an issue with michelle's concerns that get addressed.

A

Makes sense so how about um uh is michelle? Also at google.

E

A

Okay, okay, so that'll make it easier for them. Okay, I was just wondering I thought she was from. I thought she was from red hat for some reason.

E

No, she is from google, okay.

A

Wasn't there someone from red hat, who was uh involved with cozy early on, like.

B

Jeff as a jeff right, you have been working with me. Another person.

A

Yeah jeff another person. There was a lady.

E

E

She yeah she's not active in 630 for quite a long time now. Yeah.

A

E

She's in the uh what is that tlc cncfts yeah, so she has not joined the sixth stage meeting for a very long time. I.

A

Understood: okay, just.

E

A

Wondering, okay, all right good to know. um I think that's all from me. uh If anyone has anything else, now is a good time to bring it up.

A

Okay, um I think we can end the meeting now thanks thanks everyone, let's, uh let's meet again next week and- and I I have a good feeling about- I think there'll- be progress on the review by that point.

F

E

Thank you, bye.

D