Kubernetes Storage SIG COSI, 1 Mar 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Storage - Object Bucket API Standup Meeting 20210301

Description

Kubernetes Storage Special-Interest-Group (SIG) Object Bucket API Standup Meeting - 01 March 2021

Meeting Notes/Agenda: -

Find out more about the Storage SIG here: https://github.com/kubernetes/community/tree/master/sig-storage

A

Too, but but you know so far, we've done good.

A

I think we've gotten most of our recordings um anyways good good morning, everyone um so today, I'd like to uh start off uh with uh uh just trying to understand where we are in terms of our development uh milestones and uh after that, let's go into um one of the one of the questions, one of the open questions we had that we kind of uh the we ended on the question last thursday, so um I want to bring that back up and and see uh what what solution we end up with.

A

For that one question all right, so so in terms of milestones I'll quickly go over what the milestone is, then I have a few questions for some of you here, um so we originally uh wanted uh our milestone to be a demo, and the demo was supposed to include an end-to-end scenario so that all the different components are worked on uh entering scenario of provisioning, a bucket, creating access tokens for that bucket and then providing the access tokens and the bucket information to a part.

A

So we have a working version of this nav. However, that was based on uh earlier designs of cozy itself.

A

um After we finished implementing that we, you know had some more discussions about um how cozy should be designed and and some changes were made, so we are yet to uh uh create a new working version with the new design, because because we, uh you know, we've been waiting for some ps to get merged and and people have been uh transitioning in and out of how much time they can spend on cosy.

A

But those things are stabilizing now, and you know we can. We can have a little more uh stable uh progress for at least the next uh few weeks. um For instance, srini just one of the developers just uh changed his jobs, and so he was a little busy um same thing with uh rob. Rob has been working on the provisional sidecar and he was also busy lately.

A

Anyway, so things things are looking, things are getting more stable. Now, like I said um so, first thing I'd like to do is uh uh I'd like to get an update from uh those who are here that are writing code uh to figure out where we are um krish. um Can you can you go first and and kind of remind me what you're working on and and what's the progress of it.

B

Yeah so um over the last week, uh I've been working on uh some changes to the uh spec uh api, as well as the uh csi driver, csi adapter, um and so you know with the spec.

B

I added uh some logic for uh faking, the uh client for the grpc server, uh and this is required in order to facilitate testing in the sidecar, and so there was some cogen there, but that was pretty pretty straightforward and so more recently I've just been adding some more utility methods in the api, as well as better logging and some minor uh design refactoring in the csi adapter.

B

The next thing, I'll probably focus on, will be updating the sidecar to use the newest version of this back-end api.

A

Right right, um so once we do that, we should have uh well. You know code that is untested, but it's supposed to work with the new design, correct.

B

Yeah, that's the plan.

A

When I say untested, I mean uh end to end untested.

B

A

Yeah but but yeah, no, that sounds good chris. So so, if I understand correctly um after having added changes to move to the new design of the apis, um both in spec and in the api repo you're now you know moving the other repositories downstream repositories that use these, like you know, api and spec uh to start using the new version, correct.

B

A

Sounds good all right, so uh blaine is here today, hi blaine. How are you uh I'm? Okay thanks. um I know you said uh if you get a chance, you might be able to work on um uh the the you know.

A

The controller design um maybe take a look at it and uh you know make changes or bless it as it's. Okay, um did you get a chance to look at it.

C

um I haven't looked at it more since uh I guess just sort of initially looking into it. um My I guess my thoughts on it were um just around like if controller runtime makes sense to use if there is some boilerplate that could be avoided, and some like, uh I think you mentioned a race condition.

C

Maybe you had already finished it, but yeah yeah using it. The controller runtime might help with uh just having a little bit less code. To maintain true true, um if.

D

I um why the guys thinking I I could consider bringing up using controller runtime as well. I was wondering back then, um whether or not that may impact inclusion of the controller code in kubernetes core controller manager at some point in time.

A

Could you repeat that and follow.

D

So the intent is still to have the cozy controller being part of kubernetes core right now, that's something you need to install in the cluster right, so I I was wondering to which extent using um control controller runtime could prevent the code that is currently written to be merged into controller manager or is controller manager nowadays using controller runtime as well.

D

Okay, so controller.

A

Manager does not use control runtime when, when we say we want to be a part of upstream kubernetes, it would still be a separate component that gets um you know released along with kubernetes. uh However, uh it's not going to be a part of the core repo, except for the cubelet.

D

A

Yeah, uh because there is really no need for it. um I agree: yeah, okay,.

D

Makes sense yeah.

A

uh That being said, uh if we wear writing code in the controller, you know controller manager, which we've done before as a part of different sigs.

A

You know in the name of uniformity, we might think it. It makes more sense to just use what they already use.

A

However, you know, I would say we should go with which are the which are approaches superior in terms of flexibility and and uh yeah flexibility. Really, if controller controller runtime is better than what we already have in kubernetes controller manager. Today, then I would say we should use controller runtime, but I don't know as of yet um yeah, okay. So uh all right, let let me quickly see who else is here?

A

um Srini is here hi sweeney? How are you hi good morning good morning? uh Did you uh so you were working on the csi adapter right? I started.

E

Looking at the csv adapter yeah, I have a few more tasks on the central controller, but I need to spend time but yeah.

A

Okay, could you remind us what you were working on just so that I get a sense of where we are at.

E

uh Central controller I fixed the uh yeah, um I added the delete uh part uh and then now.

B

E

Yeah the delete uh you remember that we had those finalizers for the for the delete, so I added the finalizers that pr I needed to update because the spec has changed. So I update it. That part is done.

E

The pr still needs to be reviewed, but uh I my plan is to get uh the admission controller prn and then there is one more task on the central controller that.

A

E

Application controller.

A

E

There are three pr that I need to still finish on the central controller.

E

uh One is the admission controller uh piece that needs to go in and then the eta test. Oh.

F

E

G

E

Three are in then I'll start. Looking at the theater, I'm a csi adapter.

A

So so what kind of admission controllers are we talking about? So I know we have.

E

Yeah, this is a basic uh check uh for the bucket class default bucket shelf, so uh tell them that that can be, then anything can be added once we have that in place. Okay,.

A

So so I have a question here um about the ignition controls because you've been thinking about it and um do we want to uh prevent updates on buckets unless bucket mutation policy is set to allow I mean we don't have the concept of bucket mutation policy yet, but just a theoretical question at this point.

E

uh Theoretically, it can be done, but uh thinking about should we do that or not.

F

E

If we decide it should be immutable, then.

A

Yeah, I'm saying yeah something like that. So so, if someone were to try to update parameters in a bucket and the bucket mutation policy, was you know on that bucket was originally said to uh deny, um then um you know we should, should we just reject the request in the admission control itself or how should we do it? Is there any benefit to that.

E

um If we don't do it in the admission controller, then.

G

E

We do not yeah, we do not have a uh any good way of indicating to the user that this is not allowed right. I mean um having.

C

E

Check would not make much sense because then.

A

I see what you mean yeah. If we didn't reject it, then we might have a bucket object. That's actually not the source of truth of what's actually in the bucket or what was originally there before it was mutated.

A

However, even the mutated thing would be wrong, like the mutated bucket values in that, in that, in that bucket object would be, yeah would be pointing to the wrong values right um yeah I mean I, I think I think that's okay to do like if you were to add a admission controller for buckets, um not allowing any updates to um buckets that yeah, just any updates right now and then in the future. If you add mutation policy yeah, we can make it granular, yeah yeah.

H

Yeah, um I I think I'm agreeing with what you're saying sid, even if we don't have a mutation policy having a updated mission controller gives the user the air sooner and maybe the air will be cleaner, rather clearer, rather than a user thinking, they've updated a vr or an ad thinking that she's updated a bc. You get a clear message, saying it's an immutable resource right and I think then people will go well. That's weird, I don't know why.

H

I can't update it and there might be issue raised a dialogue, maybe a use case for mutability for certain fields. It sets the framework for for allowing mutation on a field by field basis down the road.

I

Right, it isn't isn't the plan, assuming we don't initially implement mutation to not put any fields that could, in principle be mutated, except on the storage class.

A

Right right, the idea is even with that. We don't want to allow any changes to buckets at this point, but.

I

More importantly, like kubernetes doesn't want to get involved in an api level with stuff that kubernetes isn't interested in managing right. We want to pass through stuff from the bucket this.

F

I

Yeah yeah and just like, ignore the rest like it's not relevant to kubernetes, it might be relevant to the end user, so they put it in their bucket class.

A

Right right, but bucket has fields that are that. Are uh that are not you know there are not parameters like like. Let's say you have uh uh some other bucket fields like let's, let me open that up. um I'm sure there are fields on the bucket that that don't have anything to do with parameters, uh but those that update.

I

Them yeah, maybe that's where we need to have the discussion is like do we need all the stuff that we're currently putting into the api, or can we just drop some of it.

A

um That is absolutely important to have, because you know, given that we've said uh uh you know we're not going to deal with uh fields that kubernetes doesn't care about. Why should we have those fields, yeah.

I

I feel like less is more when it comes to api design.

A

I agree: 100 agreed.

A

I mean stuff like allowed, namespaces see okay, so yes, one. Let's remove feels that that you know we represent like anonymous access mode, but really we don't care about. I agree we should remove them, um but coming into stuff like allowed namespaces, we shouldn't allow people to update this value.

A

I mean um in the sense, if they either we say that if they update it, it's it's your problem. You've said it wrong or we should just you know, disallow it by having an admission controller that just doesn't allow updates to this.

I

Wait, this is on the bucket right, so the only person who could edit this would be an admin anyways and if the admin gets it wrong, what's he supposed to do.

A

Well, they won't get it wrong if you have an admission controller. Is that my question is more along the lines of? Is that even something that we follow? In kubernetes like, like with storage classes, they're supposed to be immutable, storage.

J

I

They're immutable right right, they immutable so so, but but but further than that, like storage classes are only consulted at creation time. So even if you could mutate them, it wouldn't matter like the reason they're immutable is to avoid the confusion that would result from people mutating them and nothing happening.

A

Fair fair, so so, in this case, allowed namespace is brought in from the bucket class and um let you know once it is set on the bucket. If someone were to go and update it, what should I? What should our uh response be? Should we so.

I

This seems, like I mean I, it all depends on if this is the correct mechanism for sharing buckets across namespaces, which I understand is up for debate, given some of tim's feedback, but assuming this is the right way. um I don't see why it wouldn't be mutable. I mean things like the deletion policy are inherited from the class from the bucket class, but then they are mutable uh typically, because if the admin changes his mind, you know.

H

The allowed namespaces is in the bucket instance, too. It's copied from the bc to the button yeah, whatever.

I

That's that's what I mean yeah so like so so with the with the snapshot. When you know you have a deletion policy in your snapshot class and you take a snapshot and it gets the deletion policy from the snapshot class. But then, if you, if the admin wants to change it, he can because the admin is always right um and and all all that that field is telling you is what should I do when I delete this thing right.

I

So if the admin wants to do something different, there's no reason not to give him what he wants. Similarly here, if, if the purpose of the allowed namespaces is so that some controller knows what can bind to this, um if the admin changes his mind, I don't see any reason to not give him what he wants, because the alternative is. The admin has to just delete the whole thing and go make a new one, which is which he has the power to do, but.

A

You might have to lose bucket data in that process, which is not good.

I

No, no, I mean the the admin is all powerful right. He can. He can just delete the object and replace it with a new object that is identical in every respect, except different value for that field.

I

Sure, and- and you can't stop him from doing that so like, given that we are empowered to interpret the change that he wants like we should just let him make it the main.

K

Reason we are, but here we are saying we are keeping the physical bucket right. It's really just to change this on the buckets back without changing the underlying uh bucket, so I mean not without deleting it right. We keep it, but just to change the the name spaces. Some permissions on that bucket.

F

K

It right so, but I think the question I have is do we know. Is this supported by everyone? uh The or the object store can support this kind of thing you can actually change it afterwards. Should? Is this really a problem at all? Well,.

A

A lot of namespaces is internal to kubernetes yeah. It's all a kubernetes.

J

I

It has nothing to do with the storage, so.

K

uh So you don't really have to change that. Okay,.

I

Yeah the way I look at it is the reason to make something. Immutable is if it would be hard for kubernetes to affect the change. But if, if it's easy for kubernetes to affect the change, then.

A

Then do it okay, so let's talk about protocol structure then so whatever is in the protocol structure is what ends up going to the application.

A

F

So how about that one.

K

Probably can't I don't think it can, but the problem is, do you know the underlying uh storage? You can change it. I guess that's the question. Well,.

A

In protocol- let's say you know, region is said see the idea is protocol is supposed to be the source of truth for how to access the bucket. Now, if we say admin can go and change it, um one we might be allowing them to do something that that you know cozy got wrong or they didn't set initially or something like that, but two that could also be a bad thing. If uh you know they make the wrong change well, the way.

I

I would look at it is the the protocol is: is information that's going to get passed through to pods that are attached to this bucket? If a pod was attached at the time you changed the protocol like what is supposed to happen um like? Obviously, you can't go muck with a pod. That is running at least I I think you can't um yeah so like that would be a good reason to say.

I

Okay, you can't change the protocol, because, if there's any pods that are using it like your changes will not be reflected in those pods, and so it's easier just to disallow mutation of the field and say look. This is a once and done field, um and if, if you want something different, you basically need to go, create a new bucket and attach your new pods to the new bucket and then they'll get the new values and that cleans up any confusion around what to do about existing pods. um Okay,.

A

But so in this case of protocol, we're saying that admin can't make mistakes like admin can't change it. Even if I mean they really wanted.

I

To they could but yeah I'm not opposed to allowing it but like if we can't define what's supposed to happen in response to that, then that's.

A

A problem for us new value goes into new parts. It's like it's like no schedule. Instead of like no, it's like only update on new schedule, new parts that start up with the protocol rather than existing ones. I mean that could be impossible, but.

K

The packet, but the back is back you're saying this is like managing one. The bucket is not you could not going to create new ones register. Well,.

A

In the sense that uh you could have one bucket that's accessed by different applications, so let's say you have one application which which mounts the bucket, which gets all the bucket information, and then the admin goes and changes something in the protocol and then a second application requests. This, you know, gets access to the same bucket. In that case, the question is one: should we even allow the admin to make such a change or anyone to make that change in the protocol or two.

J

If so, what's the behavior, we should look at what happens with pv specs, because that's the I.

K

Think we need to be careful. We actually just run into a bug the access mode in the pv pvc, uh so we found out that uh so in pvc, you are not allowed to change the access mode. However, in pve you are actually allowed to change pv access mode. Not only that, I have to change that. The access node in pvc is also changed. So now that becomes a bug.

A

K

You mean like, when you say.

A

Access mode like read, write once rewrite money. That kind of thing.

K

Yeah yeah, so actually this about that, we actually need to try to figure out how to fix it, because our customer actually opened this bag. So I'm saying we need to be careful to say.

I

K

You can allow the change in the bucket.

I

Fair enough, my inclination is to follow the pvc pv model if, if pvs are generally mutable in terms of the the low level like csi information that comes back from the csi driver, then then we could copy that. But if they're not, then we would probably want to copy that behavior because um yeah it's it's complicated, I mean. If we did allow it to change, then it would have all the problems. You just mentioned that it would only affect new pods and the old pods would get the old values et cetera.

K

I

Curious about the issue that you raised shing, but we shouldn't discuss it.

K

I

K

Should discuss.

I

It somewhere else.

K

Yeah yeah that that's actually you know the issue, I actually we need to investigate that one so uh yeah, but actually it's a problem because the volume is already created. That's only supports one mode. Now it's a change in the pv. It's actually not compatible with the voting cell. It's the problem.

I

But but there are, there are situations where having the ability to change it um gets you out of a jam where, like if yeah.

K

So maybe then that maybe should be a capability or something because our driver doesn't afford.

I

So so anyone who makes that kind of a change would would be um admin they would be taking the risk on to themselves like they would have to know, but.

K

In this case, the pvc was also changing, which is pretty strange. That's actually yeah.

I

I have no understanding of. I don't understand how that could happen, but yeah.

K

That's that's not designed. Actually, I would just just assuming that.

I

Assuming that it wasn't that way, and it was just that you were allowed to mutate, the pv's value like I can see cases where that's a nice feature, yeah.

K

The problem is, you know, the problem is in the pve. We never really go call the driver like, even uh if you you know, even just for static provisioning case right no.

I

No right right, the whole end the whole. That's.

K

A problem actually.

I

Changing the access mode is not a feature that you allow but like if you happen to know that kubernetes has the wrong value relative to whatever.

K

I I know what you're saying this can go both ways, but then.

I

K

Our case, this is the problem we actually have to go. Do something to prevent users from making this mistake. Yeah.

I

We don't want users to think that, like mutating the the protocol in the bucket would actually make anything happen right like we would need to be very clear about you know. This is just this is set by the controller. You can change it, but if you change it to the wrong values, all bets are off right. It's it's on you because you screwed up your bucket, but there are situations where, like due to you, know software upgrades and new revisions of this of the cozy driver.

I

The value of your bucket might not be perfectly accurate. You might want to fix it, but.

A

If that's the case like, we should call it like bucket upgrade policy or something like we should have a way to upgrade buckets. uh No, that that's over.

J

Engineering man, just no.

K

I think that's actually better it's safer because otherwise you're.

F

K

Existing pv design- I think that's a problem they assume adam, can knows everything we found again again in customers. Environment. That's not true, a lot of bugs open because because actually the customers actually have access to those. uh You know me no name space object, they change it and I mean mistakes. They didn't know that.

A

Yeah, yeah, yeah and- and I think for now, like ben is saying I think uh bucket upgrade- is a little little much.

I

um I mean it, it's the philosophy of you know: do you give the users a lot of rope and let them hang themselves or not yeah, like yeah.

K

I would say not to give them.

I

And that's that's always a problem with usability and support, but like in kubernetes in general, we try to just say you know: here's the api use use it the way that we documented and everything will go right and if you go off on you know and try to cook up your own solution or hack on stuff that you're not supposed to hack on. We don't promise that it'll work right, yeah.

A

And- and you know, if usability is, I mean we, we I mean, as as the developers of the kubernetes platform itself, um I think we should we should design for maximum flexibility rather than usability, and I'm coming from a very uh like vendor perspective, saying that usability is should be something that that vendors figure out I mean vendors can write controllers to make this uh more usable. They could add whatever admission controllers they wanted, but we would provide maximum flexibility.

A

Like you know, you can do whatever you want, but if you break it, it's on you.

I

Yeah and my guiding principle for for the mutability of fields is like it. You know it should be mutable unless there's a good reason. It shouldn't be, and the kind of thing that qualifies as a good reason is.

I

If, if it's totally not obvious what would happen in response to that or if it's going to create new paradoxes, um then then you'd say: okay, you can't change that, but I would default to mutable. I I I.

A

Get that, uh but it still doesn't answer the question of what to do about.

G

I

Yeah, I don't want to take a hard stand, one way or the other yeah like like like I. If it was mutable, we would just have to document that like look, this is you know if you change it, it's not going to affect any existing pods, etc, and if you make it immutable, then we just have to live with the fact that the only workaround for when it has the wrong values. People are going to be deleting these things and recreating them with new values.

A

I

A

Sounds really bad.

I

That's not going to create new problems.

A

Yeah that sounds pretty bad. To be honest, like let's say you know, we got, we got the region wrong or something- and uh you know some so so, for instance, uh state for instance, okay. So this I think we should leave it mutable. I just thought of a use case where that might be needed, so so say. For instance, you are migrating from one cluster to another, uh and then you migrate, the bucket request, which is not the bucket itself and one cluster is on the cloud.

A

One is on prem, so say using aws and min io. um The concept of regions are not utilized in manila, while provisioning buckets, I mean yeah. In the general scenario, everything falls into one region.

A

um You could create geographically separate clusters and uses the regions, but, let's just say, there's only one region and the default region is usc one and uh you know if there's only one region, it kind of gets ignored, um but during an io call, if you set the region wrong like when creating the bucket the region gets ignored, because only one reason, but then, when creating buckets or when using the buckets, I o calls you set the region wrong, I'm not sure, but it's possible that you won't be able to access the bucket, because it'll think you're trying to ask the bucket that doesn't exist.

A

um So so, in a scenario like that, you know, let's say: you've created a bucket put some date. You know whatever brought in some data into it. Somehow, and you know it might just be easier to go to the protocol and update the region rather than rather than having to go, create the whole thing again.

I

Yeah and someone.

D

Who knew exactly.

F

I

Were doing could do that safely and someone who someone who didn't know what they were doing could royally screw everything out so.

J

I

The two edges of that sword.

A

So so shrink um so you're saying it's better to you know prevent them from making mistakes like like that's what stuff, like that's.

K

What I thought I think it's actually, I think it's it's better to start with immutable. So I'm like opposite with what they're saying, uh unless, if those changes are like retaining policy that I agree, you don't really have to go check uh and then and then the allow allow namespace, I think, seems to be finding the protocol. If those things you might need to go check, if your underlying object logic can support it or not those type of things, I think it's uh dangerous to just allow them to change.

K

This may end up have something that is broken right away. So.

A

Yeah yeah, that's true yeah, you know if you change it and then you clone a bucket and then you're like. Oh, I made a wrong change, so you go back and you know update the original, but then it's not going to propagate to the clones um yeah stuff like that, can also happen.

A

um Yeah now that I think about it, so I think it might be okay to say you know what, if you, if you got the values wrong when you provision the bucket uh it's on you now, so you got to go, create a new bucket to fix it rather than saying here. Go ahead, make the change.

I

And to be clear, though, the way that you get things wrong is, if you have like an old version of all these values, come from the cozy plugin, so you'd have to have a buggy, cozy plug-in or a cozy plug-in. That was missing some important feature in order to get incorrect values and we don't have control of the plugin, though so.

I

But vendors will write buggy plug-ins without a doubt, yeah.

A

I

A

We make mistakes too anyway, so so um yeah. I think I think the I think we're leaning towards saying this is immutable.

A

I don't feel strongly about it. Okay, let's go with that. I think shin feels strongly about it and uh when I think about that that one use case of having you know cloned copies having you know different data that that that seems like a very bad problem to deal with.

I

I I would recommend looking at what how we handle pv specs. Okay. How is that, if someone can go, I I don't have a. I don't have a test bed set up where I can just easily go test this, but um oh you mean that way.

I

Yeah I mean you could just go provision some pvs and try to try to edit or create some pvcs. Let them bind and try to edit the pv directly and see if it blows up. I don't so.

K

There are certain certain fields I think most fields are actually immutable. Some feels mutable, so you can actually go check because I was actually I did some experiment with this.

A

K

A

How do you make them immutable.

K

uh The api levels, uh I think the there were some checks. There was a.

I

K

You have to write.

I

A validating web hook that rejects edits to those fields right, so if.

K

You're, a crd, yes, you have to write, but this is entry right entry, so it's actually relatively easy to prevent it.

A

Right right I mean like it's the same thing right. We probably have an admission controller in qap server itself, yeah.

K

Yeah, so that's but kind of building. So it's easier to first to just write some some tests.

I

Like but if you have any validating web hook at all, adding these particular checks to it is very simple: it's if you don't.

K

Have the already have it yeah is that did you say actually, maybe.

A

It doesn't say in the spec behind the api, but I bet it's there in uh it's there in actual kubernetes.

E

J

I

K

Nothing, you haven't done, it's gonna.

I

Be it's gonna be hard to find that code. Otherwise I would look for the code myself. Oh I've.

A

Written something here so I.

A

Oh, it's changed yeah. I wrote it like four years ago or something I'm.

K

Just saying that maybe take a look of the the. If you want to check a pv and see if the the api definition actually mentioned, which is.

A

No, I just checked yeah, it doesn't mention nothing: okay, no, but persist, and volume.

K

Because I know there are some fields like the: I think there is some some node affinity thing. I know that is actually we would like that to be a mutable, but then that is actually immutable or something so I know that it's a different depending on which field you're looking at basically now. This one is immutable because we already tested this one, but then we have this problem that.

A

Is important now yeah um is the storage class name. This should be immutable right.

K

No, you cannot, I don't think you can change this now.

A

Okay, so okay, but it doesn't mention it though so so I guess what we should do is like, like you were saying, validating web admission, controller.

K

um Like the node affinity, that one is immutable that one you know so so some of those are mutable. Some are immutable yeah, this one is. We actually hope this one can be changed, but then this one is actually immutable, so yeah so yeah, so it can go yeah. The problem can go both ways so, depending on so you probably really need to look at exactly.

K

uh You know the what field you want it to be mutable immutable and what's the processing of language.

A

Yeah, I think we should start tagging ghost truck like with immutable immutable uh fields that will be the best documentation for it and we can respond to it in code.

A

um We'll do that in cozy, yeah, we'll say you know, we'll have a parameter like immutable or yeah just immutable if it exists, you know it's it's true, something like that.

I

um Do we already have the validating web hook framework in place where we can implement that kind of stuff, or is it like? Is that beta material.

K

Right, so that's the alpha, so that's we are uh like for snapshot. We added that.

I

For snapshot, we added it like only for ga.

K

Right right, we added that later yeah, so so.

I

For alpha and beta, there was no validation at all.

K

A

I see well, srini is currently working on admission controllers, so I would think we should just you know at least have the control in place. Maybe not have all the rules but yeah.

I

It's also also as long as we're in alpha. We can easily change it if we're like. Oh we screwed up here. It's once we get to beta it'll, be harder.

A

I

But even then, like changing an admission controller rule is not is, is easier than like changing a field right. Oh so.

A

You're saying we you shouldn't have the truck tag here.

A

I was thinking we'll add a new struct tag like you know here, like a field type of those.

K

Forward, I'm sorry.

I

K

Think that I don't.

I

K

That the code cares.

I

This is all for the code gen.

K

Yeah, that's not yeah, you don't add it here. Actually you need to have an extra.

A

But you know no, we, you know to code gen, we can, you know like just like we added here. We just add the good thing about go. Is you know.

K

What hold on a second actually, maybe that's actually available. Now I have to check uh maybe the latest the api. The schema, maybe maybe supports that. I I have to double check yeah.

F

K

Think there are certain things we have still have to use the validation hook, but certainly things like a mutable immutable might be already built in so I won't have to actually check the snapshot. I forgot yeah.

A

Find that out, yeah yeah. If not, you know you can quickly write something that will do it for us code generator. I think.

I

It might not be the code gen, it might be the if it gets embedded in the metadata for the ghost structs.

F

I

Maybe the off off the shelf uh validating web hook knows how to look at the metadata on their structs and implement certain checks yeah for.

A

Free metadata is pretty straightforward, like you know, it's just in the um reflect package. That's how like json does it. um You know, struck field yeah there is uh struck field, has a tag. So just look at the tag.

A

A

Yeah, so you can, you can have custom tags whatever color blue yeah, so you know we can quickly. uh Like you said we will have the admission controller, see what fields are coming in and if the struct tag for that field is you know immutable, then we don't allow it automated and clean and we'll know it works. Every time sounds good like what do you all think.

I

It should work, I mean I, I have a sneaking suspicion, we're going to want it to be mutable later, but I don't have a problem starting this way. Okay,.

K

Half alright.

A

Yeah I I yeah. I have that same suspicion too, but again, the the cost of you know getting mutability wrong or getting the changes wrong is very high. If you ask me, that's the reason I want to make it immutable actually like if this field is changed in one copy of the bucket, but not in the others, and and you want to change it back or something it gets tricky.

A

So yeah, I want to start with immutable.

A

All right so so, uh coming back to uh the original question that we had, which is uh you know, we were talking about reducing or taking out some fields that really kubernetes just treats us as pass through, um for instance, anonymous access mode.

A

I I would, though we should remove it and not have it here. Rather it can be a part of the protocol or parameters.

I

It feels like a parameters thing because you set it once at creation time and um and it's not relevant to the individual workloads.

A

A

I

Because the workloads are going to obtain access through the mechanism of being bound to the bucket, that's the whole point of the the bucket pod binding is is.

A

That access there could be some scenario where they might want to know if it's publicly well, I don't think that this is the best way to give it to them. Any yes, yeah, no fair enough! uh What about? Why should we in parameters now, because earlier we had a, I mean this? Is we had this pretty restrictive definition of parameters which is fields that apply to uh bucket creation that are that are render specific.

A

Learning's access mode seems more universal to me, or it is more universal.

A

My question should be in protocol or parameters.

I

Sorry I was running an experiment. I just proved to myself that the pvs are immutable, so good point of comparison. What was the question.

A

So anonymous access mode should go in parameters of protocol.

I

I say parameters, it's an opaque thing that kubernetes doesn't even know about.

A

uh But that could be set of the protocol fields too.

K

Oh so, but you already have a parameters that is opaque right. You need.

A

Yeah the parameters we we defined it narrowly. We said it's for specific fields, anything.

K

Product was supposed to be like first class field, yeah.

A

Protocol first class fields: yes,.

E

I

I'm willing to have arguments about the individual components of protocol too, and whether we should take them.

A

Out, okay, yeah: let's do that also right, um I think I think that's what.

K

A

K

I said what ben was saying: what do you mean? You mean move those fields out, making them first class outside of the particle struct. No.

I

No, no just just dropping them right like.

K

If we don't really need.

I

K

I

Necessary for the functioning of the sidecar or the functioning of the bind controller or.

A

I

Then then, why are they there um fair.

A

Enough all right, so with that argument ben I I guess I know what you're saying so this goes away and you know we expect it to be passed through parameters. I I like that. Actually um anonymous access mode goes away and then going into the protocol struct itself.

A

um The definition.

I

Of version is kind of tricky right now I I I I I do not like version in here and would like to see it.

J

A

But but I also want to get a sense of what uh shang things and you know, jeff also.

K

H

Some of those changes are in the cap, if you guys could look at the pr on the cap. Several that have just been mentioned that are in one of the commits on that cat pr. So I'm just waiting for reviews on that. Okay,.

I

What do you mean? You mean recommendations to.

H

Remove these things, this discussion is sort of a repeat of earlier ones, and earlier, when we said allowed, namespaces is, I mean, excuse me um anonymous um the anonymous month. Access mode was, uh should not be uh a first-class field in the api and as part of my kept pr, I have taken that out and there were similar things like default for a bucket class as a first class field. The feedback from tim's review is that we should do what is done for storage classes and make it an annotation rather than a first class field.

H

I

I've made I've.

H

Made about a half a dozen of changes to api and rpc fields in the latest, kep pr.

I

Okay and what about the version field.

H

No that's a new one, but I'm happy to add. I mean that would be a good place to collect feedbacks in a public setting where, for instance, then you could read the pr and add your comments on the changes, but then also add a comment that hey: why isn't version just like anonymous access mode if you're getting rid of anonymous access, get rid of version? You know things like that would be useful.

J

I agree this is a good forum to have that conversation.

I

um But I I'm not opposed to to writing up this feedback in the cap, um but but I I think I I I'm.

F

I

Finding contrarian viewpoints on the version field before I just say this.

A

I

Let's get rid of it right.

A

Right yeah, that was my point of view to asking everyone so so shin I'll, give you some background on what this version field was thought of originally and why it doesn't make sense anymore. So originally we thought the version would be like like um so, for instance, with with aws api, we have a date as a version like 2011, something something and that version, even though it's not passed in through any of the calls to aws itself.

A

It denotes what version of aws api we're using and then it gets tricky when we go into s3 protocol, because inside of sd protocol there's another version field or we luckily removed it.

A

But we actually had one more working plate which you've taken out actually, so that version was supposed to be um something specific to s3 and it started.

J

I

A

The signature version field, the.

I

Signature version field, I'm getting a weird.

C

I

Okay, I think it's coming from jeff's microphone.

G

I think it's coming from jeff's microphone.

A

You said it twice.

G

You said it right.

G

K

G

Is mute I don't know.

K

The chief is not mute yet. Okay, that's better.

I

Okay, so so the yeah, the protocol version or sorry, the signature version is absolutely necessary and because today we have s3 v2 and s3 v4, and we don't know if there will ever be an s3v5. But if there was, you definitely need to know about it right, yeah. You need nobody to make every call yeah yeah. You need something.

A

K

H

I

Version I absolutely support. Keeping here, did you say.

A

J

Another version.

I

A

Was another version here? um What was that it's tricky? I I couldn't tell you to be honest: okay, um yeah.

F

I

F

J

I

It the amazon, sdk version.

A

Something, no, no, it wasn't amazon, sdk version, so so, like um aws api version.

K

And this is only specific to aws, you don't have that for.

I

Well, no, like the whole point of s3, is it's supposed to be for anything that supports s3. It's not supposed to be aws specific.

A

Yeah, like this like like this, this is a this is the other origin field.

I

Okay yeah, so so initially, this was the version that was suggested we put in the top level version at the top level of the protocol. Something like that and- and I never saw the point in that so I agree- I would be happy to remove it um and and if and if jeff wants me to write that in the in the cap, I'm happy to put that feedback there, but yeah.

I

I don't. I don't see how a workload could consume that value and get anything useful from it and I'd, and I don't see how a cozy plug-in could supply a value that was meaningful to workloads um in that field. Yeah cause.

A

All of these have to come from the.

I

A

All the application could use is the actual protocol like signature version, but more than that yeah all right, so yeah yeah. I think I think we are more or less in agreement me and you ben um uh jeff, you you, you didn't, give us a response for what you thought. So you said we could comment on the cap. But what do you think about removing version?

A

H

Know enough about it: okay, yeah, but I'm I'm very open to listening to um both sides of it and and then forming an opinion right now. I don't have an opinion. Okay,.

I

H

E

Remember I remember.

I

Oh we're getting echo from jeff again I thanks jeff. I don't know what's wrong with your audio, but thank you. I remember that the gentleman who was arguing for it and it was.

F

I

Aws s3 specific motivation, and it had to do with like knowing whether certain extensions were available after a certain version, and my argument back was like well. If we want to support specific extensions like we should do that, not just try to convey that in some version.

B

I

A

Right, right and and uh the thing is there is no such thing uh like, like specific extension based on some version number uh either it's a part of the you know s3 v2 or s3 v4, and that is a standardized api. If it's not in them, then it's it doesn't mean anything like if there.

J

Is an extension.

A

J

Aws does implement.

A

Like funny extensions, well, the extensions could be for other things like authentication but yeah, uh but but but we support only one authentication model right now, tokens and later on. I am like those two work with any version, so I don't see.

I

Right, I'm just I'm just representing the stance of the the guy, whose name I can't remember. The argument was that if you want to take advantage of aws's special magic, then you need to know if if the version is high enough- and- and I I say well- no you, let's just not do that.

K

You know so why is this version at this level? It's a but you're, saying this. Is it up? It's an s3 thing: it's not really for azures.

I

Well, the idea was that maybe every protocol would have something like an s3 version, so put the common stuff up here, but yeah. I think we can not do that. I don't know if you can make the same argument about the name field I mean. Obviously we need the oh. No I see the name field is your. Is your structure?

I

That's not the name. That's the protocol name, yeah, okay, okay, never, mind yeah! So so yeah, all you need at the top level. Is the union discriminator and then all of the different union members um and virgin can go. I agree and- and I I'm happy to write that in the cab.

A

Okay- let's do that then then. uh Please write that in the cab and jeff, please remove it from the cap and.

H

In the review, I just want to keep it as much public as we can, so it doesn't look like um changes are being made without community input. I mean the community is right here in front of you, jeff yeah,.

K

That's the same community. There are people that not everybody's here.

F

Yeah yeah: no, no, there's another reason that you know it's like. I want to encourage the house.

H

For uh equal voice, when people can't attend meetings in person when there's time, zone, differences or other things come up in certain meetings on a regular basis, and so the kep github is very good for bringing in other voices.

I

Yeah, I absolutely agree that it's more searchable, there's more history. You know no one wants to go, read or listen to hours of meetings on youtube that have been recorded. So so I I don't disagree there, but um there has to be a balance, though, because this is this is higher bandwidth than github reviews by a long shot, yeah.

A

Yeah, I think, for major things. uh You know like api changes. I think it's it's, it's good to write it down and think, but other things I guess we don't. We don't need to necessarily write everything down um for this version field. Let's, let's just write it down and just up you know make a comment on the cap. That should be good enough.

K

You just see an update from jan saying that the the access mode on pve seems like it should not be editable, and this seems like to be something missing from the we'll see. If we should block that or not.

A

So um going forward so within the individual protocols trucks, I don't think we should go in and check what should be. I mean, like I, don't know if we should. We should start reasoning about removing these fields at this point, yet.

A

Like like, the exercise we started off with was was seeing what's what's on necessity and then taking it off the remaining.

J

I

In in the s3 union, make sense to me right, you need all four of those. Let me see them again, real, quick, you need the end point, you need the bucket name, the reason you need the region, you need the signature version yeah and then, of course, and of course you need credentials.

J

I

Key, but that goes in the same.

J

I

But they go in a separate channel, so yeah those are the things you need. Yeah.

A

Yeah, these are absolutely new, yes, um and- and you know some people might say, the region is a part of the endpoint. That's not necessarily true especially say. The end point is like the gov cloud um in you know: aws uh it's a single endpoint, but then the region can be any one of the many regions it supports, rather than the general aws.amazon.com.

I

And if, if you're running, if you're running a s3 api in a port other than 443, where would you communicate that information? As my.

B

I

Field you'd have a colon in the endpoint field,.

A

So it should be a url, maybe it should be. Should you make it a url.url?

A

I don't know I mean it's subtle.

A

I mean it'll naturally lead to adding. um You know validation, because uh you know when you parse it it can lead to a power setter.

I

But like are we gonna allow http, as opposed to https.

A

That's custom validation uh like with this. We we don't check even and then http is a valid uh value ben because um you know in test clusters- and you know in air gap, setups people, people just use http.

A

I mean not to aws cloud itself, but let's say you know mini or some.

I

There probably are implementations that don't even support http, but that wouldn't matter here, because they would just never return that as their as their end point. So that would be fine.

I

I guess the problem would be if there were, if there were clients that couldn't support http or clients that couldn't support any a port other than 443, then supplying those things here would break those clients. Wait. How so I mean if the client hard codes, port, 443, and then this thing tells you it's running on a different port. What are you going to.

F

I

That's that's not cozy's problem, though right right, I'm just saying like should we you know the consumer of this information is an object, storage, client and if, if object, storage, clients have made a choice not to support something. Maybe we should ask why? But but if everyone supports http and non-standard ports, then then it's perfectly reasonable thing to allow. In general I mean amazon doesn't support non-standard ports, but it's amazon.

A

Right right same with like major clubs, but um well actually I couldn't even say that about amazon, because amazon has an on-prem version of amazon itself, where you get the whole cloud on-prem and there I guess even there is https, but it's a custom. Endpoint.

I

F

I

F

I

In front of amazon yeah, we got a time check. Is that what you're saying.

D

Yes, oh we're already out of time. Okay, yeah, quick, quick comment on this one as well is signature version, unique.

D

As in from what I recall at some point, you could do both s2 sorry version 2 as well as version 4.

F

Three requests well.

A

No definitely it's.

D

A

It's up to the client.

D

A

Yeah, so so yeah, that's those are the plan to choose. So if both are supported, that means s3v4, so uh the s3 v2 signature was easily. uh I believe you know people could uh um you know without knowing what the actual key was you could you could actually end up signing it, so paul's client could pretend to be a valid client um and s34 is a new version and at the point both are supported to allow uh clients to migrate over from one to the other.

A

So that was like an interim thing, but right now uh I think it's still true where you can use both um like mineo supports both as well, but the expectation is it'll, get fully deprecated and you know not not functioning in the future.

D

But what happens if there is a v5, and there is a transient.