Kubernetes API Expression Working Group, 2 Feb 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: WG API Expression Bi-Weekly Meeting for 20210202

Description

WG API Expression Bi-Weekly Meeting for 20210202

A

All right welcome to looking group api expression, it's the 2nd of february 2021, and we have things on our agenda jeff. If you want to start.

B

Sure sure um so just wanted to give a quick reminder that the enhancement freeze is exactly a week um so february, 9th next tuesday.

B

uh So I just looked at what we have um I'm updating the ssi ga cap, but I also saw that we have, um I think, two other caps that are also meant to then one twenty one um one wouldn't see client go apply that one. I think it's in a pretty good state.

B

We just need to decide on which method um these drugs for the builder space method we are going to use, and I think that's all we need to update for at least for the enhancement, queries manage fields, I'm not sure about the status of that one. um Just wanna, like sort of just confirm um all the caps that we want uh before the.

C

C

Which one do you want to know about the.

B

C

B

C

It's kind of stuck for now. uh We don't have consensus, I don't know if we're actually going to do that, um we might delay by release or to um if they decide. If, like the people who decide decide, they don't want it, then it's probably not working anyway.

C

um I suspect. If we only need to check one flag on good battle, uh then I don't think we need the cap and that's about all I have to say about it. I think we should be good I'll I'll, try to see if we can get consensus, but it's not. If we don't. I think that's fine.

B

That's it. Okay, I guess, if that's the case, we'll just drop it from the requirements, then correct. Yes, okay, I think these are the only two um did I miss any caps last one. Oh sorry, go ahead.

C

Yeah, sorry, I um I I don't know if you saw in uh sega api machinery channel, but anna said that there are five enhancements obtained by api machinery and server side apply is known as at-risk.

C

I'm not excited to see it at risk. So I was talking to anna and she said she said the cap needs to be updated with pr whatever. That is, that's planned, oh.

B

Yes, I have that.

C

Okay, sure, okay, let's try and get them to show as already as possible.

B

Yeah yeah, I have like a work in progress. Pr already um cool awesome.

B

Yeah! That's all for me.

A

All right um I've I sent the kleinko stuff too, to some openshift teams on an internet that I know wanted to look and apply and um yeah. I hope to see some.

A

A

All right, then, um let's switch to where is it uh no.

A

uh J requirements.

A

How do I switch to a different tab.

C

A

A

A

Okay, we have an update here right. We had that.

A

Is andrea here yeah.

D

um Yeah, I've updated the pr a little bit. I think that now it's a matter of iterating and get it done thanks antoine and kevin for the reviews.

D

um I think I have to do some changes according to the comments that left there, but yeah don't have many updates other than that.

A

A

Yes, we talked about.

C

Thank you, andrea, because um I think the solution we have now is going to work, and it's just a question of how do we do it? Yeah yep.

A

um I don't know how the pr itself is doing here or, if you're, only waiting on the cap right now.

C

uh There are two different things there is, uh that is the cap actually three things. That is the cap that describes what we want to change in call and on the api server. There is a pr that changes the code on the api server and there is a pr that changes uh could cuddle.

C

I think the pl that changes good call needs to be updated so that it does what we sort of agreed on the ccli channel, like everyone seems to agree that we want to have a show managed fields come in, um so you need to update the pr. So it does that.

C

And uh that's about it.

A

All right um and the cap is the only part of this that is currently putting it at at risk because it's not merged yet right.

C

Yeah but I mean that the capital change does not probably doesn't require a gap.

A

A

um Dangling field anything here.

C

I'm I'm almost done with that. um I actually have all the test passing now. I have made a small change to how we track some specific fields which uh I I'll discuss with joe.

C

I think it's fine, but basically I'm probably going to make a pr for that today.

A

Cool yeah um I've been looking at that on on saturday and fighting most of the time to get the test running locally, and I still don't know why it doesn't like. No none of the admission tests work locally doesn't make any sense, but it works in ci and I've.

A

It slows me down a bit, but I'm trying to find to reproduce at least a broken web hook that sends involved managed fields and from what I've heard from.

A

Jordan, it should be, it should cause validation to fail, but our validation currently doesn't look at all of the managed field fields to be valid, like it doesn't make sure fields. B1 is even present.

A

So once I consistently replicate the problem in the tests, I'd try to change our validation to make sure we actually get well administered fields. um Yeah sounds good. I saw your question. uh I think jordan didn't answer. I I think validation runs after all the chain, but I'm not sure because it.

D

A

Mutation twice, so I would suspect.

C

I'd like to understand that.

A

um What we want to do in, in that case like well well, there's there's only so much we can do if it really runs at the end it fails, but we can't reset in a case right. We will always fail validation, then so we're it will collide with what we wanted to do now, which is resetting the managed fields.

C

Yeah so wait: what can you repeat that.

A

um If validation should handle it, and we extend our validation to really validate against the problem of corrupt managed fields, which it doesn't really now, then uh the web hooks or the request would get a failure instead of what we wanted to do, which is resetting them. If, in case of admission failures of admission, sending in complete managed fields.

E

C

I I don't know yeah. This is all super fuzzy in my head. Somehow um I we don't it's it's kind of different, because I don't think we can fail yeah. Is it different from a controller updating like? Are we not happy with just the validation as it is.

A

The validation as it is, doesn't validate against the corruption that a weapon cost.

C

A

Do that field 31.

C

We do that later in the process. Right later, when we actually look at the managed fields, if they're invalid, then we trap them is.

A

C

Correct okay, yeah.

A

C

A

C

So we should probably do that after all, the mutation has, if we don't do it after the mutation happens, then it's going to happen the next time someone tries to look at the object, but the problem is someone might read the object with infinite things in it.

C

See what I'm saying.

A

C

A

So so I I don't think we should ever persist in valid managed fields.

C

That seems reasonable.

A

And that means we either have to reset them or decline, requests that would persist in valid managed fields.

C

Yeah, so we do not decline ever requests because of that like, even if today, if you send an object and everything looks good from the managed fields, entry perspective that the actual set is not valid, we are not going to reject your request.

A

Yeah, what I mean right now, we check for api version and field manager name length, but we don't check if fields we want, or even there yeah and that's yeah and and that's what the corrupt repo cost like it sent the patch that removed api version and it removed fields. V1.

C

Yeah we have a specific check for that and I can't remember how it exactly works, but I suspect yeah. Okay, so let's say we should do after the mutation exactly what we would do when we receive a new message yeah and that is uh run the validation and maybe after all of them I mean I don't know if validation runs after each individual mutating webhook, but at least for sure, after all of them, before we persist, we should validate that we should trap the field sets that are invalid.

A

E

D

A

We dropped them instead, like we can't just drop the fields that are invalid, because then we would have unowned fields, we would have to drop the entire fields and take the old.

C

Oh, I don't think I don't think that's so bad. Is it that bad?

C

Oh, you want to reset you. You do want to reset. Aren't you.

A

Yeah, that's what what was what we talked about, because just removing a single set that would be broken. Yeah yeah I either probably all would be broken anyways, so we would just drop all of them, but dropping just a single one would cause fields not to be owned, and I thought that's not good.

C

Okay, yeah, maybe you're right. Maybe we just reset.

A

And and and then we, the question is, can we reset in the validation part, is that something allowed to do, because it's not supposed to.

C

No, but probably not, but you can do it somehow somewhere else. I can.

E

C

You can do it after all, right we have a specific chain or when we say we do, the the mutation web hosts other web hosts, and then you can run another.

E

Function, probably yeah, if I find where this is why, together.

E

I mean yeah, yes, I'm.

C

Yes, that sounds good. I mean eventually, we may want to do something about what the mutation has changed anyway,.

A

And is that a better approach than what I've been trying right now to wrap the admission chain, basically into a reset? If they are invalid and not you fall not depending on validation in general,.

C

A

Right now, my approach was to wrap the admission chain in a let's say, another admission controller that just looks at managed feeds before and after if they were broken after it takes before.

A

Okay and the validation was just hard coded right now, but it should use the same validation as before. Yeah.

F

A

That be better or worse than this new approach, it's kind of it's kind of the same, but also not.

C

I don't see the difference. What's the.

A

Difference um instead of instead of doing validation and resetting after admission, it would use the validation that's already there and then check if match feeds are gone afterwards. I don't know it's. It's kind of mixed puts the pulls this thing apart into two places.

C

Yeah it's it's already like this right, like even when you send the request. We have the validation part, and then we have the passing of the field setback. um If we are I'm assuming, we already have some validation, and so, if it's completely broken, it should already do something.

C

But then it's going to fail. Maybe.

C

Or maybe it's gonna trap some things it's still going to be valid and then uh we're going to think it's better than we're. Gonna keep the thing uh changed, even if we didn't want to.

C

Let's I don't know, let's look at the code again and maybe the tests and see what happens.

A

A

Good yeah, I'm I'm not happy with the time it took, but.

F

A

A

All right uh anything about status, wiping.

C

Not yet I'm probably gonna try and talk to children to understand what uh is is satisfying and start with that I'd like to understand what is the, what would make him happy like the the minimal set of things we can do to make him happy with that.

A

Yeah, that's that's good. I I like, I think, did he already talk to him. I thought we wanted his opinion in general, anyways.

C

Yeah, I talked to him briefly the other day, but he was going to a meeting, and so I need to book some time.

A

All right, everything else has checks on it.

A

D

Done a lot of things.

A

A

um Anything here that we need to address.

C

If we can, uh we can talk about immutability all right in the india in the other document.

A

Yep, the other document is the work items.

C

C

C

I'm going to move that up.

A

I just found it.

A

A

C

um Yeah, do you want to talk about it.

F

um Yeah yeah, I think so so uh do you mind opening up the dog, sorry uh which dog, oh yeah, dog? I have help.

C

The one you shot with me, the other yeah yeah, it's not public yet, but you can probably make it public if you want to. uh You, have to share with the working group kubernetes.

E

C

Let me see- oh, it's already shared with the group okay, so I'm gonna, I'm gonna, put the link in the document. um Thank you.

F

So yeah anyway, I think uh um I had a brief discussion with anton about this. uh I think we identified some uh use cases that we definitely support. uh There's a uh legal question then, if we wanna support, uh like let's say, uh specify if if we wanna allowed uh specify a unspecified after creation, uh basically you can scroll down to the question of the first question. I think. Maybe then there are more content.

F

Here so yeah feel free uh for you to take a look at the dog, uh even though it's not like finalized, but uh any comment of feedback on welcome. So yeah, basically about this question, so we we it's a pretty like a intuitive to see. uh I have a field with f with the value x and then I change it to uh value y. Then it's a uh it's a change.

F

If the field is marked as a immutable, then it's not going to be allowed, but there are some educations to see what if this field is not specified before, uh can we do we allow the specifier after what or all the opposite uh way that if their field fear the field has been specified, uh we allowed to remove it.

F

So yeah, that's uh yeah! If you guys have some, you know um thought on that. uh I'm all ears.

F

So we talked about the idea that we can introduce a mode um name tbd, but let's see if it's a loose mode, then yeah you can set it afterward or offset it afterward if it hasn't been specified before or if it's a streak mode, then this is not allowed.

F

uh But the thing is that I uh from my uh appearance, I'm my person is a bit mostly based on the crd parts. I I don't know much about the building types. uh So um in my own experience uh we kind of like this behavior, we kind of like that you can specify fields uh if it's not specified on the first place. uh So I'll give a comment in there uh the center use case. We have growing too.

F

Basically then this case that if the user don't specify a opinion on some fields on creation controllers can dynamically determine if the initial values are right back. So uh that's why yeah? We probably want to support the loose case at the loose mode, but yeah I'm done with talking. Let me know if you guys have any thoughts on this problem.

A

um Who would define who would define the mode strict or lose.

F

uh My thought is it's up to the user, oh, who is it? Who is responsible to defy the immutability for that field? uh The president should define the mode, but we can always give a default mode, which is the loose mode.

A

So this, the crd author, would say this feels immutable and loose or strict. Okay.

F

A

D

A

If it can can cause any problems, if, like the immutability changes across versions,.

C

uh We thought about that. If you want to talk about that, yeah.

F

um Why don't you go ahead? I I I kind of forgot what the conclusion.

C

So yeah I can talk about, uh does it make change? Does it make sense to change immutability between across the same version like if you, if it's not immutable and then becomes immutable, uh or vice versa?

C

I think that works okay, because even if we have multiple api servers with different understanding of the immutability of the field, I don't think it would necessarily break now what you mentioned across different versions. It's kind of it's kind of words. If one version accepts the field to be modified, but not the yellow version, it means you would have to use a specific.

C

If you use a specific version, then you can change your field and if you use another version, you can't, but it can be an interesting uh way of saying the users can mostly use this version and they can change it. But this controller uses this dll version and can actually change the fields. That's kind of a good way to walk around.

A

Yeah- and I mean yeah and okay,.

C

So you can actually that's interesting because then, if you say only control, only this specific outback can use this version and only this specific card. I can use this version then now you can do permission per field.

A

Yeah that would create a few basic permissions. That would be pretty interesting. I wonder how many versions we end up with because of some person.

C

So that's probably something we should talk about in the camp.

A

F

A

F

Good idea, I I wonder it's not another. Generally generally of you know, following the principles until we change the use of bdpt with the same version.

F

Like lack of breaking change.

C

No, I don't think so because, typically when we change validation on uh on an object across the same version within the same version, if we say this field, um yeah.

D

Yeah you used to be able to have this.

C

Value and can't have the same value, this value anymore. It breaks because when we change the api server, it can't read the objects from the database anymore because they become invalid, um vice versa, and when we have multiple api api server at the same time, sometimes some are going to accept some things that some other api servers are still rejecting and that's not good, because one is going to write something to the database and the other is going to read it and find it invalid, and so that's very broken.

C

But here it's very different because it doesn't matter what we had beef. Well, it doesn't matter what we had before, but um I think it would still work because the api server when it reads the objects from the database- it's not going to see a change at this time, it's mostly when it comes in from the from the user that it's going to fail, and so for a moment yeah. Maybe some api servers are going to reject the request and some are going to accept it, but uh I, I suspect, it's fine.

C

We need to ask jordan what he thinks about it. So we probably need a section about it. I think it's fine, but we need to have a broader discussion about that.

F

A

Yeah it, but it could a lot. It could cause users to make their apis not backwards, compatible. How if they set a field to immutable, it was not immutable for and their client still tried to change it and the clients would have to use the previous version then right or is.

C

That I mean yeah yeah. If a client is trying to change the field, then suddenly it's going to stop failing yeah.

A

But I think that's that's okay and that's something the user has to decide on.

C

Yeah but yeah I mean to be thought about. I don't know I mean somehow. I can see the argument that if you don't want people to change this field, it's good, if suddenly they can't change it anymore, but on the other hand, if you start saying I don't want this field to have these values anymore.

C

We don't accept that today.

A

C

But the problem is also a lot. We studies a lot that objects in the database already have these values and they can't be read from the database anymore, because of that, in this case this wouldn't happen.

C

A

Will also allow go for it, it will also allow a mechanism to for, for. If you want to replace a field or change how it works, you could set the old one to immutable as a read-only, basically and and introduce a new one for a certain time, so it could allow a mechanism to gracefully change, how your fields work, but duplicate.

B

A

A

Sure, I'm just thinking from the from the api expression perspective how that could make designing apis better or worse.

F

A

C

I mean the poem is good. It's going to break controllers, so that's never all super good.

A

D

The controllers.

A

Usually are for series, usually the controls are built by the people that build makes a change to the api in the first place, right yeah, if you're talking about series, if you talk about internal resources- and we add this field, it might break people.

C

Yeah yeah, we need to think about it. I am yeah here make sure you add a section about that and we'll uh look at each individual use case. Yeah.

F

C

F

So basically, it's about the versions right, changing.

D

A

Should we, when there is a cap or something should we add a similar goal, that we find a place inside kubernetes that can leverage that already and do it or do we have that like field selectors and demon sets or something that's already already immutable.

A

C

um Yes, we could do that. I don't know if um I don't know if it's going to be easy, because in lots of places uh the immutable semantic is like, if I I think I'm thinking about top template, but template is mostly immutable.

C

Yeah, no sorry the pad, the part is mostly immutable, but not all of it and how to say all of these fields are immutable, but not these specific fields.

A

Oh yeah, because you would have specified everywhere, um one.

D

A

I I I always run into, and it it's one of the immutable parts that actually actually kind of annoys me. Is the demon set label selector or deployment label selector? That could be a small example. Yeah.

C

I'm sure the example, if you can look for them.

C

That's a good point.

A

F

Cool yeah, I guess another interesting topic uh come up with uh with the docket and so uh how? How should? How should we define the behavior about map? um Basically, uh we can use simple use cases we can see the the map itself is immutable. That means you cannot uh educate the monkeys or change the value for existing case. uh But uh do you also want a like us lose mode? Something like that?

F

We don't allow the changing kids for existing, so change value for existing case, but we allow users to add or remove case so yeah, that's also um a question, but uh I I don't quite follow some good use cases like that uh in my with my playing with the crds, but if you guys can think about something that is definitely require that the behavior uh uh yeah please come on talk uh that'd, be good argument to support this mode.

C

Yes, I think if you go through as you go through some examples that we have to document.

D

C

It currently works in kubernetes. You might find some interesting answers to these questions. Like do we do we have maps where we never allow anyone to add or remove an items. uh Do we have structs where we never allow anyone to add or remove an item? We need to look at built-ins and what does it mean to remove an item for a built-in it's slightly different because it's uncerealized into a strike.

C

um All of these are, if you look for examples, I think it's going to help and how do we detect immutability currently for buildings is going to be interesting. I'm curious about that and do you know something.

F

Yeah, that's a good suggestion. I I just want uh if to some dog that I I can read upon and find the answer or it's basically, you need to try it out with some building fields, to figure out.

C

No, no, no, let me look it up. uh There is one the the validation for kubernetes is written in specific places, so just go through the code that that's the validation, uh especially the updates right, because creation doesn't matter, but for if you look at the validation good for updates you're going to see uh some patterns.

F

Okay, I'm good.

A

Maybe you can also look at the storage strategies, because that's what I where I remember, I think some pod ip stuff that only allows one ip in there and if you try to add more, they get removed again stuff. Like that. um I think I I I could come up with a case on on in a series. I'm I'm working with where network interfaces can't be removed, I think um but added, but I'm not sure.

F

A

You think that's.

F

Allowed to add, but not to move okay.

A

F

A

um But the maps I uh should the list type or apply to this like if it's an atomic year, the whole map gets immutable, including its its contents or and if not, you can modify the map like add and remove, but.

F

uh We decided if it's a time to come up, if you yes go down to the top, there's a section talking about that, but for the atomic list. Basically, the list itself is not allowed. um It's not allowed to change at all, uh and it's since, since incumbent, we don't have the concept of a list um like the order list. So atomic list itself is just the immutable. You cannot change anything on it, but for set uh you, you is the same thing uh except it will ignore orders uh yeah.

F

You can change the order, though the in the third, but the set itself is not immutable. When do it. When do we compare.

A

What I meant was um like uh the pods, the pots- uh oh no containers in a pod like uh making the containers immutable. Would it mean that all the containers in it are also mutable or could? Could it mean that, depending on the list map type that place type, I don't remember.

F

Yeah it will determine either up to the type. uh If it's uh I think uh the map is called associated list uh yeah, but yeah. If you associate this, then uh it will have the map behavior. That means you can uh basically I'm amper present uh you. uh You can change the values in it, uh but if it's a it's a just set or atomic list, then the release itself. Sorry itself is nothing. um It's not mutable. If you mark the field is immutable.

F

Map is a little bit interesting as a basically object and you handle it differently.

F

A

I found something super interesting for you.

C

Oh great yeah, I will look through it yeah. There is.

F

C

Named validate immutable field already in the api.

C

If you look at the third link.

F

Sure I'm checking now.

A

um Would that mean that um you cannot change any fields anymore? That are key fields if well so.

C

Yeah we have to look at. We have to look at what this means and what, how it's used.

F

Yeah, so uh in my kind of proposal it basically works like that, if it's a map, it's a social list and you have the ability to define the invisibility for non-key fields if yeah, if a key nonkey field, you said it's immutable, then it's not. You are not allowed to change the value for existing key uh but yeah. If you mark the field as a mutable equals force, then you can do whatever you want and then how to define the uh adding key and removing key into that map is a separate topic.

F

So it might also, if you want to have the like the loose as a strict mode, then it's going to be the same um philosophy uh in the strict mode. um You don't you cannot do that in the loose mode yeah you can, you can add in new keys or removing all the keys, but yeah again. I think I agree with you.

F

Let's start with some examples and uh try to learn more, uh the behaviors, all the existing patterns that we have the community have, and that might be a good uh guys for us to pursue this talk.

F

Yeah so feel free to come to any more uh samples that you are aware of. uh That would be super helpful.

A

Yeah I I last question I have because I'm not sure if I'm understanding it correctly, it does and if yes, how far does immutability traverse down the tree?

A

If you, if you put a, I don't know if you said spec is immutable. What else becomes immutable because of that.

F

A

F

Should be progressively yeah, if you mark the top down at the top field immutable, uh then yeah. Everything already cannot be changed, but if you don't mark the top field, then you need to go down the next level and determine the behavior from there.

A

A

Somebody else have any, I think, to our questions.

A

A

All right, thanks for working on this uh very interesting topic, I haven't thought much about immutability before.

F

Yeah glad to have thanks, I think it's important.

A

Good, then, I think somebody else has something to add we're through for today.

A

Then I'll stop the recording.

C

Everyone see you in two weeks um after feature phrase.