Kubernetes API Machinery Special Interest Group, 23 Feb 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG API Machinery 20220223

Description

Items for today's meeting

1) remove audit.k8s.io/v1[alpha|beta]1 versions: https://github.com/kubernetes/kubernetes/pull/108092
2) add StartsWith and EndsWith operators to label selectors. Want to get feedback and see if the community would like to move forward
https://github.com/kubernetes/kubernetes/pull/107972
https://kubernetes.slack.com/archives/C0EG7JC6T/p1644383070192189

A

Recording- and we are on um good morning good afternoon, hello, everybody- this is the api machinery uh today is the 23rd of february 2022 and we are having our bi-weekly meeting. People is slowly joining. uh Thank you for that. So we are going to jump into the agenda. We have two items today. Maybe we can keep it short.

A

um Let's start with the first one. I think um we found this one during uh one of the three ashes weekly triashes and david thought. It was a good idea to at least bring it up in this meeting and to make everybody available and have some discussion. So I don't know david. Do you want to take it over.

B

A

B

So uh if you recall, there's a capability for to register audit web hook back ends and to interact with those you have to write a serialization format.

B

We have had an alpha, a beta and a v1 version of this webhook contract, where the qbpi server would serialize into one of those versions and send it. uh V1 has been available since kubernetes one two.

A

B

That's not like a typo, that's uh kubernetes version. One two had the v1 api and this pr removes the old serializations. I am comfortable doing that. I think it's been long enough. There was an issue that was actually linked from there where, after marking it deprecated, we waited until version 124, but I want to make sure that you're on the same page daniel, so that when the pitchforks come for us, we're all holding hands.

C

So, to be clear, these are uh this is a on the wire serialization format. It's not something that we store in ncd.

B

Correct it is on the wire, and that means that, if a web hook uh consuming the auto log has not been updated since cube one two which is cube one one, I guess it may not accept the new format and it will start to fail.

B

I think that one two was what three years ago.

C

Just a small amount of time are you sure it was only three years ago, one.

D

Two that's like six years ago, dude yeah.

C

I think I think that's more than three.

B

A

It's been a while.

B

uh So I feel like it's a fairly defensible choice, but uh I felt it was worth bringing up here before I approved the pr.

C

So um I I don't see how anyone can argue for keeping alpha after six years, um when we never promised that we would keep alpha working at all like basically ever for anything. uh So I think the only debatable thing is uh whether it's okay to remove beta, also.

D

um Gated, like thinking about like, if it breaks and we've removed it, do you just roll back versus like? Is this like feature flaggable just like give it a one release and be like here's the flag? It's awesome so.

B

A year ago, we actually went through the trouble of of changing this so that it was specifically listed as deprecated.

D

Right, I just I'm just thinking about the principle of least surprise here I mean it isn't gonna be maximal. Surprise. I I kind of agree of all of the apis. It's just I mean what's different about removing this from any removing any other beta api is we're just going to yank it in one release.

D

C

I think the I think the relevant difference is this is not stored in std, so we're not going to corrupt people's or make people's uh databases unreadable um can.

E

You can you confirm.

C

David is this is: is this something where you tell api server, I guess by a config file, which version uh it should be sending to this web hook or is.

B

This something you pass it as a flash.

C

B

And we've been printing out a warning that says the version you're using is deprecated. If you chose anything except v1. Oh, we have a warning. Well then they deserve what they get. Okay, excellent. I.

C

Mean but who who who looks at the api server log file when they start up api, certainly.

B

I mean I sort of agree, but it has been many years well if, when.

D

We remove this if the flag just starts failing the api server wouldn't start so you know to roll back. So could we do that.

B

um I can check, I believe, jordan specifically requested that we make it so the flag still exists, but only accepts v1, because that's.

A

What the previous.

B

Configuration would do right, they would set v1, they would now. The previous configuration really said we want. Would they.

D

Have to set v1 alpha 1 to be.

B

Using this yeah, if they wanted to set v1 alpha 1, they would have to accept that explicitly. Okay, I mean that.

D

That certainly seems to meet the bar of their. The old value will stop being accepted. That binary won't accept it. They won't be able to restart you'll only have one api server down if you're doing a rolling restart, and you should have known better it'd be like it. That seems like a very reasonable this value, deprecated, etc. Here's.

C

Here's the way I would want it to work if this was something that was super important um and hadn't had six years of lead time. uh I I would, I would want it to work. I think it would be fine to just delete the v1 alpha 1 stuff.

C

I would keep the v1 beta 1 stuff, but I would make the flag like fail to start api server, if you just say v1 beta 1., the only way to make it start and also use the v1 beta 1 format is to change the flag to v1 beta1, and I understand that this is going away next release right some string indicating that they know that they have to update.

C

um I don't really think it's necessary.

D

It's weird because, if the, if like we took away v1 beta 1 and v1 alpha 1- and this has been a deprecation notice for your it's- no real different from you upgrading to a version that removes crds, which is except we've got a flag, you won't even be able to start the new binary.

D

And so your alternative is you just can't do the upgrade first yet and since you upgrade the api server first, like it's a pretty clear place where it's you've had a year's notice of the beta deprecation, and so you just like, I hope you weren't planning on doing a minor version upgrade today, um but nothing broke and you just roll back to the previous binary.

D

I mean that's not terribly different than if you were using a v1 or if you wanted one version, and you had it specifically enabled by a feature gate, the featured it stops working when you can't enable it anymore.

C

Yeah, I think it's reasonable to just remove this stuff, and if people especially people, have had a log message for a year.

B

Yeah, all right, uh I will go ahead and review and if it looks good I'll, go ahead and merge again.

A

Do you want me to put anything like assign this one to you? I think it's not a sign sure you can go ahead and assign it to me and I'll have a look. Okay. Thank you very good um cameron. Are you fine if we start with your topic now.

F

Yeah, let's go okay.

F

um uh Cool, so I suppose I'll introduce myself um my name's cameron and uh I um I've been wanting to introduce, uh starts with an endsmith operators to field selectors. um So I understand that in I think it was november. um Someone else tried to propose adding regex support um to field selectors um and that made sense rejected for performance concerns um and uh but I've well.

F

I've yeah added um starts with the next operators, uh because the the time complexity is linear, so um the performance can like concerns, shouldn't really be as big of a problem.

F

So I was wondering what the feedback is from everyone about adding those.

C

David, do you want to go first, I'm still making up my mind about this.

B

uh I'll go first I'll I'll bring up the the previous discussion as well. So uh the previous discussion there was concern about the time complexity of a regex.

B

At the time I recall suggesting that they could build a um a benchmark test that showed that they were able to somehow constrain a regex to avoid a a runaway or having too great a time complexity, but I predicted that wouldn't be possible.

B

I think that starts with an ends with resolves that time. Complexity, problem.

A

B

Think it adds significant value to the way I've seen people use, field selectors or the way that they want to select something based on a name or selecting in name spaces. That start with a particular prefix. I see this pretty frequently um I I would not accept the change without a cap. I think it's a worthwhile thing to sit and describe and say I want to create these new operators, and this is how we would vet additional operators in the future, but I don't think the concerns about time.

B

Complexity, stand and I do see value in these particular operators.

C

Yeah, it looks like I wasn't here for the for the regex discussion.

C

D

I had to comment daniel if you need a minute to gather your thoughts.

C

D

um So I saw this comment um in the previous one, which is the original justification for label selectors, as they were, were, was a concern about being able to build indexes indices after the fact to accelerate this.

D

We have an index in the watch cache and I'm a little unhappy about it for other reasons, um because it's not general uh on field the field selector side- I don't know that ends with and starts with starts with, is an interesting one ends with concerning me a little bit, but neither one of those are particularly problematic for indices, so that, but we've never actually documented the justification for the scope of field or label selector, and so at least in a cap.

D

I would expect to go in I'd, want to make some normative statements about like this is the kind of constraint that we have to do to support this also label selectors are unversioned except as part of the meta api version, they're in and so there's a there's, a door that I'm a little concerned about with unsupported operations um and knowing that they're going to be supported. So that's a it doesn't. I don't think that should block people accomplishing and solving real problems, we've been pretty pragmatic with label and field selectors in the past.

D

We just have a lot of investment into them in the last seven years, and so I'm at least on this, like I think there are things that we could. We should be doing to improve people's ability to select resources in cube, because you have a lot of valuable resources in cube.

C

So uh maybe I'll just start talking like because I I think I agree with a lot of that. um uh I'm I'm concerned about how uh we're organically growing these things, uh and we don't really have a theory of how they're supposed to work like where we're trying to get to that doesn't tend to get people to places where they want to end up that strategy.

C

I did glance at this, I'm uh I'm not a huge fan of generating like like using inventing a new operator every time we want to do something additional with uh label. Selectors like you've, got star equal and equal star. I I don't. Maybe those are common in other languages or other contexts, but I'm not familiar with them, and I could imagine people not expecting them to do the thing that uh that that you want to make them do um so. uh Those are slightly concerning.

C

Another thing that I that occurred to me is um uh as bad as this problem is with label selectors. The problem with field selectors is even worse because we basically just froze that because we didn't know what to do with it, um and I know uh um I've got a couple people I think maybe on this call who have been investigating uh now, I don't think I see them on this call.

C

Who've been investigating uh adding a cell syntax for field selectors, um so.

E

C

I'm here I can talk about that yeah. Why don't you just say something about what the plan that you're working on for field selectors is uh not that it's set.

A

In stone or anything.

C

But we should, we should try and make them somewhat similar.

E

Yeah, so the quick background is, um as part of adding improved validation for crds we're using a very constrained type, safe expression, language that understands kubernetes types really well, and so you can do things like the reference into a field from you know higher up in the object and do all sorts of things. It seems like a really good primitive to use for something like field selection, because you can basically write a condition on any kubernetes object. Like does this field? Have this value?

E

You could do any functions that we choose to support so right now we have a pretty constrained set, so we can control running time, but um you know, if you want to like say I only want objects back where this field has this value or where you know some other particular condition is met.

E

It would do that really easily and it would do it both for crds and native types, which kind of kills two birds with one stone, because right now we don't really have much in the way of field selection for cds at all and part of the problem is just mechanically doing that, but, like cell, we already integrated with crds.

E

So we could actually have that up and going pretty quickly and then on the kubernetes side with the native types um you can do field selection, but you can only do certain fields and it's very concerning what you can do and because there's no typing like what is what's a start with with the number mean versus a string versus a date, um it gets really confusing. But with cell. That's all really clear and you've got a grammar supporting you and backing you, and it's all documented.

C

D

C

D

Was gonna, I I very much. I have been unhappy for seven years that field selectors have been as they are and I'm glad about the sell stuff, and originally we realized that label selectors were insufficient to solve the problems that people would have selected. Specifically spec note name. We created field selectors to solve changing the storage model from storing all the pods that were on a node in a separate object to letting you issue a query that says, give me all the pods that are assigned to a node.

D

That's the only reason we have field selectors in cube. In the meantime, we've had to go, do terribly horrific things like we use label selectors and we add a metadata name field that we populate and store because that's the best we can do, and so there's.

A

A little bit of.

D

I I feel like with the cell stuff coming. There are some angles there, which is it's a good opportunity for us to concretely say what problems we expect field selector to solve what problems we expect label selector to solve and accept the set of use cases for label selector, that we don't support and identify them as being in or out of scope and give a clear justification. Why.

A

D

C

To say um like originally that, like uh kubernetes api was not supposed to be a archive retrieval database system that supported arbitrary queries right, the label, selectors and field selectors are to support targeted things that the system itself needed to do, and the theory uh back in the day was that when you wanted to do things like write reports on who used how much cpu core or whatever like that, you would not be running those queries against the kubernetes api you'd be running them against an archival system that is recording that information and putting it into an easy to search format.

C

So um now I can definitely see the argument that we have drifted from that and nobody's ever actually written that archival system in maybe seven or eight years or whatever it is uh after we started this. We should admit to ourselves that nobody's going to write a system like that. um That's.

B

D

Too, I'm not building an archival system. People already have actually of turning cube into sql, so there's certainly elements of that. It's just is that solving a does that solve a day-to-day user problem. I I think, like a point, that I really like what you're saying daniel, because we are accountable to the users of our system.

D

uh We had a theory. We haven't reassessed the theory in a way that you know took those user concerns into account.

C

D

C

We didn't know what we were doing when we put this. In I mean maybe we still don't but uh yeah.

B

Field selectors, as I recall, the problem that we had with them was that with different serialization types right, a v1, a v1 beta, 1, uh v1 beta 3 back in the day right, we actually moved the fields around right. So what used to be at spec dot service account name moves to spec.service account right, and so the field selector problem that we had was not actually related to the set of operations that we could perform. It was related to how do we select the fields to feed those operations.

D

It's the yeah name, mapping and honestly, that's still a problem. If we were to expand the label selector syntax, we would now have that problem on label selectors as well, because you also need to identify the version of the label selector language that you're using um there is an angle here which is at some like we already have mechanisms today, like coincidentally, we now have mechanisms that allow you to say how you wish your api objects to be interpreted.

D

We don't necessarily leverage it, but you can, for instance, tell the server that you want your queries uh executed in a specific version, and so some of that is like we could solve that, but you're right we would have to probably address that to grow field.

B

Selectors in order to grow them with the with the notion of I want to select different fields, the more narrow scoped, and that would be what cell is trying to do right, which I agree is cool. I wish I had that on crds or on custom resources rather, but in terms of given the fields I can already select, I want to match them with this different operator. I think I think that falls into a different problem.

B

Domain cell attacks both but it's more narrow to say I want to add a starts with than it is to say I want to have a way to select a field across different versions and serializations yeah.

C

I agree: that's a more narrow problem, but I agree it's a more narrow problem. The question I ask is: is it?

C

Is it uh a problem that enough people have like, for example, um when you want to do a prefix or suffix match? I get really worried about uh prefixes that are prefixes of other things um and uh it seems like it would be much safer to just double label your objects, one with the prefix and one with the whole thing, and then you can search on the prefix and it's safer because you're, you don't have any chance of accidentally searching for a prefix that happens to be a prefix of another prefix and identifying objects.

C

You didn't expect right. It's there's a there's, an easy way to misuse. This is what I'm saying.

B

So I did not look to see whether it was whether the actual change itself only changed label selectors, but if this also affects field selectors, which I did expect it to, um there is no convenient way to name your object twice. So you know selecting things with a prefix on the name is pretty convenient. It's a thing that you know a user can't fight with you over what that label should be changed to uh and the name itself is fully immutable, but you can't name two of them.

C

Yeah, that's that's a reasonable uh response and I think it would affect both. um I I guess the way to address my concern is to somehow make it safer. It's like have a standard way of dividing the things that you can't match. Less less a smaller part of it than you expect.

D

Maybe so daniel, I think one of the questions I'd ask too, is: what are we optimizing for? We optimizing for our limited time being used to address this one specific problem. Are we optimizing for a particular class of users, like, I think, that's the like david? I heard you almost advocate for the minimal thing that unblocks a very reasonable practical thing. I think daniel you're kind of advocating a little bit for the we.

D

We we're a little unclear about what that next step is, and then I kind of have the like I'll put my top level hat on, which is at the end of the day. Like do we actually care about the philosophy of this, and is this a very important philosophical thing once we add an operator, it's very hard to take it away, and so we have a conservatism to expanding api service area that tends to lead us towards delaying practical improvements.

D

If we could imagine that we would actually deliver a better improvement sometime in the near term, but it often leads us to push it off for a very long time, as we've done with some aspects of field selector. And how do we weigh?

D

How do we weigh use cases like? Are we applying a consistent principle for the data in the cube? Api server is accessible to a set of clients with the most flexibility. Are we weighing towards the minimal flexibility or we weighing towards whatever comes up on our backlog?.

C

Yeah, I I agree with that. I think the what we should worry about at this meeting is: um can we set forth an algorithm that says yes or no to an infinite number of feature? Requests of this nature right.

B

I'll volunteer, something that would get me to say. No, um the idea of using ceo for field selectors is compelling. I think it has a lot of challenges if it were impossible to express this operation in ceo. I would be against accepting it, because I don't want to constrain that future choice.

B

I believe this particular one is expressible in ceo, so that future state is not prevented.

D

And just so, I make sure that I catch a subtle thing there. If you have better field selectors, you can do arbitrary things with labels with field selectors.

D

That is a interesting statement that previously, I feel like we've actually said well like we could do this, like you should do this with label selectors instead of field selectors, because field selectors are limited. That does change. Some of that underlying philosophy which is field selector is the generic tool label. Selector is the very specific tool.

D

If you have the generic tool, you do not need it. If it, as david notes, is expressible.

C

Yeah there was um at one point: it was thought to be an important property of the label, selectors that it was reversible um you could select, which which selectors would match a particular uh string, um but I don't know that we make use of that property anywhere. uh So if it's not an important property, we.

D

Need to preserve.

C

D

A little bit like there's some assumptions around how the watch cache handles it. I think, and that's as far as we've ever gone, I.

C

Mean honestly, the watch cache can handle can index anything because it examines all objects. uh It can just run the selector on every object. We.

D

Did it was we have accidentally leveraged that property? It was not intentional and I don't even think it's a real blocker. Maybe it's a better way to say it and honestly, like so there's, there's a there's, a philosophy here, and this gets into like some of the other stuff that folks are talking about, including people that I've poked and said scurrying in various directions, which is at some point.

D

Are we assuming a watch cache? Are we assuming every bit of data under our kcp aps or cube api servers in memory? Will we be locked into the storage info engine forever? I don't wanna, I'm not coupling those to this discussion, but general field. Selectors are a useful thing in many different domains.

C

D

Selectors being limited has advantages.

C

Yeah there's, I don't think, we've mentioned it yet, but one thing I am worried about is: if we make it easy and convenient to do this, then people will be uh much more happy to do it uh and uh using one of these things is actually super expensive. It's a linear scan over everything unless we come up with a fancy indexing system uh right, but if you can describe basically an infinite number of things in here, uh we'd only be able to like at best index the commonly used ones.

D

Or we simply require crds and resources to define an index, and then you now have an operational problem, which is every time you install a crv that needs a new access pattern, you're defining its access pattern as what you index and now we've gotten to the point where we are a really crappy db.

C

Yeah is that being nice to our users, I don't think that's being nice to our users to.

D

Be fair. uh The only reason that the watch cache really exists today is to support pod spec, node, name selection um and because the historic version of ncd needed it and because we pushed ourselves up against the thresholds with very large clusters in 510k, where we're we've pushed ourselves into a spot to support that breadth that we need to watch cash. For other reasons, we did not start from the principle that we were going to necessarily you know only optimize for that one use case it's just that was what mattered for our users.

D

There isn't a scenario where we need to at least consider um what makes pods special and is there a camp like maybe david or daniel? I don't know like. Has anyone done a canvas recently of use of field selectors in the ecosystem, like something that would point us towards how broadly this is used?

D

I certainly know in a lot of controllers. I advise people to just select everything and let the chips fall. Where may that works for 85 90 of problems, I haven't seen an extensive use of field selection or label selection, except in the scheduler, and in some of the more demanding use cases that do hacks to work around this, where even the wind from not serializing. Everything is enough.

B

Yeah so field selectors- uh I because they are so special case only name and name space are things people commonly use. um You know, have I actually counted them all? No. I do know that we have many operators that make use of being able to field select on both name and space, where we know what we're looking for.

C

Yeah and this list of lists might be implemented as field selector of on namespace. If I recall, I have some vague record collection, I.

B

Don't recall whether it is or isn't, but it's like it sounds strange, but no.

D

All all all watches and lists inside of namespace scope are technically implemented as a as a as a field. Selector.

C

Yeah, jordan and I had a disagreement over this and turns out. He was right, but I was the one that made it work that way so.

D

I think I think I approved part of that and we're like well. This seems like a time yeah another wrinkle here, and I don't think this blocks it, but if we were to choose to go with a more general field, selector approach that removes the ability to use some of these from controllers that only support label selector as their filtering criteria, so network policy, some of the things tim's been chasing recently label selector, is deliberately limited, which has value. It forces you to express your problem in exactly what domain labels.

D

There is an argument, I think that's kind of the opposite of what david just made, which is a thing that would potentially push me towards the more general thing on label. Selectors could be.

D

We actually have use cases from network policy or something that's pretty much like nobody's asking for that today. But if someone was asking for that, that would be a data thing.

C

Okay, so I think I thought of the the like general dilemma uh so uh suppose that this feature is super popular and will get used a lot. That means it's probably a scalability problem. It will increase the cost of running a control plane, not because not because it's particularly cpu intensive to run, but because the it incentivizes people to do more linear scans of everything.

C

um So that's one fork of the dilemma. The other fork is if it's not popular and won't trigger that many calls, then maybe that is not that many people that will want to use it and we don't need it. um So uh uh do you want to argue with that? Well,.

D

The argument there is, if field selector is even more inefficient and there's use of field selector. That then gives us the input to come back and do it if this problem can be solved with field selector doing field selector more generally, is there an argument that is being proposed right now for self or feed selection? Where we would say this is just a terrible idea and we shouldn't do it. Does anyone have that argument? Yeah.

B

So I might, but I I didn't want to sidetrack this one uh I have reservations about. I haven't. I haven't read this proposal. I didn't know it was here.

A

B

But I have reservations about how it handles the different serialization of various formats. um What that means for custom resources that go through conversion web hooks. What.

D

It what it means for efficiency of out-of-box things that have two versions that are very expensive and heavy, and the fact that we actually have to convert everything internally to an external version to do some aspects of this. And we have this authorization cache. And we have the watch cache on internal versions. And so that does.

B

Open up some pandora's boxes, yeah.

D

B

I also have reservations about the set of functionality that that cell allows, while it is more bounded than some, I seem to recall that it had regex matching. For instance, it.

C

Does have regex matching and I I uh maybe I need to go, watch the recording, but I'm not sure if regex is really expensive enough. I thought.

D

Here and go anyway, I thought that was the whole point of yeah it's expensive, but it's no more expensive than sell.

B

D

B

B

I I have those those concerns and reservations about it. They don't have none of those applied to this operator, because this operator has a fixed run time. It doesn't change the way, the actual fields that you're comparing are selected- and I have wanted to do this in the past right, like we, we make use of patterns like this, so.

D

David, I would say, I think, you're.

D

uh It is possible to craft a a expensive query, and so the argument would be that it's better to have a pattern that implicitly endorses or is accepting of expensive queries, which is a little bit more feed selectiveness, so general purpose and label selector has problems another way, though to say it which would be if we were to add these to label selector.

D

I would probably ask for an approach that allows us to version them effectively, which is the same thing that you're worried about with field selector, but a smaller problem, and that's not trying to block it or to say it, but it's the would we have to do alpha gates for these query. Selectors, yes, would we have to think about what happens if we decide to turn them off yeah, so they're places where these are stored in the data engine? Would we allow these in the validation mechanisms for workload controllers that select on label selectors?

D

Yes, how does that so? There's a bunch of problems that come out of a change to this schema that I think are similar between any change to field selector. In this.

B

D

B

See a difference with field selector because it is trying to solve a serialization problem of how do I get the values to compare when I look at label selector? I think this is very similar to when we added the ability to do an in to label selection when.

D

B

Was that one one one.

D

C

Was a really long time ago, a while.

B

Back uh but when you go to consider a change like that, if you roll it out uh over the course of you know well minimally, it takes three releases, but uh if you roll it out over a number of releases, then you're guaranteed that the any server that allows you to set it will have the n minus one version also be able to accept it, uh and so you can effectively handle the upgrade case.

D

Will that work for crds.

B

And people who haven't updated their crd code- yes right, because this is not about how does it get the values this is about once you retrieve these values, which we already know how to retrieve and are not changing the labels, we compare them label queries are serialized into objects in our storage engine right, and so, if you end up in a spot where you have an n minus one, server that can honor the the actual label select query, and then you do not allow it to be set as part of the label selector field, a serialized label, selector field until n, then you are guaranteed that, for whatever level of skew anything that accepts it, even during an upgrade, every server will honor the selector.

C

Yeah, I think you're talking tactics, though, uh which I've been trying to avoid uh you're right. Yes, if we, if.

B

We decide that this is a valuable thing to do. Then we need to have a different gating mechanism on it. Do we need to have different versioning mechanisms on it, and I can look at this and I can say I think our existing versioning mechanisms support what we are doing right and here.

C

uh Yeah, I I I agree with that and I don't think the there's two versioning problems. One is the the syntax by which you express the query and the other is the uh version of the data or the schema of the data against you, which you wish to run. The query.

C

Maybe you could solve them with the same mechanism, but it's not totally obvious to me that you should. I might be actually extra confusing to combine those two versioning problems into one.

D

I only I only wanted to poke at that a little bit uh david, just because I I think it's you know, you're arguing the um what's: what's one more wafer thin operator added to label selector to support a reasonable thing that has parallels in other ecosystems, which is funny because I'd usually be the one advocating for the wafer thin thing and you'd be the one being like. I just don't think this matches the principles of the things. So um I appreciate the the role reversal.

D

uh One of the interesting things here is: is this the last two operators.

D

In was pre in was like a a big one starts with an ends with a relatively common. Is it end here or do we add something in the future.

C

Yeah, that seems like a good question.

E

uh The lowest barrier, one I saw that was probably missing, is I think if you do field selectors, you can do more than one but they're conjunctive, so they're only an and operated hater. You could imagine easily somebody being like. I want this and not this and that's.

D

The philosophy I've actually run into that in practice trying to craft things where it was. You know I can't even express that with a label selector query without writing code to do it twice and then merging my original.

C

My memory is perhaps a little fuzzy, but I if I recall correctly, which I may not, uh that was a deliberate choice, because you, you can't invert uh that query uh without solving an mp complete problem or something like that.

D

That was, that was the reasoning I remember as well.

C

Yeah, uh so so that lack of conjunction or a disjunction is deliberate.

C

But yeah I actually agree that people would really like that and if we don't ever actually invert these things, then maybe maybe we should consider.

D

I certainly feel that things that we agreed on seven years ago that have not practically been a concern. There is a chesterson's fence kind, but I do think that this group is accountable to at least document whether we can like there's always like. Don't do things until you need them, but that can be taken too far, and so I do think with the inverted index like if there was a cap for this, we should specifically attack that specifically set up as a straw man and get enough feedback that we can say.

D

This is not something that we believe is necessary at this point. For these reasons, and those reasons be reasonably principled, I think we should we can use that as an opportunity to either endorse or articulate that, because I think that's one of the concerns was the inverted index on ends with uh gets up.

C

Yeah, so I think.

C

Let's see just things that would be relevant for my own thoughts on this is: is this the last two we add or uh are we gonna add more?

C

um Are we going to move ahead with something like cell for field selectors um and yet how how many people will use this, and what is the scalability effect of encouraging that many additional linear scans.

E

Yeah, I'm really curious on the appetite for the cell one because I mean what we do is what we have now is like a sketch of an idea, but you know my anticipation is right.

E

Now is that most people doing advanced things just get everything into do whatever filtering they want on the client, um and I don't I don't know- maybe that's fine, maybe that's most people just continue doing that anyways and there wouldn't be much value- add for sell um because there's a ton of other stuff we actually have has higher priority for so right now. But if there was interest in this in some shape or form, even if it was really constrained, I'd be really interested in knowing what that is.

B

So daniel I I don't know that I'm fully convinced that you would say: oh and now I suddenly have an even larger storm of these requests. I think it would be more likely.

B

Oh, I had a bunch of requests that used to scan everything, and now they say I want this thing with a prefix, so my load got reduced right, like I think about our usage, where we go through and a lot of places that just list everything and then they filter out and say I want these prefix ones and instead they would come in and say I want the namespaces with. I want the contents of pods and namespaces prefixed by the whatever.

D

Yes, realization you're still you're in the serialization cost.

C

Yeah you, you still pay half the cost, the the read half, but you omit the right half, which can be significant.

B

um Especially when you're looking at a narrow window of these namespaces.

D

But you were probably already doing the whole list today like there's. Probably I I think is there an argument here that there's any scenario where you wouldn't be doing the full list. I guess you could be like instead of a full list, you'd be doing 10 partial lists as you're concerned daniel. That's.

B

Actually, that's that's actually. What is functionally done today, right where we've done it in a couple operators that we wrote where they go list all the name. Spaces start all 100 watches in order to watch all the contents of namespaces that are prefixed by whatever name so.

C

So I don't think I've completely com communicated my uh my theory of how this makes our life lives hard.

C

So uh I I believe, um when you add a lane to a freeway, it doesn't reduce the average travel time over that freeway, because instead, what happens? Is it incentivizes, a bunch of extra people to drive on that freeway and you end up at the same speed just with more cars going through. So my theory is by adding additional capabilities we incentivize additional use. uh So I'm I'm hypothesizing that we're not replacing additional or existing list we're incentivizing people to add new lists, because we enabled new features that they couldn't do before.

D

People in the neighborhoods other cars are driving through they're, not driving through those neighborhoods anymore, so the people in the neighborhoods are like man. This is really great. My kids can go play in the street um and I think that's the that's my counter to that particular now.

C

Yeah so the the question is: do we want to incentivize people's kids to play in the street here.

B

Nice uh clearly, kids go play in the street, um so so what I'm saying is that, like I've, I've seen this problem on the ground and the way that it's solved is either with somebody deciding yeah list everything in that filter or with someone saying well I'll, do a pre-query and then a list watch on that to see if it ever changes and then I'll establish all the distinct lists downstream of it and all of the distinct watches downstream of it and then restart those on any change.

B

Right and- and you know we'll, perhaps perhaps people were just satisfied with- I can't possibly solve this problem so I'll do nothing, but it seems more likely to me that people solved their problem at greater expense to an api server. Today.

D

That's really interesting david, because that then triggers the next question for me is: are we already building the label selector watch or index today? I actually don't know how much we index of it. I haven't looked at it.

B

Are you talking about the index and the watch cache? It was a special case for like two items: okay,.

D

So the problem is, is that I have an access pattern that cube does not efficiently solve and there are two ways to solve that, which is you either guess and you do magic or you force someone to tell you about their access pattern, and then you decide whether to accept it or not. So the access pattern for a number of these examples is, I want to make a bunch of destroy watches, and I want to describe the thing that I wanted that these are. These are coming.

D

There's very few people, like we don't dynamically, add pods to a cube cluster like the cubelets are just gonna start, hammering the api server for spec.node name, no matter what I think these are not like, like no one's coming and going and adding these like custom label selectors in a meaningful way to run ten thousand jobs, they're building controllers, which run for a period of time where the access pattern is a part of the controller design, and so the approach like as mentioned before is, if you need an index.

D

There is a step there which, at some part of me, is like we should probably be generalizing. What's in watch cash to be more effective in general, because it's actually, you know that and that'd be a worthwhile investment for some sets of people like steve, I know, would probably be happy to go. Help generalize watch cash um for various reasons, because I'm forcing him to, but that could be a that could be an approach, that's not unreasonable, and then we could say. Oh pods are just a special case of that. Oh.

C

You know at the point at which we're we're telling people uh yeah pre-declare your indexes, um that's going to make it really hard to continue to claim that we're not a database.

D

I mean people treat scd as a store of record. I mean whether that's an effective database or not and you're, going to have some sort in get ops, and you have some stuff to store in your cloud control planes that you sync I mean. Maybe the answer there is like you know, then, if these are the use cases that people need and we can accomplish them for a reasonable effort and there's people willing to do that effort, which is someone has to sign up to do it.

D

That could potentially be a way out of this particular hole, which is you can make these indices efficient? If you have to.

C

All right, I I think you all have mostly convinced me that I don't. I shouldn't worry about uh incentivizing additional use.

D

I I do worry that this is a this would be if this was 25 of users were like. I freaking hate writing controllers, because I can't subdivide on label selectors effectively. This would be a clearer one. This is one of those issues. That's right at the border of there's, probably a bunch of nice controller efficiencies.

D

Even if we introduced this, a small subset of people are probably going to use it and then it'll take a couple of years to reach some saturation where we realize how far we've gone, which is a little bit of an argument for there may be just to hold your nose and add the starts where it ends with and then see how.

C

I I think I I think my the the threat vector I have in mind is actually not controllers.

C

It's uis and configuration utilities uh where somebody wants to reconcile a list of things that they believe that their cluster should be running with the things that are actually running in their cluster and they fire up this ui and it does a gazillion uh expensive lists um right, like one for each resource type or something something like that. um Every.

D

Ui, I know of does that today and opens watches on all of those across every space that they're in across all the users of that console. At the same time, yeah.

C

Yeah yeah and then the user's like, oh well, this isn't quite right and they kill it and they start it again and they do this like f, four or five times and yeah. um Well, I don't feel like we've arrived at any concrete guidance here.

B

F

I I guess that.

B

It would not be a waste of cameron's time to write a kept to describe this feature that included sections about how it could actually be safely rolled out across ha servers on upgrades, which pieces of validation would need to change first and other second um to be able to make it happen, and that the complexity, the time complexity of evaluating this we're not worried about the additional cost. For that.

C

Yet I'm actually not worried about any of that stuff. I can imagine how to solve it. The the thing I'm worried about is just uh do we want to add features here and if so, what are what are our guidelines? What are the boundaries? uh How do we know like? What's the what's the theory of these things, I I think we've established that the original theory of when we added these.

C

We don't really use any of those properties, um but does that mean anything goes so I I that that would help me that would help me. A lot uh is if we could just write down what our new theory of label selectors is.

B

Is that a clear next step for you cameron, I mean it's a question. I guess how motivated is it for the future.

F

um I think I should probably share my motivation um on my agenda.

F

uh I um I actually I've recently started like um scaling using custom metrics for scaling, and I've been so, for instance, wherever I please write, mq and the the metrics uh have don't have at the metric they're, just straight coming out of rabbitmq's there's no like fancy transformations or extra labels or anything attached to it, and the problem is: is that our application name for the queue is at the very start and that's like delimited by three components: you've got the application name uh and then the resource, and then the type of message, um and so my my frustration.

F

The reason why I added this was because I I didn't want to have to actually match um or explicitly write out every single queue that I want to scale up on. I want to be able to just ride along and say, hey any queue that starts with this application name go nuts. I can scale based on that.

F

um So that's um that's sort of where I was trying to head with this and the problem with that uh with the prometheus adapter implementation at the moment is that it doesn't support. um Well, it's actually the validation rules on um for label selectors is that I can't go along and throw in just uh prometheus operators because it gets like knocked out because of the validation.

F

So just eventually I was gonna get there. Okay,.

D

So this was um this: is an aggregated api, prometheus adapter, trying to add a label selector to a metric exposed by the custom adapter, and you want to be able to do a custom label selector on non-real cube resources because they're under an aggregated api and they're virtual instead of physical.

F

That is exactly right.

D

And the validation, the specific blocker is the validation of the label. Selector happens early in the server processing chain and is blocking you from actually just saying you know what I don't want to use that library. I just want to pass validation all the way through. Do you run into client validation issues like? Are you being blocked in a client-side library, doing validation on the label selector before sending it.

F

No, it was service, side, validation,.

C

uh If that's your setup, why can't you generate a virtual label? That is the queue name.

F

I've done that as a workaround in the meantime, but then I thought that um there was additional value to this as well um like being able to select. You know just a generic use case like being able to select a bunch of pods and exiles over it to delete pods or something, if um yeah there's a bunch of generic use cases that this feature has as well. But that was my specific motivation for actually adding this.

D

Do we so you're like just to be totally certain you're, actually failing because you're reusing? Are you using the generic, the prometheus adapters, that use reuse, generic registry and that's doing the validation ahead?

D

Because you could argue that this is just a missed feature or a flexible point that would be not unreasonable in a generic registry for an aggregated api server.

D

I don't actually think we require that aggregated api servers obey the same semantics on field selectors and so then there's a question of like this is a separate question, which is, if you call yourself compatible, can you support more operators than what label selectors or field selectors support today, which is another side of it, which I would probably be much more willing to at least think about that and be like yeah? This is just a arbitrary limitation. Aggregated api should be able to virtualize this stuff.

C

um So before we go there, uh this sounds like a pretty niche use case and I'd be a lot more comfortable, adding something like this. If we had either a lot of people with this use case or um a lot of people with other use cases, that also would like this same feature.

B

D

B

Actually seen a use case coming.

D

I have seen aggregated apis that would want more flexibility in field selectors and can implement it uh because in in that, it's delegated to the strategy label. Selectors are not delegated strategies, so someone reusing, keep api server for an aggregated can do it for field selectors, but not.

C

Yeah, the the thing is, I actually think generating a virtual label is much safer because you don't have to worry about if some queue has a prefix of the other queue which which actually raises.

D

Another point which, like it's, probably pretty reasonable for the prometheus adapter to be able to do virtual labels based on labels of the underlying metrics anyway, in some limited cases, in which case this is a very reasonable feature, request for the prometheus adapter, and you probably get support from other folks who need to do similar things.

C

Yeah yeah yeah, given this understanding, I I guess I kind of have to lean no on this pending pending other other users asking for it.

D

But I would be willing to help push on the prometheus adapter side cameron, um because I know some other people who have this problem and I helped push prometheus adapter at one point, so I at least am guilty for some of the elements of its design.

C

Cool okay! Well, I think I think we're just about out of time and just in time.

A

Okay, um so I I just want to make sure everybody's clear on what are the next steps.

A

All right are, we, are we not.

B

I think this one is now in a state of uh what is it awaiting more evidence? uh Is that yeah, okay,.

A

So cameron doesn't have to go and write a gap yet unless he wants.

A

Okay, very good. Well, I think we are at time I will upload later the recording and let everybody know thank you for joining and thank you for your time. As usual, stay safe have a good week. Bye.