Kubernetes API Machinery Special Interest Group, 20 Apr 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG API Machinery - 20220420

Description

April 20th, 2022
[35 minutes] [wojtekt] API pagination in watchcache
https://github.com/kubernetes/enhancements/pull/3274
[20 minutes] [liggitt] etcd client test gaps
context:
https://groups.google.com/a/kubernetes.io/g/steering/c/e-O-tVSCJOk
https://github.com/kubernetes/kubernetes/pull/106591
https://github.com/etcd-io/etcd/pull/13737

A

There we go recording. Thank you for joining. Welcome everybody see api machinery by weekly meeting today is april, 20 2022 um a lot of tools on the number. um We have a couple of very interesting topics today and um I think we should get into them uh to use the time appropriately. So the first one is for wojciech. Thank you for joining voice check. I will leave it with you.

B

um Okay, so um I what I, what I wanted to discuss um is the proposal that I or the cap change or update to the cap, a long living one for the api um api list pagination.

B

I would like to finally graduate to ga um I mean the original cap is isn't coming from me, but it's coming from clayton from one eight or one seven, or something like that. So it's like five years old or so, um and we still didn't graduate to ga.

B

um So so I think it's it's high time to actually do that um uh and um one of the things that is, or probably the only really controversial thing is um um that uh I believe that we one one one of the things that we should do for for for ga um is to add the pagination support. Also to the watch.

B

Cache and- and um this is this- is the core of the proposal or the core of the change in the proposal, um um and I think I'm not sure do you want me to to briefly go over what I'm proposing or like. I think that most of people here read it because there were a bunch of comments. So um I.

C

Think it's worth giving at least a recap of the delta from where we are, which I could imagine g.a, as is because it's been there for years versus where you're looking to take it.

B

Okay, so um the where we are currently is: basically, we have a support for pagination when listing from hcb. um We don't have any support for pagination when listing from watch cash. So we have this strange semantic or, in my opinion, at least strange semantic where, um if you pass a resource version equal to zero, then the limit parameter is actually ignored and watchcast is returning, like all the items uh requested by the list, um if you don't specify resource version, I mean it's empty string. It's basically always is is redirected to at cd.

B

So we are. We are just honoring, the the delimiter, the limit parameter and and do the pagination. If you set the resource version to some actual number not equal to zero, then if the limit is said, we send the request to hcd and use pagination. If the request is, if the limit is not limit is not set, then we serve it from cash.

B

um So so this is roughly where we are. What I'm proposing is basically to start honoring the limit, um even when the list is happening from cash, um which is um fairly like pretty helpful for the performance reasons, and then you will unify the semantic of the wrist. So we will be always honoring the the limit parameter and in terms of implementation. I think that the biggest concern is that it's potentially complex change, um the the in in terms of implementation.

B

What what I'm proposing is to basically switch the um the data structure in which we are storing the the current state from uh from the current like more or less hash map to a b3 that is actually the same like reuse, the same b3 that scd is using underneath um so hopefully from hcd. We we get a lot of like so we have a lot of soak time.

B

So so, hopefully it is very, very fairly reliable structure um and build the and basically wrap the the this b3 into the interface that that watch cache is currently using to and then um yes wrap the b3 and to implement the same interface. That watch cache is currently using and simply replace um the existing. The existing cache with the b3 based cache.

B

D

Level there are.

B

Like a couple like smaller changes, that needs to happen there, but um I think we can start from from here.

E

Yeah, I think that's good enough to have a discussion.

E

um I I was gonna say some other stuff, but before I before I dive in that uh actually david raised a good point, which is um you know why what's what's wrong with just gang the way it is, uh I mean sure the semantic is inconsistent, but um I'm you know is that really. Is that really that bad.

B

I think this misleading, um I think it would be useful also to support the like um the problems that we have is that we can't basically redirect many of the existing, or there are many requests that we can't redirect to at cd, in particular like listing pots that cubelets are doing when on on start, because.

E

B

E

B

E

From a from a performance standpoint, not from a.

B

E

C

Okay, um so what if.

E

C

Said we want to ga this as is but before we do, we want to have a ga level for three releases of the initial lists from resource version. Zero are handled by using a watch instead and we get all of our known and well-behaved clients fixed, and then we say: okay now, when you set limit, it's going to be behave consistently and we will just always send it to etcd, and if you crush your lcd well, that's probably because you aren't using the recommended way to list watch, don't.

E

Do that so I I was actually going to say before we go into specifics like that, like we could have two different discussions here. One is how to it sounds like there's two different discussions.

E

One is uh how to how to make this ga to tie up loose ends, and the other is uh I, I think if you read the subtext here, you're saying that we need to do this watch cash thing for performance reasons uh which it's not really a ga or not reason this performance, and this is just a convenient time I wish to do it. Does that sound like what you're saying wojtek, or did I mischaracterize you.

B

um No, I think it's it's fair summary.

E

Okay, all right, then, then, then uh yeah, so why not the thing that david said.

B

Yeah, so I think um I was thinking about it and I'm still seeing a bunch of um especially third-party um third-party components that are using like client go from like one, seven or one, nine or whatever 111, um and it's hard to convince all those people, and we don't have like a good leverage to convince them to actually migrate to newer versions.

B

um And with that, like any changes that we do currently will take years to propagate, to all those components and um from operators perspective. We it I'm a little bit afraid that we will. This will har.

E

I mean we've: we've got some tools right like we could. We could uh uh tweak api priority and fairness to like serialize all uh non-ideal list requests or something right like it would make those clients slow and that would give people an incentive to update them.

B

um Yes, and no I mean we can prevent them from from overloading api server. I 100 agree with that one, um but we will potentially break that functionality that those components, um those components uh um those components provide and like in some cases, were um where cluster admins actually own, also those components they will.

B

They will probably be able to fix that, but in some cases those are like some third-party components that they just use and um all they can do is ask those third-party providers to change that which is sometimes happening.

E

I understand I at the same time, like of all the strategies I've seen to like get people to upgrade things like just making it slower is like the least disruptive uh and and like the disruptiveness, to like probability that you'll actually go up upgrade it ratio is. It seems pretty good with that. One.

B

um Yes, on the other hand, like the biggest escalations that I've seen from customers are also coming from. Why did you break my scalability? Something worked and I didn't well.

E

B

Yeah, it's just both ends. Yes,.

E

Okay, so uh here's here's what I hear: here's here's the thought I have about this, and it's actually not just in relation to this proposal, but some other discussions that have happened in a few different places.

E

I feel like uh api server has a kind of a choice to make like historically, we have decided- or we have at least pretended that we're not a database and we're not offering particular features of a database such as indexing, fast queries, uh complex queries, um transactions right, we're, not api servers, not in the business of doing those things, um so that and I'll call that the status quo option that that's that's kind of. Historically what we've done.

E

In contrast, I see two potential roads that we could go down uh the so the second option is the road where we admit that we're a database- and we start implementing some database features, uh and I would consider this uh b tree and uh mvcc thing a database feature. um So I think I see that as moving us down that road other other other similar features again are like uh implementing uh selection over uh uh particular fields on crds right. If we made that efficient. That would that's definitely a database type feature.

E

So I see that that is like option. Two is like okay, I guess we're a database and we better act like it and implement all this stuff, and then I see option three as like. Okay, the contract users want is a database, but our business value is not adding database features. That's that's not necessarily the thing that we're good at um or maybe we are. I don't know if we haven't really tried that much but uh to implement the the user requests for these database type features.

E

um Maybe it's time to get a real database uh or you know somehow right, and that would be super disruptive. It would involve like, like, uh like our storage format, would have to change right. You can't store opaque, proto, double double proto blobs in a database and expect the database search features to work right. So that would be. That would be super disruptive, but it would maybe be the most. The more correct way of getting database type features that users the users want.

E

So I see like we've got maybe there's more options, but those are the the obvious three that I see and I think we've got like a strategic choice to make.

E

I wouldn't want to accidentally start building a database any more than we already have without like consciously deciding that yeah. That's the route that we want to go down all right. That's.

F

I'm not sure we need.

E

Yeah, how about whichever then mike.

B

Yeah, I think um I I understand the concern. I think um the the reason why I don't fully agree that it's it's now this the decision point is that, like we already added support for pagination like it's, it's already in the api, so it's a matter if we implement it like consistently or we implement it in the currently um partial partial way. So um I think it's um it's true.

B

It would be true if we would be adding like pagination now and then the question like the extending the api to support pagination, but I think, given that we already added it, it's only a matter of like.

E

Partly right, like the imagination that we added uh currently passes through to to etcd, so it's it's that's sort of in category three right where we're using the database as it's intended.

E

uh I think uh mike and then yeah basically.

F

I think I wanted to agree with way tech I mean, if you you know, just stop and think about the api. We have forget about the implementation. Think about the api right. We've already told clients they can paginate right, we've already told clients, they could do field selectors, you know, um so I'm not sure yeah. I don't see it as you know, signing up for a bunch of new features. I see it as finishing implementation of the features we've already got.

F

C

Option for finishing that implementation is to say we have the current state that we have and we consider that finished or we could say we have the state that we have and we have given you a way now to be efficient and we are going to make your limit request, always honored. The side effect of that is that it can be slow, and I see that as completing the feature we started.

E

um Almog, hopefully I didn't mispronounce your name too badly.

G

Yeah, so so maybe that's repeating what uh you guys said before, but I think like you can see that as option which is not revisiting and not not building a database but just enhancing what we have. It's, not uh it's not like building new features but more like improving the implementation.

E

G

E

I I would say yeah I I get where you all are coming from, but I would say that the databasi features that we have are incomplete at best right, like field selectors, are hand curated and extremely unperformant. uh Pagination works only in particular scenarios, which is the complaint here.

E

There's no transaction support at all right, like um our queries in general, even of label selectors are rather are extremely inefficient, uh except for a couple that we hand optimized right. So the the question isn't like. Like yeah, we gave people some half-baked semantics, so the the the question is: do we want to like turn those fully baked uh or live with it? The way?

E

The way that we left it? um That's the that's the question I see it. uh Steve has a hand up.

D

um I think it would ask like what you know when we're considering finishing out things like label selectors or field selectors or whatnot from david's framework like if you're, not delegating that to a database that has that done and you're implementing, b trees in the api server like. How is that not building the database right like vitamins.

C

Yeah, I'm not proposing that we build these trees, uh I'm proposing that we explicitly don't say we have the features we have. We have the scale that we have uh if you are looking for something different or something more you're. Looking for a system on top you're, not looking for something that we provided here.

D

Yeah, I think that was option one right, so I think one and three make a lot of sense. You know kind of as a framework and then a two which is maybe where this cap is seems a bit.

E

Harder to justify, I mean we, but we already have aspects of two right: we've already got, watch caches uh and stuff um boy, tech.

B

Yeah yeah, so I think think I wanted to better understand, like the concerns that you have so do I understand correctly that um you are mostly afraid about introducing additional complexity due to this feature or is? Is it something else.

E

um Complexity is definitely a large part of the concern uh and like uh not just not just the like complexity of getting it working at all, but also the ongoing maintenance um so yeah, I I don't know if that's completely, uh I don't know if that's completely the concern, but that is definitely a part of it. Yeah.

C

That is my primary concern. I am primarily concerned about the number of people available who could effectively debug problems in this area in.

E

Yeah, I I feel like adding uh yeah, I feel like adding a b tree uh and mvcc is definitely like a like. Maybe it's arguable that we're not a database before that, but if you add something like that in uh it's time to stop pretending.

B

I'm wondering if there is something that we can address um that or I can address um to the concert of like complexity like would better testing or what do like anything like that, like better extraction of things, would help here or it doesn't address in anything. In your opinion,.

C

I don't think I don't think that I can say.

C

That additional testing is gonna, make it easier for us to find people to pick up maintenance in an ongoing way, and I think I would point as evidence to the issues that we have currently in ncd, which is based on similar technology.

C

Right um and ncd is suffering from a lack of people, either willing or able, or both to to maintain it.

E

I mean you can also argue that there's not that many people that can maintain the existing watch cash.

H

What about you know, memory consumption? It seems like this solution will actually consume more memory and I think yeah.

E

H

Like in general, like the watch, cash right.

E

Yeah, I was a lot yeah, that's that's a tactical yeah concern and I I noticed that reading this morning, I'm not sure how you're going to fit this in any sort of ram budget.

H

B

H

I think you know usually databases they usually like maintain a sparse index, then like a buffer memory buffer. That is, you know, capped at certain, like one gigabyte or something data, and then you bring data from disk to the buffer and then index right. So this is how they, uh I think, I mean at least some databases. This is how.

I

H

Constrain memory, consumption. In this case, I think you can go very high, there's a limit essentially, which is dangerous.

B

Now so I I think I I didn't have time to respond to that, um since I saw that so this comment, but I think it's it's doable to to to make it work within the existing limit, like it won't decrease that the existing memory usage, but it won't visibly increase it. I I I'm I'm completely fine with saying that, like it's, it's a requirement that we can't increase, that.

E

You help me understand why you think it won't increase the the memory usage.

B

um So I think your main argument was, um if I regret correctly, that like we will increase it, because we we will need to store more history.

E

uh Not that we'll store more history, but uh well yeah that we would. We will store more history for uh specific objects, especially if, if a single object changes a lot.

B

um Yeah, we are storing the objects themselves anyway, because we are storing the transaction locks anything. So it's it's not that we will be storing more objects. It's only the.

E

We store 75 seconds and you propose to raise that to two and a half minutes, which is definitely yes.

B

So that one we can probably fix and say we we are still going to store like 75 seconds and if we are, if, if we exceeded that and like that, the request is basically older than that we just um forward it to its cd. It's it's fairly rare that, like um we are like doing continuations from that. Also, we can probably like afford um redirecting those to fcd.

E

um I mean, while we're talking about like tactical alternatives. uh I think, should we consider some other thing like like like it occurs to me that a lot of times people just want to list the names of objects like they don't want to. They don't want the contents of all the objects right. If we implemented a list api that just listed names, could we maybe return all the names in one in one response and not need to rely on imagination so heavily.

B

Is it really the case that, like names are enough, I think that's.

E

The that's kind of the question right like how how much analysis did we do we do to determine that the way the only way to get a win here is to make pagination universal.

B

um Yeah, so um I should probably have added, I think, all the cases that I'm aware like, which is like number of different components from different areas like they. Neither of those need like full objects, but all pretty much. All of them need more than even object metadata.

F

Yeah, um so one of the things we find in databases is, you can do a read with a projection. I say return this fragment or these pieces of the object. um I don't know we're taking cases you're talking about is something like could something like that. Yeah.

E

That probably would help if our primary problem was like network bandwidth between api server and clients which actually, in some cases it is. um But I think in many.

D

E

Cases it's just it's just cpu like going through all the objects right, because api server has to perform that projection.

J

Well, but cpu cpu decentralization is actually, I still think, if not the overwhelming majority of the time. It's still a big chunk of the time right. We optimized out almost all the other components but like serialization, deserialization and re-serialization are still fundamentally expensive and the problem is, is there's a lot of overhead in that approach and it's diminishing returns. The more crds exist in the system.

J

Right like that's, another problem is like. Are we optimizing for people having more and more positive notes like? Is that the only thing that we're optimizing for in cube? Because that's that's a thing? It's not the only thing.

C

It would be a very different feature to come in and say I think the most useful thing we could do is to create a binary format for custom resources, and maybe it is or maybe it isn't, but.

J

I mean even I I only bring that up just because, like uh avoiding deceleration, avoiding serialization, there's a lot of things there. What is the thing you're optimizing for as a project? Is it partial lists of pods on nodes.

E

I yeah I I mean that that goes back to the. I don't know if you were here to to hear my uh trilemma clayton, but that that kind.

J

Of goes for not being.

E

Here, for that, that's, okay, that goes back to the the choice of like you know: status quo, uh which is like don't implement any more database features or admit that we're a database and go ahead and implement database features ourself or uh admit that we're database features but we're a database. But we want to outsource all the features to an external database right.

F

E

By the way is steve, are you on the agenda yet for telling us about that.

J

But before we even get into that, but like there's another one which is kick the can down the road, because this is the most important like how important is this characteristic of scale versus? Maybe it's not the right time to tackle this? Maybe the that's like uh sure.

E

We could delay it, which is equivalent to the status quo for now,.

J

Delay it deliberately, while we invest in a better assignment like one of the things I struggle with is I the use cases in the ecosystem are diverging, but I'm not sure that the high scale people really still care about anything other than base pod or node performance. um It's really two different audiences.

F

Hey I'm going to raise my hand here, yeah, I'm I'm right now working on something that has a lot of stuff. That's not pods. There are things defined by crds.

E

Yeah, I I think uh I actually agree with mike that we need to consider non-built-in resources.

B

Yeah, but I wanted to say that it's somewhat orthogonal to this cap right because it doesn't say anything about um it's not specific to building resources serves. It works the same way for us.

E

But we do do we even have the watch cache on for crds.

B

E

B

I mean okay, I mean I'm not sure like how many people are disabling, that which is like hardly possible, but we we do have it by default.

E

Okay, okay um yeah. I.

F

Do think that the only way for things defined by crds is to turn it off period.

E

uh I'm not sure about that, but I I.

F

Tried that's what.

E

I found okay, then you're, probably right if you tried. um Maybe that's a bug, though,.

F

uh Yeah, maybe it is, you guys tell.

E

Me I do think uh like improving crd scale, is something that maybe um uh also fits into this trilemma right.

J

Yeah and again like we, I don't I don't know, we've fully like hit all of the performance aspects of like deserialization. I'm not saying that and in fact I'm not I'm not trying to apply void, but, like that's, this isn't the right thing to spend our resources on I'm just a little there's. A lot of things like we've got the protobuf um uh undigestable lump sitting in our stomachs of protobuf2 and gogo protobuf.

J

I'd like to, I think, there's some wins there. They may not be obvious wins but like it is a, it is another type of problem, that's not performance that has to be addressed and it does hit on some of the other aspects of serialization and crdb and or um crds.

E

But this I mean yeah, this gets back to like you know, should we decide that? Actually we want to have a real database right. We would change our storage format significantly and that would render a lot of optimization. There kind of mood wait, wait.

F

E

Wait there now, I think, you're leaping I mean just imagine, imagine that we decided, like I don't know, postgres is going to be our database now I just this is just an example. uh Would we store opaque, protoblobs and postgres? No, we would split things out by whoa.

A

We would define schemas well because you can't.

E

It's hard to search opaque, protoblobs right. If it was just one layer of protoblob, then then maybe we could do that, but we don't have that. We've got an envelope with a protoblob inside it as raw bites.

J

Right that's another example. Daniel, though of there are json indices in the world that are just as effective at some of that yeah and like these tradeoffs, and I think that's the short.

E

But total valid.

J

E

You're switching to to like uh uh which uses bson, which is basically json but binary like even that we wouldn't be able to search, because we have this double encoded. Proto thing right like we would have to adjust the storage format to make it work with mongo's indexing and that's.

J

Maybe what I was referring to with like google for buff right, like we have, we have a we are going to have to make a serialization change potentially to unlock there. We're gonna take a performance hit at the same time. We do it and that's going to revert the high scale pod node use case.

J

I would suspect, although you know it's possible, it doesn't c crds benefit nothing from optimizations. We make in that space. So, like I guess it's a question of how much are we willing to invest to get the real improvement.

E

Well, yeah, I think it also like it's not just how much are we investing, but what are we investing towards right? If we're going to switch storage engines, then that's a very different storage. Optimization then, assuming we're not going to switch storage engines, then we just want to make deserialization faster yeah.

J

And storage engines- mate is probably a very, is a significant amount of time to get to a point where any of us would be confident about doing anything other than basic first steps towards an exploration. So that's basically kicking the can down the road.

E

Yeah, but even even when we're kicking the can down the road, I think it's good to have some idea of which road we're kicking it down right like otherwise, there's not that many of us to go around, and I I don't want us to be wasting effort and going down the wrong or going down a road where we're ultimately going to change our mind.

E

Okay, let's try and somehow wrap this discussion up a little bit. uh Almag.

G

um Just a quick question: do we have some place that we documented the use cases for this uh considerations.

G

For pagination, uh I not for pagination, but if we're talking about the possibility that we're admitting that we should behave like a a database, all right, at least for me, that sounds a little bit like. Maybe it's an overkill and if that's the use case, what uh what our? How? What our use cases for that right.

C

D

For instance, asking people uh hey I'd like to create a user provided an index on my crds, like that sort of thing.

C

Yeah uh field queries are sorry fields, selectors label selectors. Those to do efficiently require an index to be built and cleverly using the xcd transactions, for instance, could give you a look aside index and I.

E

Additionally, other people from.

C

Years ago, so I don't know if we wrote it down.

E

G

It's probably worth writing something down right if, if you're doing like in in a small capacity uh field search, that's not really uh hard enough for us to to implement all of this kind of overhead inside of the api right, but.

E

C

Think the search is actually pretty hard. Yeah, I'm gonna, take the other side.

E

C

Once you build a version aware or serialization version aware, field index, plus a query, planner then you're, like 90 of the way into code, that the vast majority of developers are not going to be suited to maintain and that that probably includes myself right.

G

So what they meant is what is the use case that require you to do like field search? Maybe the use case doesn't require it to be high, very high efficient. That's what I mean.

F

So let me tell you one use case that I think is not. I is not as widely appreciated as it should be. So one of the things that we find ourselves wanting to do with kube api machinery is keep is we have references between objects and uh we want to keep the referenced.

F

um Let's see, the the reference object is around as long as the the referencing object is using it. So, for example, while a pod is using a pvc, we want to keep the pvc around and similarly, while a pvc is latched or bound to a pv, we want to keep the pv round now. We do have transactions in in the kube api, but they're single object, transactions, they're single reads and writes, um and so you can't maintain referential integrity. That's what this is called in database land. You can't maintain referential integrity with single object transactions.

F

Okay, so we just can't do it uh in directly in the api machinery, but.

D

What we do is we have.

F

E

You it is possible, but you have to do a a bit of a dance.

C

During an absolute ordering, I believe you could fundamentally do this. I would not claim that it is.

E

Right, you put, you put both objects into a like: I'm hooking them upstate, and then you make them reference each other, and then you take them out of the I'm hooking them upstate right, okay, so okay, you're kind of going where.

F

I'm going okay, so uh the point is the api machinery itself can't do it.

F

uh What you do, what you're talking about is enlisting the services of a controller uh and so right we have existing demonstrations of how to do it in the existing stuff that keeps the pvc around while the pod is used and keeps the pv around while while a pvc is using it and so the key, the key way of you know- and I wrote up a blog entry about you- know the abstract idea here right and the idea is that you have uh you use a finalizer and you keep an object around while it's in use by another object and you enlist the services of a controller to you know, remove that finalizer when it's okay to to have the object deleted and by careful ordering of the actions of the controller that you can do.

F

um But in order to do this pattern, you basically have to query for things that are using a particular thing. Okay, so now we're getting into query by a field selector.

F

um So if the existing code, I think, will actually do that uh filtering on the client side and that's you know functionally correct, but it's more efficient if you can do it on the server side.

E

Yeah, I I think to take what what mike is saying uh and uh what david is saying and combine them. uh What we're saying is we don't want to write this hard code so uh that that we don't know how to maintain so we're gonna make users do something similar to work around it?

E

um That's a that's! That's one way of interpreting this, which doesn't put us in a terribly good light.

F

Right, it does have the slight advantage in that you know when you're implementing a database server, you have to implement the combination of all the features, whereas typically each client is not asking for the combination of all the features. So any given client has a simpler job yeah, but.

E

Like just observing people, try to uh try to do this dance correctly like like, if you've got a multi, if you want to do a manipulation that requires multi-stage, locking to do correctly, uh it's just really challenging for most people. To do that. I agree ways that I wouldn't have appreciated before I saw people struggle.

E

F

Yeah, so you know again, I can I just want to emphasize you know I I think you know without yeah, I'm not so much interested in um I kind of am so. I have a short list of features that I would like to add to the queries, um but you know mainly, I think, we're talking about you know we started here talking about completing the existing features, um but anyway you know.

F

Overall, you know I kind of sympathize with the idea of you know. Why are we repeating the work of implementing a database when there's already other people that are implementing maintaining databases? Maybe we should just sign up for storing in a more serious database, and you know steve tells me he's going to present something about that. Sometime soon,.

E

All right yeah, I I uh steve, had an item here and I booted it to next time, because I thought that this item would take a while, uh and on that note I think we should move on to the last topic and, unfortunately, jordan, uh chatted at me and said that he's stuck and can't join us. um So hopefully we can figure out what he wanted to tell us from these contact links.

F

So can I one other remark that you know it's kind of bugging me about this previous discussion about people. You know not having people available to maintain stuff sure you know this is something that it was marked. It came up in ncd, it comes up again and again. Right I mean we've had cases of like absolutely foundational stuff right, like the log for jbug right, there's, absolutely foundational stuff, which is just pitifully under-maintained yeah. It seems to me: we've got an industry problem here. Right we've got everybody trying to free ride off of almost nothing.

F

You know that there's a there's a there's, a big industry structure problem here that you know, needs a more fundamental aggressive solution than us. Not trying to you know finish implementing our features.

E

I I agree: it's an industry right problem, but I don't know how we can solve it and I don't know if it's prudent to ignore it. Yeah.

C

That's my point: I'm interested in finding a way to solve the needs for my little spot. uh I'd be very happy with something generic, but I don't know that I see a practical way for me to influence log for j, for instance, right.

F

Of course, we're not going to solve it even for ourselves now. This is this. Is a problem that's bigger than us all we can do. Is you know kick it upstream or up up? You know up to high level people right. Just add our voices to the people saying there's an industry problem here right, tell our execs there's an industry problem here.

C

I'm hoping for something, maybe a little bit more concrete uh and lower level for etcd in particular, but I am sensitive to not. I am aware enough that I don't want to dig that stack deeper. I have one basket I need to watch. I don't want to make a second one.

F

I I think we, you know, look the big problems aren't going to still get solved quickly. Obviously we have to take. You know tactical steps and do what we can coping with industry as it is today, but I think we you know should also you know, add our voices to saying you know there is a big problem. It needs a bigger, different way of being solved.

E

I don't know concretely how to do that.

J

The easy answer is find the use cases in kubernetes that have people who need to go solve those problems and find a way that sustainably overlaps that investment. That's the only way that open source works. That is how open source works. We find problems that need to be solved, and then we align the investment in that new thing and we get the win on fixing the things that we come out like that's the only way that anything ever gets done anywhere, especially in open source, and I think we can do it.

J

That's that's partially why I was asking about watch cache performance like is this a scale problem? um That's why steve is working on the database stuff because, like we think, there's to use cases around cubans crds that are can be improved. um I know there's people who are looking for it, like I'm, trying to find the reasons that we need to improve some of these specific things as dan. You were saying like. Why are we doing this? What is the use case broader than just get to.

E

Yeah, I think, um yeah, I I'm worried that we didn't give wojtek any guidance whatsoever, any concrete guidance about this cap. um I I uh uh I I do think we should take a minute and and talk about the ftd client stuff that uh jordan has put here um uh uh from what he told me in in over chat. There are, uh let's see.

K

I I actually am here I'm on mobile, so I'm not sure my signal but uh yeah I'm waiting for a tow truck. So who knows what this will be like, but just to summarize briefly, the context is um in the uh the thread about ncd.

K

um I linked to a particular uh pull requests: updating the sap client, where there were some questions about like test coverage and there's at least one area where it doesn't seem like there's sufficient coverage upstream and there's no coverage, I'm aware of in our tests and so like in terms of like concrete impact. It means when there's a update to an std, client library like as a reviewer. I don't know if it's safe or not.

K

If it's not well tested upstream and we don't have tests, I don't know if it's safe to accept, and so I just want to raise that particular area and see like do. We want to try to help add test coverage upstream or get coverage in our kubernetes suites, and what do we do in the meantime until that coverage? Is there.

E

Yeah, I I think uh to go back to the trial trial that I'm that I'm positing.

E

If we're not going to switch databases, then we had better make sure that fcd is functioning the way we think it is functioning and anyone who's been following, probably knows that atd has had some bugs and it was suggested that you not use the first couple versions of 3.5 um which is concerning. So uh this is yeah.

E

I I don't know if anyone else has.

E

Ideas about things we can do to help the situation.

E

But yeah, I'm concerned in general, and this this is, I mean it's: it's both an argument for finding a real database that nct is having uh issues at the moment and it's also an argument against attempting to implement our own database, because why are we going to do better? um So that's that's my uh that's my current take on. uh I I to be clear like like.

E

If you'd asked me like a month ago, uh what I thought of rolling over to a different database, I would have been like 100 against it and today I'm only like 80 against it.

C

I'm definitely in the. Why do I think we do better camp.

E

E

D

uh Jordan, if you're still here, I was hoping that you could expand on what kind of testing we could add and like does it does appropriate testing here, look like going upstream and filling in the gap as reviewers for etcd or something cube side like what is what do you expect.

K

um Yeah so the the particular gaps that I was noticing were around things like failover scenarios and multi-server scenarios and tls handling. So those are the three areas I know have been problematic in the past and I'm I'm just trying to chase uh issues and test stuff upstream and std.

K

It seems like there's been some efforts made to to close those gaps, but there are still gaps flexible, and I like to um it's not clear to me that the tests that are there actually match what production environments are encountering and so that, like trying to highlight the gap and then it seems like closing the gap upstream, is the right place to do it. um But I I don't know enough about their test infrastructure to know how reasonable that is.

K

E

I would not, I would personally not expect to have ncd gaps tested in like api machinery or kubernetes repos. That doesn't make a lot of sense to me.

C

I could certainly see having it in both. We have tests to make sure cube works, for instance,.

E

I I wouldn't want right, like other people, have these problems and or potentially could have these problems, not just us, so it doesn't yeah. They fail.

J

E

J

I mean it is worrying that we don't have failover tests so, like certainly, I would say when openshift was doing, like our rolling upgrade tests and all that, like we built in things to make sure that ncd stayed available and we definitely hit client hangover. Bugs and sam put some things upstream. But we didn't quite get all the way because we don't run a ton of aha and cd rollover. Upgrade scenarios that expect rolling updates of std like we're not hitting that scenario.

J

So I would lean a lot of the failover connectivity stuff to the upstream because it's going to have fewer dependencies, but it also points that like do should we have. um Should we be thinking about spending a little bit more time on rolling update of cube api server and net cd upstream.

E

K

It would be totally reasonable to test.

E

That uh failover from api perspective, api servers, perspective, works um but testing it to verify. If we think that most of the problems are in the nct client like, I think we should. We should test that uh in the etsy. So.

K

One one nuance of that is that um the failover behaviors, like a lot of the weird edge cases, tend to live at the intersection of the scd logic and the grpc libraries and the versions of the grpc libraries. That etcd is testing with may or may not be. The versions that we are building into the cube api server like we might have picked up a newer version of grpc david.

J

Definitely hit some of those like early like we had a lot of grpc hangs. I haven't seen those in a year or two. I don't know david if you've seen them recently, but I I I would worry about version skew there, because a regression could creep in and it would hit us at scale almost immediately like us.

J

If the tested dependency.

F

K

With the given version snapshot, so if the test lived in scd and could be invoked as like normal go tests, we can run those with the versions that we have, but the more esoteric or involved. The setup is like their integration tests or ede tests. On the sd side, we don't have good ways to like run their ci, complex, ci stuff, with our grpc versions. Right.

E

uh We we could. We could like uh uh like do that in a fork of fcd like right, like fork, cd put in the grpc version, we're going to use, run the tests right like something like that.

K

Right but that starts to look more like us testing at cd behavior.

E

If we're swapping out dependencies is reasonable right.

K

Right but like at first glance, your reaction that, like it seems like these tests belong in std, which I agree with, gets a little more complex when we also need the behavior to be tested with the particular dependency versions. We're building with right.

C

And as an example, what we do is, we can say: okay, I'm going to vendor cube for regardless of whether you're supposed to or not we're in a vendor cube cube, has test associate associated with it, we're going to pull those testing libraries and run that test inside of whatever it is. We've entered it into. You could do something similar to that in with that cd, where you have a test sure they run it, but it's structured in a way that you can actually essentially vendor that test and run it locally for yourself.

I

I mean just from an scd perspective. It makes sense to me that we would be doing especially with grpc. We would be doing more aggressive testing against recent versions of it and maybe doing against multiple versions of it, because what I mean just the way the go. Mod works, you're going to have you're going to have anybody using the client, potentially picking up, not exactly the version that you tested with. If you want to pick one version.

I

So I'd be all right, I think I'm going to upstream.

E

uh We are running out of time here. um What should we, what should we tell white tech about about this cap.

C

D

Would really like.

C

Well, I guess I also listed it in the cap itself. I would really like to see us make the cost make most clients stop using the case that doesn't work right, which I believe is resource version 0 with a limit set and then increase the cost. You know degrade the performance of clients who are making that request and say you should update to this new thing and then eventually make it just point directly to ncd, and so we gain unification. That way all limit requests are treated the same and.

J

They don't note that note that resource version 0 is wrong for components that are restarting, um because they can time travel back in time. So we already have that thing that we still haven't fixed that bug. So there's at least a part of me, that's like I don't want people using resource version 0 unless they absolutely have to anyway.

J

B

E

Go ahead watch.

B

um There's also a question, a reason that, like um the counter argument, is that, like we will be building the support, for I mean the api will be the api for like streaming. The lists will will be there but like, like a bunch of that, will be built into our client library, which is fine for components that are written in, go but like, and there are also other components written in other languages. That will not be able to directly benefit from that and they won't migrate to it easily.

E

So my my first idea about what we should do is separate the ga discussion from the optimization discussion and um take all the commits to this cap, except for the last one and merge those and call the feature. Ga and then talk about this optimization separately, um because you've got three commits here and the first. The first two are obviously great um and all the stuff that we need to discuss is in the last one. So, but that that, being my my impression for how to make progress, I don't know: does that sound.

C

Reasonable, but I'm less sure about the testing coverage that we get if our reflectors don't use pagination today right how many clients do we have in our test suite today that provide good coverage of pagination.

C

So I think there may be work needed to do that and to do things like preserve the behavior used by the storage version migrator, which okay, I saw it listed as a goal right. So I think there's still work to do even to declare ga.

E

Okay, that sounds totally reasonable and also, I would say, is a good reason to separate the discussion.

E

um I think we probably have to come back and discuss the rest of it next time or another time.

B

Sure yeah I'm happy to discuss further like the um splitting the discussion. I think it um I'm not sure. I fully agree that we should ga without it, but um I agree that there are like the the things that david just mentioned, makes sense to me like universally independently, independently of what we do with.

E

Yeah I mean we haven't changed it in like what 10 15 releases or something like that. So yeah.

C

uh I'm not saying that yeah, I'm. What I'm saying is when I looked at it. I don't think the test coverage was there to give me confidence that it remains consistent for the future.

E

Yeah, okay, right, I think we have to stop now.

A

Well, thank you for the discussion and looks like we will have room for continue this discussion either on a next meeting or separate meeting. um Thank you, everybody for joining and hope you have a good rest of your day and we'll see you next time.

E

G

Thank you, bye.