Kubernetes API Machinery Special Interest Group, 16 Jun 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG API Machinery 20210616

Description

June 16th
- [carried over from last time] Vivek P (vivekpatani): etcd Namespace Quota Design Intro
Relevant k8s issue: https://github.com/kubernetes/enhancements/issues/2754
Introduction to the feature and get feedback (10 min)
- (caesarxuchao@ roycaihw@) Shall we keep investing in the storage version migrator? (10 min)
Box the project,
Call it done when the migration API is done. Automatic triggering can be excluded from GA.
@mspreitz: IBM is using it this way (manual triggering migration)
For built-in resources, without storage version migrator, we need to indefinitely keep type definitions and the generated conversion code of removed API versions. People seem to have accepted this as a fact, there is no complaint in #52185.
@deads2k: This is still desirable because otherwise we can’t drop fields
For CRDs, how do CRD managers handle deprecated CRD versions today?
This guide suggests users either using the migrator or doing migration manually.
deads2k - What we have today seems to actually work quite well and is in use. How about promoting what we have to stable? I see sttts and sanchezl have been working on it fairly recently.
- [@tallclair] Safer rollout of admission webhooks #102019 (30 min)
Good discussion, needs someone to drive forward a KEP. Will circulate across orgs/companies to see if someone steps up.
- [@mspreitz] Support for lengthy node disconnection (10 min)

https://docs.google.com/document/d/1x9RNaaysyO0gXHIr1y50QFbiL1x8OWnk2v3XnrdkT5Y/edit#heading=h.nxm2mmue5hc

A

Okay, here we go hello good morning um today is june, 16th 2021 and we are having our sig api machinery by weekly meeting today, uh we're in a good strike of meetings and good agenda items. So um we have a couple of items that we carried forward from last meeting. um So, let's start with those uh vivek, do you want to recap and and pick it up.

B

Sure so I think the last last meeting I couldn't explain properly what the name space coder is designed to do so. Just to give you guys a brief idea.

B

The idea is to uh like he has does not have like a storage isolation mechanism right now wherein uh every customer is just the same, so everyone can fill up the space as in how they want, and the kind of problem is that when you have a multi-tenant cluster where, like kubernetes is one of the clients and other people are other clients, uh you want some storage isolation for each client that you're using that's using lcd.

B

So this is kind of there to introduce uh that kind of mechanism in ncd. uh How does this affect kubernetes? So the idea is that kubernetes needs to like track storage on different resources, different name spaces, just different items in kubernetes itself, so this can be used for that where, like kubernetes can be like. Oh, how much is how much storage is part, the part using itself or all of the pods or maybe a namespace? How much storage is the namespace using?

B

Currently, I think it's lacking uh went through the fairness and api uh fairness and priority usage policies, and it's still like it's kind of lacking that. So this feature is angled towards that, where, like everyone can track how much storage uh kubernetes is using in general and kind of report back so there's a document. uh There's a proposal on how this is implemented within hcd and what changes are coming through um yeah, so just to kind of get it out there to the community and see what they have to say.

B

What their idea is, what their take is.

C

um I'll I'll say, uh I think it's fine. If the system administrator wants to configure this, I don't actually think it's a good idea to share a cluster between kubernetes and something else, because a cluster is already like, using basically all the capacity from etcd that that there is so. I definitely wouldn't recommend it. But if people want to do that, I mean I'm not going to stop them.

C

I I I don't think that we would want kubernetes to configure and use this feature for itself like to confine namespaces or something uh we already have a quota mechanism. uh The priority in fairness is not the quota mechanism for uh storage. uh That's that's for requests, which you know I request, and uh the amount of data stored is not it's just different different things.

C

We have the the quota right now, there's an object count quota. um One quota that we are missing is a like aggregate size quota and we the reason we don't have. That is that it's hard to predict what size an object will be after it's gone through all of our various steps in the process.

C

I'm not excited about having etcd itself be the quota enforcement just because, um like in the past, uh we've we've had problems where, like api server, had a wrong estimate of the object size and then api server admitted the request, but etcd rejected it and like that just api server is not set up to expect that and it caused a lot of problems. So I'm not very excited about having the possibility where api server could let a request go through and then xcd rejects it like right, like that, makes it very hard.

D

E

C

To to uh fix it, okay, that's all I wanted to say.

F

That could happen for other reasons too, and we do have to kind of be resilient to it in some fashion. I understand what you mean in that one. It was weird because it was like the moment you started saying that daniel I was like. Oh, I like this better than our object count. Quota system, which is, is a bit of a hack.

F

It's not a hack. Let's say it's a carefully crafted simplistic possible system, something more principled at a lower level or something more powerful at a low level has some advantages. It might complicate like your object count code as an api we have to keep supporting.

F

Is it really worth the complexity of separating out object, quota and passing it down to lcd? That complexity would worry me a bit, although I have to admit um you know the vast majority of all tenancy in every cube. Cluster is name space tenancy, and so you know from a practical perspective like what what are the knobs, that people turn and they turn resource quota. They turn limit range, um but then the tools kind of in there and it's a wild west or go build.

F

Your own, if we had a little bit more principled of a lower level, and then we were focused, maybe more on you know, empowering some of that other. You know balancing a fairness and you're right that priority and fairness have some overlap. The higher up you get. uh It is a lot of change that was. That was the word for me is like it's a lot of change. Does it actually make something better if we could just use both together and someone could bring that in if they wanted to? That, has some advantages like look?

F

If you want to do this, do it it's um you need to make sure cube deals with it. Well, like I could daniel, maybe to your point about like the two layers is. I do think cubes should react well to ncd errors. That's part of the storage, layer's responsibility.

F

So say there was a specific type of error and translating that to a cube api error and telling the user what that means like I could buy that argument.

C

There's also like there's also like, um uh if you start getting quota errors like like for a name space like you, could break a lot of stuff about the cluster right like a bunch of automated stuff, like your pods, can't record statuses anymore, it's it's it's! It's a pretty serious failure when you, when there's like an entire class of things that can't be written, yeah well,.

G

Some of the risks these.

B

Sorry go ahead.

G

uh Some of the risk is that like when you go to delete something you actually issue and update scd and add a finalizer, so the object gets bigger, um but from a practical level, our scd keys are organized first by resource, then by namespace. So if I wanted to cap the size of a particular namespace, is the lcd quota flexible enough to say, like I want these n key ranges sums together or is it like? It has to be this contiguous range.

B

uh It's sort of like a contiguous range right now where like, uh if you can think of it as a directory where, like I just cap, the directory by saying slash a is limited to like 500 gigabytes or something, and then what you put in is uh like kind of oblivious to a theory like we don't really care. What's in it, it's just the 500 gigs is the hard limit.

G

Yeah, what that means is that we can't use it to cap name spaces. We could use it to cap certain resources and name spaces, but we couldn't do something like the total of your pods config maps and secrets cannot exceed 100 megs. um We can't describe that because it's in three different key ranges.

B

Got it yeah, but it's just uh like. um Maybe there's like more. I can do it's just a feedback right now, like it's just a technical talk, so maybe we can like kind of figure something out, but the idea is that, like uh it's like, this error is just like any other error where, like uh I think you were talking about the update, so the updates, like, oh, if you're, trying to update something that's okay like it does go through like the design, is kind of resilient.

B

That kind of stuff where, like oh, we understand you're trying to delete or just update something, and if the size is going down, then that's okay, but technically we're trying to limit like the final problem.

B

We're trying to solve is like uh we just want some storage kind of definition, of what storage means to like each customer that we're providing nhd and cube is one of the biggest ones for sure it's just that, like the other guys who who are there like, they just need a little bit of space and then um kubernetes choose through a lot of stuff.

B

So it's just the idea of providing a little bit of isolation with like what we can do best and kind of still maintain like uh the kubernetes operations, just like unaffected sort of so yeah. I get like uh it's gonna break a lot of stuff, but we can kind of accommodate and then kind of move forward, rather than just like. Oh, this is how we want to do it. This is just a feedback.

G

So I think a couple concrete things that we've come up with are if the size gets, if you actually bump into the size, and then you try to delete a pod, for instance, that pod will actually have its object. Size grow before it's deleted, so a cube api action of delete requires an update action that might make the pod 20 bytes bigger right and then during the process of deletion, it will get more updates that make that another 100 bytes bigger before it finally gets removed and the queue api server knows it's semantics.

G

So it knows that it's happening and can accommodate it. I don't know that I see how ncd does, but, but I think before it can be recommended even for people to turn on of their own accord.

G

There needs to be some sort of explanation for how that would work and then the idea of limiting the size of a namespace overall bumps into the key space issue that that I described where we actually use multiple key spaces to represent a single name, space.

F

Yeah I I will note, though, like we had that problem on deletion. Today, deletion can can result in objects that are a lot larger for other reasons, so it's just. It makes it much much more likely, and so I think, david's point's fair. I just I did want to highlight like there is no difference between a delete and an update. um Normal operation. Failing for update, was kind of daniel's concern.

F

So, even though delete grows like regular updates, failing would confuse the cube, led deletes, failing, would confuse the cubelet and then there is an argument that that could still happen today and so the the other thing I might say, is it's a bug if you can't go and delete an object by clearing finalizers directly.

F

So there is an angle like if we can, you know if this goes and gets done and there's something we test like that's a reasonable bug to accept on cube, which is an administrator in this scenario, must be able to physically delete the ncd object in a reasonable way. Even if maybe that's not the api.

C

Yeah, okay: we should probably uh time box this, but yes absolutely! You can have this happen today and it is disastrous.

D

Sorry, I actually I I work with civic and uh I may also add one more sentence that initially you know if we're going to do. This is going to be a warning on the entity side. So right, really it's not going to be the air right away and, like the error, is like something like very drastic. So most likely it's going to be a warning, for you know, administrator of ncd cluster to go like check with the uh you know, cube uh to see.

D

Okay, what's what's up what's going on right, so that's like the first use case and the error is uh you know it's gonna be some! You know it's gonna be issued only when it's like okay. This is really bad right and, like um you know so so, yeah just just to clarify that part.

B

Yeah, it's like there are multiple modes to run. This, like you, don't have to like directly go all in and say like uh just error out when you exceed the quota. uh Just give me a warning and then I'll see what to do like the admin can then decide where to go from your on. So it's there's a there's, a step in between, like going from zero to 100 like directly.

C

Okay, um we should probably go on to the next uh topic. Thank you very much.

A

H

uh Okay, so uh I want to discuss a little bit about the storage version. Migrator uh I've yeah. The question I asked in the in the notes is: shall we keep investing on this project? I saw david's comments below so it seems david is still positive, keep investing. So I guess then. My purpose is just to reaffirm what our goals are with this project and.

I

Can I just chime in briefly- um I didn't see this before, but I do have colleagues that use this thing and would be sad to see it go.

H

That's nice, uh sir who's speaking.

I

uh This is mike spritzer from ibm. uh We use it.

H

Okay, nice, uh okay, great uh so so yeah. So there there are two two different types of resources: storage migrator cares about, uh for one is the built-in resources and the other one is the crds.

H

Should I should I do a very brief introduction on the storage version migrator.

G

H

It's not about.

G

10 minutes, I guess maybe just go to the level of uh on request. It can do a read, write cycle of an entire type of resource and that, in effect, can update it to have the storage uh with the data that's stored, being a different version. That's what purpose is right.

H

Right so it's going to get a data. You know sdd updated to the latest storage version, uh yeah, that's the that's its functionality so for building resources. uh Currently, because we don't have the storage version migrator, we have to keep all the deprecated types toggle and the deprecated versions, conversion conversion code in the codebase in the kubernetes code base, but it seems that people are fine with this. So I'm wondering uh let's say if we, if we get storage version migrator uh to a ga state, uh what what? What will our next steps be?

H

Well, we go ahead, delete all the deprecated types stucco and the conversion logic. Just this desire.

G

It's a thing that I desire, because, right now, if you have an old version that you're transforming from it becomes impossible to remove inert fields, and it's also possible impossible to make certain fields that have always been defaulted required and those two things. If you dropped the old types.

G

Do you think that we could try to bound the problem of the storage version migrator, rather than have it try to automatically detect when a storage migration is appropriate? We can try to call the project complete once we have confidence that the api that does the I request, a read, write cycle provides enough building blocks for people to work. On top of it right most of the uses. I've seen have been people externally, calling it once they know, say their new crd conversion.

G

Web hook is present, they'll convert everything, uh and then they can remove the old version.

I

Yeah, that's our use case, for example,.

C

Yeah, I I think what would be helpful for me is um just what what is the list of things that need to be done to get it to ga level um right like like. I think people use it and it is useful and there's a need in the system like, even if you don't want to deprecate and actually remove, or even if you don't want to actually remove deprecated things like it's still useful, to have the uh storage.

D

C

A good storage version like a modern storage storage version, so um yeah, I think it's useful. The question is just how much effort remains and uh like if we had a. If we had a good list, maybe we could get some uh volunteers to uh finish it off.

H

Okay, yeah. I think what david suggests makes the remaining effort much little much smaller, so because currently we are still working on some automation api. But if that's not the goal, then, if that's just about the migration api, then we can quickly move forward.

C

Yeah, I think I think that'd be cool to like get what we have to a ga state and then the uh automatic triggering can be a separate uh follow-on effort. Okay, that's a good.

G

Suggestion so one of the names that I listed there is louis sanchez he is on leave for a little while, I think he's back in july um can can I have him get up with you when he gets back and bring him in on this conversation as well he's pretty sure the person who's been actually using it.

C

Thank you, but uh continue on um tim. I see you're on the call. Do you want me to introduce this topic, or do you want me to uh introduce it.

J

uh I can give a brief introduction, um so this came out uh of a um an outage review that my team handled when uh an admission controller that we manage the certificate.

J

Was updated accidentally and then, when that change rolled out it broke the cluster, because suddenly the api server could no longer talk to the the webhook backend.

J

So I was thinking about how we could build in a safer roll out procedure um so that a mistake like this wouldn't break the entire cluster when it happened.

J

I shared some ideas here that we can talk it out in more detail.

J

Another question that I wanted to kind of raise as part of this.

J

This was specifically looking at validating validating webhook configurations, but I think this kind of idea would apply to any cluster-scoped resources.

J

It's just that a broken admission web hook is more likely to break the cluster um completely than some other changes so to cluster scoped resources. um So I don't know if you want to bound this conversation to specifically the admission webhooks or.

C

J

C

No, I I think uh I I think it's we've discussed some related topics before uh and I don't know that anybody has had an amazing idea about how to how to solve this. I think the problem is actually pretty broad.

G

But before daniel talks about, like the other attempts, do you actually have a concrete idea, tim.

J

um Not exactly um so, there were some ideas, I guess what I read up here um was thinking about staging the roll out um different name spaces.

J

um So if we had uh sort of a controller on uh web hook configurations, then you could say you know, send admission reviews for certain namespaces to some set of, uh or I guess to be able to kind of gradually roll out the changes to different namespaces. um This only applies for web hooks that apply to namespace resources.

I

So there are we building and launched darkly.

I

Are we building in launch darkly? Do we need two.

G

I don't know what.

A

J

G

Know how that means yeah and I don't I'm trying to think of related apis.

I

Yeah, it's not exactly the same, but you know: launch darkly. Is this service that um you know anyone running anything can subscribe to it that they think of it as feature flags? um So you know you have control over what feature flag is turned on in what context and that's you know kind of like this, except that's the binary version, and this is like instead of binary, we're doing integers or something like that. But it's you know conceptually a very similar kind of thing.

C

um I think uh I think it's a little bit more complicated than just like.

C

Is this web hook working or not, because um there I feel like there's a fundamental tension between keeping the cluster available and keeping the policies enforced, and the people who are interested in keeping the policies enforced would rather crash the cluster than have requests go through that are bypassing policies and the people who want the cluster to stay available would rather have the you know if the web book isn't working, let's turn it off uh and there's there's kind of a fundamental tension between these two groups, and I see it as an arms race between these people and at the moment I think the policy side has is is winning because uh it's easy to stop your entire cluster with a bad web hook.

C

um The problem is, uh it's very difficult to tell the difference between a web hook that is incorrectly blocking requests and a web hook. That is correctly blocking requests.

C

So I am not quite sure that I've heard anyone articulate a good middle ground between these two camps.

I

I think kim's ideas are a little bit. Okay,.

G

I

To pick one middle ground or offer the ability for a given cluster to be configured in the appropriate way for that cluster.

C

I mean I it's it's a good question. I don't know if I uh would necessarily see it in this like like we could. We could say, like we already, let you configure your cluster. However, you like, and um that's great as long as your configuration is perfect and your web books never go down. um I I think, which I think is equivalent to taking the position of security, is the most important thing.

G

We do already offer a failure policy, so I think to some.

D

G

Already offer the ability for someone to say you know it's more important for this to crash my cluster and that's what fair policy fail means and if it's more important to be available, I can make it fail open. But I do think daniel tim presented a new idea um where the failure policy is still honored, but it you, you somehow use two configurations which is a thing we don't have today um right if I've understood correctly, that's what you're proposing tim.

J

G

J

One thing to clarify here.

J

To the back end, the the thing that's actually serving the web hook is out is a separate issue um and, to some extent like if you're running your web hook in the cluster uh hosting it in the cluster, then you can already do this today with a deployment and setting a good roll out strategy on that. So that's like. If you're changing the policy, you know changing what pods are allowed or not.

J

I would expect that to go out as part of the changes to the actual deployment serving the web hook, so this is specifically for that like web hook, configuration piece so the example I gave earlier was a change to the certificate that caused the api server to no longer be able to communicate with the back end.

J

So that actually, would I mean, I think, that's it's pretty easy to separate a failure to connect there like the um the certificate validation failure is separate from an admission rejection.

C

um I I mean I can imagine up a couple things we could do there like, like uh you could already today, um just instead of modifying in place, your web configuration duplicate it uh with the change name or something and a failure policy fail, open and call your web hook twice uh once with the old and once with the new configuration and verify that your configuration is working, that's something you could do today.

C

um If we wanted to get fancier than that, then, uh and like kind of automate every every change. This way, then the api object itself would have to like have a like a a trial or a right like an experimentation uh thing and remember the last setting in the current setting. um I don't know uh typically, if we're gonna do something like that.

C

It'd be structured, like you know, deployments and replica sets, or something like that, where, like a change, causes it to create and right, like we'd, make a web book configuration canary, canarier, object and controller, which would basically do the thing that I just described since your web hooks should be stateless and it should be fine too, and item potent and side effect free. Should be fine to call it multiple times, I don't.

J

J

um I guess uh yeah, I agree with you daniel. um I think that uh you could approximate a lot of this today um with either uh yeah duplicating the web hook with a fail, open policy or um getting creative with the um the match, match, name spaces and match resources, uh labels and probably you could even build a third-party controller fairly easily. That would automate some of this rollout.

C

I I would actually say like, even even if we think we wanted to build some feature like this in into you know, upstream uh kubernetes, it would be good to experiment with uh the general approach, uh with controllers and extensions etc. uh um Just so that we like like is it? Is it? Does it actually help? um Does it catch things?

C

G

One thing that was interesting to me when you described it here tim was that you talked about doing it per name space not per api server, and I guess that's because you want to handle cases where it's not an aha cluster. It's not one of the reasons for doing it. That way,.

J

um I want to be able to handle it for a non-ha cluster, but also to have some uh consistency.

J

This request, so this: if you're changing the policy, um you don't want like a non-deterministic policy decision there. I.

C

Think um I I I think both of those uh are not good bucketing mechanisms, because you can have a hot api server. You could also have a hot name space right like so like you do all the namespaces that have zero work and those those tell you almost nothing about the namespace. That's really important to you.

C

I think it might be better to like uh hash some fact about the uh about the request and- and you send like make sure that you're sending like approximately one percent of your traffic or something like that if we were gonna, have a canary um a canary test, no now that that that might actually be reasonable to add to.

C

To the api object right like you, could you could imagine a policy you wanted to call the web hooks, but you only you're basically testing it. So you could ask it to send like one percent of traffic through the web hook and then you could. You could dial that up up or down with an external controller that might be. That might be something that's reasonable.

C

um I think uh in an ha cluster uh without a bunch of work, um you wouldn't necessarily be guaranteed that every api server has seen exactly the same uh version of the web hooks um right like we don't currently synchronize that between them. So if you, if you want policy consistency, um you may have to wait like you know, 10 seconds 30 seconds or something for for the api servers to have all seen the latest version of webhooks.

C

Hopefully that's okay,.

J

You know it would probably be a best practice to not roll out changes to the web hook configuration at the same time that you're rolling out like a new back-end image or like policy configuration. um So maybe the consistency becomes a little less important with that. If you're, assuming that you know there aren't any policy changes happening, I guess the one exception would be if you're changing the rules um to what for what requests gets sent to the webhook.

C

Yeah, actually, uh I'm the more I think about it, the the more I I like the idea of like a uh just a traffic percentage slider where api server hashes the easy to compute aspects of the requests and decides you know is that you know mod 100. Is that larger or smaller than your uh percentage and if, if so, uh send it to the web, hook otherwise skip it? um That seems pretty useful to me without like trying to solve the whole problem, it seems tractable. I don't know what do you think david.

G

uh I think if you did that, if you offered that feature- and you had- and you said for well-behaved web hooks, that can be called twice, you can create a second instance of the whiteboard configuration to test your change. Yeah. That probably gives you sufficient building blocks to have a good way of doing it without just instantly crushing uh your workbook.

G

I think the slider is important. One percent might not be enough in small small clusters.

C

Yeah yeah, exactly, I think uh also uh you might want to specify like greater than or less than so you could do. A split uh like this web hook sends so you could be guarantee, or maybe just a range, so you could be. You could uh split traffic between two versions of the web hook, rather than duplicating all traffic to both uh webhooks.

C

G

Makes sense- maybe I I am not a fan of the idea of having an api server, try to remember a previous state of an existing web book.

C

No, no, no yeah, I mean yeah. I agree. That's not.

G

What I was saying.

G

So the idea would be then that I could create a duplicate admission. Web hook configuration- and I have one take everything after all, the other 99 and one take the one percent uh yeah. Why would I want that in this case.

J

C

um Yeah, I think it makes more sense uh if you want a 50 50 traffic split, but why would I want that split? I guess uh why would you?

C

Why isn't one percent good enough, um I don't know, I mean one percent yeah who cares if you duplicate one work on one percent of your requests, uh but usually people stage things stage rollouts uh in more than like uh more than uh two groups right, like you, don't I mean for some installations, it'd be fine to go from one percent to then 100 um for other uh situations. You might want to go uh point one percent to one percent to ten percent and then to everything.

G

Yeah, I guess I'm I'm looking at saying that most of the time when I think of of trying to do it fully staged up like a b and shift over it's because I'm replacing the image that is serving the same web configuration the bits of information available in the web book.

G

Configuration itself tend to be things that I either want on or off right like I might be changing a namespace selector or I might be changing uh like the resources and verbs and those don't tend to be things where I want to you know, phase and switch off. I think I think they're things like if it works great. Let's switch.

C

Yeah tim brought up the the cert data.

G

Yeah that doesn't look like uh I mean. Maybe it is, but I wouldn't have expected that to be a a sort of scale up and gracefully switch over. I would expect that to be like okay, this configuration works, switch yeah, yeah, that's a good.

C

Point, I guess I guess it's.

G

Your use case tim, like did you did you wish you had a way to say like bring it up in stages or just want to see. Does it work yep.

J

Yeah, I guess in our case it was a. It was an all or nothing failure, so uh yeah. If we had rolled that out to one percent of traffic seen that it works and then gone to 100, that would have been fine.

C

uh I think we currently don't write anything back to the status of web hooks.

C

um Maybe we should consider doing that like just extremely basic stats, because right now, it's kind of difficult to discover like if you, if you, if we did let you like, send one percent of your traffic to a webhook configuration just to test it. um It would be kind of difficult for you to tell if those requests were actually failing. All right, um we don't make it super.

G

Easy to discover so you're saying: choose a selection of the metrics and find a way to post that in status.

C

Yeah, something like that.

G

Complicated with h8 servers, I'm not against the idea.

C

Yeah yeah it does it. Does uh you either have to be like pretty careful about um like transactionally decrementing, your memory and incrementing, the the the object or you just have to report status from each api server separately.

G

I'm I'm open to the idea and I think that's interesting- to be able to report a percentage of uh probably admission, accept or reject and connection failure. Mode. Yeah that'd be interesting to see.

C

Don't actually uh reporting it uh reporting it separately from each api. Server could be pretty interesting in case there's a connectivity problem from one particular api server and what sounds like.

G

Well, so, what's interesting about it, you could use this to inform a controller if you wrote it into api status,.

F

Controllers can.

G

F

Of services controllers could ping metrics too, I mean like I, I agree in the sense of yes. This could be useful. I would say this is something we have explicitly rejected doing in other places, which is frequently changing. Numerical values that are metrics belong to metric systems, not to status objects.

G

Frequently might be in the eye of the beholder.

F

Yeah and it's I mean it again like because you're basically just writing a crappy service mesh at this point right, like everything we've just discussed here, is like a very, very stripped down bare bones: crappy service mesh and maybe like and again like that. Sometimes that's useful for very fundamental capabilities. I it has a pretty high bar for going too far on it.

G

So I'm putting myself in the shoes of or trying to put myself in the shoes of I'm tim and I've just had to answer this terrible call about why my my web hook went down and I want to write a controller to manage it and what data is missing for him to write the controller. What's he managing.

F

Sorry he's actively managing the percentage of traffic via a spec field on a web hook, endpoint.

G

I believe the choice overall is trying to roll out a new ca bundle for an admission webhook, and the method is by creating a duplicate of that admission: configuration having it with a fail, open policy and redirecting a percentage of traffic to it, and once he does that he needs to make a decision about whether this is working well or not. Working well, and it is possible to say well tim now.

G

You need to go integrate with your metric system and work with that to get the answer, or we could say, on a every 10 minute basis, an api server, we'll write out basic stats, and now he watches a resource.

C

There's ways of compressing it so that you don't have to change it frequently um like you could sure, but you could bucket it and do percentage right like effectively.

F

Though, what we're saying is that webhook api, the webhook api, is misdesigned for deployment and we need additional sets of things. I'm not disagreeing that, like.

F

Itself is the wrong api and there should be like we can look at ways of modeling the the web hook api differently. It would let it I'm.

F

I understand what david, which you're describing, and I feel like that starts to cross into you're, taking a bunch of ideas from a bunch of different systems and putting them into a cube object and historically, when I've said that I'd be like that feels like something that we would solve with the right system behind it, so that it's somebody else's problem like the web hook proxy. That is responsible for managing all of that under the covers, and it's not the responsibility. The cube api server like you could run a proxy for web hooks.

F

That has one set of credentials in the front and it manages them back in it's kind of a special case, though right, because you do need to rotate that surf bundle eventually. So I understand the use case being novel.

C

Yeah, I don't know, I think it would be a relatively simple change uh to do this um uh and and if you're, if you're, trying to manage this like, I think it is very reasonable for status to tell you like. Is this thing working or not like? Have we sent any requests to this web hook? uh Yes or no uh of those requests have any succeeded? Yes or no.

F

Summarization, absolutely I agree with metrics. No, so if you can frame this as a summarization problem, I totally agree right, like we already summarize status. So that part yes, it's just not metrics.

C

Yeah yeah, it doesn't have to be metrics. I I like you know you. Could you could bucket uh bucket it into like? You know, exponential buckets of like? Have you sent, how many requests have we sent to this web hook like none, some lots extreme right like.

F

As long as one request passes like haven't, you already checked the the certificate at that point. Maybe that's the question. I'm asking is a significant.

G

Bundle, that's like it works in. In our experience managing api services more than admission web hooks. uh We found that that one isn't enough, and five is too many, um but it all depends on. But a lot depends on where they're from right, when, when you're, when you're trying to decide, is this good or is this bad um being able to have the individual api servers? Give you an indication is significantly valuable. Like imagine, stefan dealing with the the problems of like why?

G

Why can't this one keep api server contact that one api service, and this is what's yeah. I.

C

Think a single, a single request working tells you that there's at least one uh backing instance of the web hook, which is configured correctly. um But it doesn't tell you uh that.

F

Analogy is very useful, like api service is the same as webhook config, which is roughly the same thing as what our ede tests have to do to make sure that a config change is propagated on all three api servers when you're running an h8 server configuration which today, you can't actually tell without doing probabilistic checking of hey does the thing the load balancer routed me to have this crd yet or api service, and that which is a set of bugs and like jordan, and I have been working with a couple people. So there is a.

F

There are three things that are similar here, which is I'm talking through a load bouncer to a bit of the cube infrastructure. Do we have a condition or a summarization? That accurately indicates that all of them have ratcheted is, and there might be three different approaches, but like that to me is a reason to go. Do it if we can find the thing, that's relatively common enough and or the thing that's common, if it's common or something that we can align to api service and web hook equally.

C

Yeah yeah, so I actually I'm looking at tim's message just now. I could think of other ways to do it, which is like. um Maybe we could let users specify a header or something on a request. That's like definitely send this request to some web hook and then you could make a web hook that doesn't select things and test it selectively with requests.

C

I I, regardless of the like the uh the network layer, routing requests to to web hooks, uh regardless of that there's still the api server uh layer of like which requests get routed to which web hooks. You can't you. You can't reasonably test that in a meta web hook or a network layer unless you're gonna re-implement that selection logic. um So I think it's reasonable that people should be able to test that selection logic. Somehow.

F

We have this in web hook, emission in the core infrastructure, which is we wait for the intervals like this is like this is the config propagation problem to some degree, um and we have service readiness for pod readiness, gates and stuff. So, like I guess, I'd be looking at those for inspiration and looking for the commonalities, and it sounds. I mean like to undo some of my poo pooing. I was very specifically reacting to metrics, but gating of readiness of the things so that you can determine it, makes everybody's life easier.

F

We have a bunch of hacks in how the actual web hook and mission plug-in is implemented that work around the config propagation problem anyway, on the on the on the server side of it and then, if you're, looking at three api servers, you have that problem in three-fold. So we already kind of have this. How do I make sure that the config is propagated so that I can say something interesting about whether it's working.

C

Okay, so uh we've as a good discussion, I think we're rolling up on time. uh What are our next steps tim? Do you have time and inclination to write up a cap with some of these ideas or um you know what do we do next.

J

um I would be happy to collaborate on a cap. um I do not have cycles in the next say in the 123 time frame to drive this so yeah. I wanted to share my use cases um and I'd be happy to see someone pick this up, but I'm afraid I can't drive it.

C

Okay, all right uh are there any other volunteers.

G

I was just gonna say I think, we're interested enough that if somebody else wanted to write a cap, I'd look at that too, but I also cannot volunteer to uh to drive it forward.

C

Yeah, I'm kind of in the same.

G

Way it would make time to review yeah review it not guaranteed.

J

I can flip this around my org a bit and see if I can find anyone who's interested in picking this up.

C

All right, so I I think, let's uh record this as like, it was a good discussion. We'll play it back for whoever is next interested and um uh yeah volunteer. Volunteers uh are welcome, uh it might be complicated.

C

um All right uh mike, would you like to looks like you dropped something on predicted. Yes,.

I

um I'm getting interested in edge scenarios where it's critical, in some cases, to minimize the amount of stuff on the edge nodes and I'm thinking from a coup point of view. The simplest approach is for an edge node to just be a worker node, not a one node cluster, um and uh you know the so. I was thinking through what goes wrong there um and you know you can do a simple experiment right. You can configure the cluster to tolerate the node being disconnected for a while that that kind of works.

I

um The fundamental thing is, you know we have this advocacy of uh doing a level based control rather than edge based and as long as you're doing level based control and the things running on the node are doing a little base control. It should all work.

I

The problem comes when you do something: that's not level based control. So, for example, if you've got some pod that uses the api server to store a log of interesting events, for example, that that's not edge based. That's not level based control, that's edge based, um and it seems to me that in fact, the the logical uh cases for where you want edge based stuff really is is logs.

I

um You know, maybe if we had some kind of generic support for logging, that supports disconnected operation and it might not even have to be in tree. um I'm, although I will note the kubelet does send events.

I

um So maybe we want something intrigued, but uh you know maybe it's uh definitely about supporting stuff. That's not entry, as well as the the cool and events um you know we could have something that tolerates disconnection um and life would be good.

C

um Clarifying question: when you said configure the the cluster I'm assuming you, you were talking about the like.

I

Health control.

C

Yeah the note health and that that will kill all the pods if the note is gone and turning that off or or significantly lengthening the time period right.

I

C

Just remember there's another issue.

I

Too, which is you don't really just want to make that a long time right? What you really want to do is say: okay, you know I I semantically positively want to tolerate disconnection, but once it's connected I actually do want to hear from the node. So there's there's actually another part of the problem here.

C

Okay yeah, so you make the controller uh bimodal.

F

C

F

So I make sure I understand, because dan you said something I didn't expect um so the note the node termination controller only deletes after the node is physically deleted. Pods that tolerate disruption via the pod toleration can tolerate the controller should ignore them up to completely arbitrary intervals and not reschedule them. That is a completely pod workload, controllable input.

F

C

Podcast, it's completely possible that I have old state on that.

F

Yeah, so so mike just verifying it or you're talking about pods that have the tolerate node unready condition with a time frame that is very high like hours or are you talking about.

I

Sorry, sorry, um I wasn't even aware of what you're talking about there is there something that makes uh pods sensitive to nodes.

F

The node not ready condition and tolerating the node, not ready, condition or taint sorry should allow you to avoid the getting deleted off of the node behavior. uh Okay,.

I

Right yeah now I think what I'm trying to say is the most direct approach is to not make the node go, not ready. uh I mean the node um yeah I mean so I guess this is a valid question. Do we want to make the note make the cluster kind of aware of the node's uh disconnection as an okay state, or it's just it's. It's ready, everything's, fine. It just hasn't checked in in a long time. So.

F

You want to change the least interval on a node, but then, when you bump the amount of disconnection you tolerate, you don't want to lose the rapid uptick when it does come back, because that is something that would be a sig node kind of ask like we want to have us. We want to tolerate at least disruption for a long time, but then, when it comes back, we want to rapidly bring it back in, because the lease gets there's like bimodal as dan was saying on two sides of the lease.

I

Right well, there's multiple issues here and I'm just exploring. So um the thing that I was thinking when I put something on the agenda here was um apart from all this maintenance issue, there's a semantic issue, which is um you know as as you're one of the the kcp people right. So one of the the you know. I very much agree with the first observation in kcp, which is the kube control plane.

I

Is, you know a great way to build distributed systems in general and I've been doing that and having great fun and want to do more of that, um but I want to tolerate a node being disconnected, but if I've got an app running on the node in a pod, while it's disconnected, you know what what goes wrong if it's a level-based, app everything's fine it'll learn about new levels, new levels uh coming in from the outside world when it reconnects and it can report the current level when it reconnects everything is fine for level based logic running on the node in you know, in apps, on the pod on the node, the problem is when you have an app on the node.

I

That is doing something that is edge-based and it needs to report or do something on an edge. You know it can't do that, while it's disconnected so you have an.

G

App that is reporting something not something deployed for the app's purpose of reporting, but it's actually reporting via the cube api servers for some purpose.

I

Right yeah, I mean imagine something: that's doing some kind of compliance checking on the node and any uh violation. You know it wants to post a notification to something that's listening to notifications and it might do that through an api object. Why.

G

Wouldn't we expect such apps to keep track of their local state and manage distributing that knowledge when they reconnect at an app level? Why would we that's right.

I

That that is what I'm doing, that's what that's that's the direction.

G

I

Okay and I'm thinking about so as I think through that right, there's basically two cases: one is the app reports, its state, it it's a state-based app. It's a level-based, app, okay, and in that case it just works. We're done. Okay, the other case where we're not done is when the app needs to notify somebody or log something that's where we run into trouble, because it can't for now it has to cue those up buffer those up and deliver them once reconnection happens,.

G

Are you proposing a way to change that.

F

Maybe I'm misunderstanding: what you're so mike? I I kind of I and I it was funny because, like we've had a number of discussions in a couple forums on how the cube node could make temporary disruption from api servers, more amenable to cubelets and the workloads are the um the supporting infrastructure on the cube on the node. That's required. So like the networking q proxy et cetera, where you know you could imagine a local api server or something like that. That's really not workload! Centric! That's about letting someone write it.

F

A csi driver that tolerates disruption of the node. That would be kind of a different class of problem than maybe what I can't tell whether you're asking you want your app workload to leverage an infrastructure like that. Or do you want your app workload to be uh not impacted by that and you view it as cube's job or it could potentially be cube's job to help. You mediate that.

I

um I was thinking about the case of an app that has the need to um you know, do something in response to an edge. uh Basically has the need to log something that's the kind of scenario that occurred to me, um and so I was thinking that you know the api servers that we have they're great for doing state-based stuff right by design and not edge-based. Stuff logging is, in some sense an edge-based thing. It's about just accumulating these log entries.

I

So um you know if, if thinking about kube is an application infrastructure right and I'm motivated to think about this, because I know the kubelet logs events right, creates events as basically means of vlogging right. So that's why I'm tangling it up with coop, and maybe I just shouldn't. Maybe the fact of the matter is kublai, never logs anything while disconnected, in which case, maybe we're completely done, and it's not coop's problem at all. But if maybe it is coop's problem to support things that want to log stuff, while disconnected.

F

Yeah, which you described as kind of the while disconnected from an api server, the ecosystem of cube-like things on the node, could do things that then tolerate disruption, which is one cap and then the other is. I want to use that in a general purpose.

J

So you're kind of looking for a somebody's making noise, okay.

I

Horrible echo and I I was hoping to complete- to get some clarification but you're understanding the distinction.

E

We can just take this to the slack. I don't think it says. Okay, thank you and we're out of time anyway,.

A

Yeah, we are on time uh I hate to cut the discussion, but we need to respect the boundaries. um Okay, I will. I hope we can continue discussion either next session or slack um we'll upload this later today. Thank you, everybody for joining. I hope everybody stays safe and see you next time.

C

Thank you, bye.

C