Kubernetes API Machinery Special Interest Group, 11 Mar 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG API Machinery 20200311

Description

March 11th, 2020
- [micahhausler] Need for graceful handling of storage level failures on individual objects (#69579)
--We need a document (probably KEP?) listing the alternatives. Need someone to volunteer.
--Add a metric counting deserialization errors on read paths (Owner: micahhausler, mentor: ?)
--Add a metric counting validation errors of the *old* object during an update (Owner: micahhausler, mentor: ?)
--Add better logging when an object can't be deserialized?
-[lavalamp] Need for more guidance around modifying APIs. Need to revamp current documentation in light of CRDs.
--E.g.: https://github.com/kubernetes/community/pull/4571
Discussion.
--A tool evaluating an API change for safety might be good.
-[fedebongio] Reminders
--KubeCon EU was postponed
--Open SIG API Machinery bug/pr triages twice a week (Tue - Thu)
--Agenda closes the day before the meeting (Tuesday afternoon PST)

A

Recording hello, everybody thank you for joining us today. This is the C API machinery. Meeting from Cornette is today is March 11 2020. We had some interesting items in the agenda, so I guess we will start the first. One is mica indical.

B

A

How are you so? Maybe you want to start talking about your item or first today, sorry.

B

Yeah so we've seen a handful of times is.

B

So what we keep this going on is that there's a bit clip happening, post, post, API, server validation and on on right and.

B

It's in the term, not caring, where that happens with that memory in API server or a CD or network, but it's looking like he one gets booked our one resource or as it's get stored, and so the next time the object is read, it feels to be serialized and this problem occurred when this problem occurs. It causes gets to that single object or like gets on that object type for that namespace or it's not a stock that all gets on that height to start returning a 500.

B

So it's particularly problematic when it's like a node or it just things like scheduling in operation. How often is happening? Yeah we've heard we've seen this happen, probably four times in the last.

C

Year and a half so can you tell us like it seems really unlikely I mean I'm, not gonna, not gonna say that it couldn't ever happen, but yeah are we pretty sure that it's not a bugging code somewhere or is it always our one I mean it's.

B

Not always our one no can happen in different object types and look different this in the object new.

D

B

In the bug right, the other thought about this is that this. This is only really we've only caught this on when there is a seer, D, serialization deserialization error, because it obviously belted to serial I actually realize and if it were to happen to say in like config, Mac, beta or a label key value. That's not that so passes, validation, I, never noticed yeah exactly so. We don't really have that that firm. We can't really get an accurate count on the other times has happened up, we haven't noticed we for AWS.

B

We just released this week yesterday, encryption of sports secrets, so we sort of anticipate that this.

B

The observed occurrence of this could go up with that because if it were happening say after the encryption occurs, a bit flip on an encrypted value or the data encryption key and the whole secrets gonna, be you know the destroyed and affected not destroyed and and gets to that just secrets in that namespace or all secrets would start starting to so yeah I I kind of wanted to like just kick off the discussion here to see like I mean it again.

B

It's not something that has occurred and caused it choose very frequently, but if, if there was like any thoughts in make engineering around invalidation or any fallback, behavior yeah.

D

It's it's! It's actually very troubling for this to happen, because if it's like a ramp problem, there's actually no intervention on the right path. That is gonna be successful in detecting this.

D

Just just just if you detect in one place that just means they don't get instances caused by some other place that you couldn't detect like even having the serialize value won't help, because it could have happened before you before, like in memory and the in the stored object before he realized it and yeah. It could also happen in sed Easter stuff and ran before it. Writes it to disk.

D

I could happen at many places in the stack, so I think the only the only intervention point we have to like keep this from, like we can't keep it.

C

From I, don't think we can stop from happening. Okay,.

D

C

You only can tolerate it. Well, choose have fail like so so things things that I would don't like I, don't think I want a generic list to appear to work and just not return all the data I, don't think. That's a good outcome! No.

D

I I, don't think it's a good outcome for you to get a 500 and not be able to see anything in the collection either.

D

Finding a middle ground I think the ideal outcome is like it's corrupted and it's obviously corrupted, but it's a it's an object that you can see from the API and delete and recreate okay.

C

Well, mmm good I don't want to return a mostly correct response, because I think people would accidentally use it. Okay,.

D

So I party agree with that, but also since we're only going to be detecting things that make it impossible to deserialize we're, probably returning partially correct responses already.

C

Okay, well, I can really only do much about the things that I know but like so if it happens infrequently. What's the recovery path right, so I'm thinking about a recovery path, I'm thinking that that recovery path is either gonna, be a delete which could have some pretty significant consequences or that recovery path is some sort of manual direct right to @cd.

C

Or could we actually do it through the API server I'm, trying to think where there are as long as you didn't use, apply our right path doesn't actually read the old object right.

D

Yeah yeah I think the recovery is gonna have to be a delete. Even that might do an example.

B

As opposed to status, yeah well.

C

I mean that's yeah, that that's where you found a Aaron.

D

We're worried don't think.

C

About a recovery path, yeah.

D

Our goal is: is that a user I think our goal should be as that somebody without administrator permissions on the cluster can recover.

C

But our delete path requires a get in order to do finalizer access right.

D

C

D

A straw man might be the idea might look like when we find an object that can't be serialized. We construct a wrapper object of a different type that includes as a field like the raw, the raw store advice that couldn't be to serialize and then and then return it always yeah yeah. But.

C

Then, how do you, even if you had that, how do you can't.

D

E

It yeah well, we destroyed.

C

The schema, but but even if you had it, how would you have an unprivileged user, clear the error, then, if they can't delete it and they can't no.

D

We'd have to we'd have to make delete yeah. It's really another.

F

Option so then another option would be we could we could and I. This is I hate, saying this, but I'm gonna say it and we can always refine it down. We could do read repair which, if we can do serialize it, we write it to a new object as you're kind of proposing and that's just a new resource that can be viewed by admins and they can make the choice.

F

So it's the same as you're kind of suggesting, but when we read it when it cannot be just be realized, we attempt to write it and then delete it after we've written it. We'll.

D

Walk them around, they have to be a new like a new resource. That is just like a message or something, but.

C

We wouldn't allow a regular user to access that, though right.

F

And it would be I guess the one where he would be like secrets that get corrupted. Would you have to encrypt them, but like in theory, you do the read repair, you move it to lost and found, and you have a controller that once they reach a certain age, get automatically deleted or cross a certain number. And then you alert based on how many of those are getting created. I do.

C

Like the idea of having an end user able to fix it, if he sees it like I, definitely want the cluster, Adam he'll fix it. He clearly can if we can give him the key, but.

E

So I wanted to ask this sounds very similar to the conversation and say Gotha while back about what do you do when you're kms can no longer decrypt a secret and they can't do it because you accidentally rotated the key encryption key. So it's never going to be able to do it. But your cluster is completely bricked because nothing can list secrets and everything is gone.

E

Like the consensus on that one was the API server is not responsible for storage level errors relatively yeah.

F

But I mean except this biffle, it happened bit. Flip happened on a right to n CD with an encrypted thing: you're not recovering any of it, so the argument might be. We do Lost and Found for everything, except for secrets, and we write an empty entry in secret. That says: hey you had a secret. It's gone now, sorry about that.

E

But how do you know for sure that you've actually lost it and it's not like a canvas.

C

Well, so so I think the kms provider difference. Is that, like the use case, there was a cluster admin who was managing these keys? Did the wrong thing right deleting.

F

All of your NCD stuff off disk, you did.

C

Right I mean really yeah the this, though there's no clear like there was no user action, no, no user. Actually, no cluster admin action that led to this I.

F

Think here is it like, so breaking this down to its fundamentals? There is nothing that we can do that will prevent a an object getting corrupted on its way through our system to the remote side. It can happen. We can just reduce the probability to near zero, and so usually in those scenarios we would start by saying what is the when it is corrupted.

F

Can we detect it and give you the crit the clear action like because we could go like add, checksums and all the layers and we, the checksum, could still get corrupted so there's a certain amount of like I, don't want to I, don't want to patch I want to say like here's the mitigation, this is a database databases get corrupted.

F

Although it would be nice if we can pass a checksum down to a CD and have that CD do a double check, but it.

D

Would also be very nice if at CD itself compared check sums across the different replicas, which it does not do that.

F

Would actually be a better short term like if we were like independent and whether we fix this, the data layer should have the most checksums and productions working its way out from there. Well.

C

No, this was a PV right, so so we said the action is delete, but what if it's a PV, do we still think.

G

There is a check that could be enabled by flag. It would check the peer and see if there is corruption happening, but.

D

It doesn't happen continuously on every right. Correct I need to check the details, I believe it doesn't I know I described I, think Jo and I had talked about a system that would do that. I, don't know if he's written it up yet yeah. It certainly.

G

Needs some improvement. It.

F

To your point so like effectively, what you're saying is that, if we lie about the list being successful, it's no different than if we silently dropped an event on the watch stream or the informer or Delta FIFO was broken again, because.

C

No, that's not entirely true, because what would happen is like. If you did a realist and it just disappeared, then every controller would see that as oh, the Delta here is. It got deleted. Yes,.

F

C

F

It's the it has been corrupted being indistinguishable from it doesn't exist. Anymore is something that we did not take into account in our design, but our entire design is predicated on getting an authoritative list of all the things that you have.

F

Now the data would still be there right until the next time the PVC controller ran and until the cleanup controller ran and purged the PV well.

C

So what I'm, what I'm thinking is like like finalized errs, are important. In some cases, right people rely on a finalizar running when it gets deleted. Do we need to find a path that allows them to like.

D

We might need to repair metadata right.

C

Like so some of it I understand, we can't we can't replace right like if a if a bit flips in a secret, we're, never gonna, know we're, never gonna be able to fix it and the person who's messing with it won't know we're be able to fix it either, but.

F

Be able to repair all of the key metadata, except for youíd that's guaranteed to be correct. So the problem is: is that without you, it'd being correct, you have no ability to.

F

You have no ability to guarantee that, before after relationship on same-named objects, which a number of controllers do depend on so like I, think garbage collection like if we tried to say like oh, we got namespace name in our V garbage collector would still go and delete you. So.

D

It comes to so it occurs to me that there's there's two sorts of corruption that are possible and we might be able to do different things in response to them. There's the sort of query option that just changes, the the data, but the thing is still D serializable and then there's this sort of corruption like the in this bug, where the it can't even be deserialized, and it seems like the can't, even beat a serialized thing.

D

We actually could do something about by hashing or check something be like this would be super expensive, but imagine API server, C realizes the thing DS. He realizes the result, checksums the the the thing that that desk utilized and since the object and the and and the hash off to the stores layer right like that.

F

You know actually dance funny. You said that because it was like what are the odds that something's gonna make it d serializable it's very low for it's very low for protobuf and it's actually higher, for these are like the two that came up here. These are very rare because these have to flip the bit in the kind and api version which is like the least that's like a tiny fraction of the object, so the actual failure rate is much much higher yeah. Well, there's. Actually, the other one changes a quote to some.

D

Other character and jason has a lot of quotes in it, so that's gonna be a little less rare than this one but yeah. I agree that the the true failure rate must be much higher well.

F

The right rate on json objects is much lower than it's going to be for four core objects, which are all important us. So actually, that is a really good question. Are the.

B

Stored and protobuf are they supportive, but the json, the json that was being stored, looked like. It was a field on the the the status right so that, like it, was json being embedded in yeah.

F

It's object like let's.

H

Did an assumption there that most of the traffic is for core objects.

F

That's not necessarily so in every production cluster I've ever seen. It's like 92 10, even like crazy clusters using crts insane. It's still a fraction of the write load. Okay,.

H

F

Have some colleagues designing clusters? You have not seen those people are crazy and I'm, not saying that they're not valid I'm, just saying like production clusters today, CRTs are just not mainstream enough, and if there are people doing it, it's not the right. I'm, not I'm, not.

D

Saying it's not about saying.

F

Yeah I wouldn't say.

D

That that's universally true I've definitely seen clusters with often into misbehaving third-party controllers, C or D, but yeah I definitely seen how.

C

Expensive is it to do something that we can actually like at least have a cluster admin look at for relieve like so so yeah. We could try to find something better later, but presumably you could get some improvement by name of the object right all.

F

I meant to go in here is I was gonna make the argument that at least DC undi serializable errors are the rarest ones in likelihood they're less than their minority at best they're, probably a small fraction of them, because the odds of a bit flip into any other data field like changing false to fats or whatever, like those sorts of changes in strings and binary data or framing data and perturb off, are going to be much more common.

F

If we only target the do, serializable case doesn't mean we shouldn't have an exit case for that, but, like the argument might be, if this is happening enough, that we're noticing and deserializing it, it's happening a lot more on on a regular data. Corruption such that the argument would be. We should tackle both sides of it, not just the deserialized side, yeah.

D

Okay, I, don't know if I buy your argument about the probability but but I think.

I

A consequence by the yeah.

D

I think I buy the conclusion.

F

And David I think you're you're right, like this is kind of like a hard fault, but at least it faults correctly, which is nobody's pb's, get deleted today on the deserialization error, having some way to free that, having that be an alert having that be something that's like I mean it's no different than a node hard crashing or your single master going down like it's rare.

F

It should be more rare than your master going down like 1% of nodes that I've seen in the wild are down at any one time the odds of a bit flip here, causing a node to go down I. We could probably get some data from the real world if we had an alert, but in general we're just trying to get to a point where we can say you can recover on the other side of it.

F

Putting in some mitigation that forces a reread or rewrite, or a transaction error, probably cuts the total probability down to again, where it's less likely than all of your machine spontaneously. Combust.

D

So I think we're knock. It I think it's clear. We're not gonna have as quick or simple fix for this, so we probably need a cap of some sort or a design or something like.

F

That so is there. Is there any reason that you couldn't have a sub resource that lets you get raw bytes on every object and we yeah.

D

I was actually yeah.

F

D

Wondering if maybe I don't.

C

Need that delete I, don't know that I agree that delete should be. These were accessible, though, like a cluster admin sure, but.

F

La bytes should only be cluster admin, accessible I'm, saying like it's not outside the realm of possibility that there is a sub resource or existing structure, but yeah. It needs a cap. Yeah.

D

Yeah, a raw so there's two problems: if you want to go that way, you have to help people find the one that is actually corrupt, which I think you cannot do right now, because list is broken and then you have to let them delete or change it. So I think the a special verb or a sub resource with raw bytes can let you do this? Second, no. It's really an awful hack, but yeah. You.

F

D

F

Well, it doesn't even have to be raw bytes per se, but it could be the representation on disk and that's no different than reading the object that we have today like we don't have any hidden data. Maybe we do in the future. We should keep that in mind, but certainly so the error. So we have structured errors. I would expect list where addy serialization error occurs to return the first 25 or so on. These serializable objects as a cause with the right structure. That's why we put cause in.

D

F

It actually do that. You know it could right.

C

So without changing the schema, we can make a human, readable message.

F

We can put, we can put. We can basically define the message or a reason as necessary, which is we could give you something machine readable. So.

E

Now I don't know in this specific case, but in the sed can't or kms can't decrypt. We send you back the key that we couldn't decrypt from sed, so it's not great, but you can't figure it out all.

C

E

C

That one was us in the km/s provider now I guess: I wanted no okay, yeah.

F

So if we tried to decrypt sorry, if we, if we failed, we would just accumulate until we hit a certain number of errors. If we hit any errors, we returned a new status error, which is like probably we get the federal code it is and the causes would be something that's roughly machine, readable because I think you need something that's halfway between. Maybe we just say like the message will be machine and human readable and we won't change the format ever or something like that.

F

We put a été test in that prevents regression so that you could automate this at a large enough scale. Yeah we cap the number of causes, so we don't blow up the objects. We definitely do return in a form that a human can access. So it could be like I hate to say this, but a message could be like the metadata, only JSON blob or something where the reason could be because we say reason is defined as machine readable today.

D

Yeah so I think the action item here is a document listing the alternatives, and maybe that needs to be a cap or something who wants to own that I also like what is the just. What was the return on investment in this effort is. This is a pretty obscure problem.

F

So I think not obscure. I would be willing to put a match, so we should probably put a metric in and then get it out there in the wild, and then we certainly would like start rolling that up to our telemetry systems and we could get a better feel. Those people pick up new versions of openshift or whatever like like. Do we have a statistically significant enough sample size and I mean I'm I'm, pretty sure I would know about this.

F

If it had happened in one of our clusters, I have not seen a customer report for any of our customers either. Do.

J

You think this to include the HDD version information, because it's not included in all of the columns in the box, and it can be yeah. It happens for the older version of communities and their Huff's older versions of it.

D

Yeah, that's that's. Definitely a reasonable thing to ask for yeah. Also, the the title is misleading, because this isn't necessarily at CD. We have actually seen and fixed several different kinds of corruption in a CD, but none.

F

Of them were like this I I mean if I was gonna hypothesize. Where does this? Where does the data spend the most amount of its time like it's? This, the risk of a bid flip at the disk level on the NCD right path seems low, but like faulty memory or something could certainly do it more on a particular n CD server, and so maybe an open question would be just getting.

F

The metric in place would probably be the metric in place and rolling it out, like would probably be a good first step, no matter what not no kept required like. If we get deserialization errors, we should increment a counter which Goods these realizations with the resource or something, and even that would help us maybe make a decision about how far to go down the next step.

D

Yeah I agree: a metric is always a good place to start and that's relatively cheap.

F

It is funny cuz, it's like this is the one where it's like. You can imagine all of the terrible things that exist out there in the world. A lot of the flips are gonna, be inert, and so it could be that the incident rate is just low enough when you have a one class system, but then the people who are most likely to be impacted by this for us of the people who either don't report it or it the system just- is slightly degraded.

B

And they don't know, we've only caught this on customer or even like issues a handful of times, and it's typically been like two or three days after the fact that we've actually like gotten to it, and so it's like far enough back- that we don't still have a TD ball logs and since just internally like we have much better at the business unit, has much cheaper, Reitan older hardware, then like new hardware, which all customers want like there's incentive for us to use, not the latest and greatest. Partly so like. That's, probably where you know.

B

That makes good sense for where the source of the problem yeah.

D

It need to be a ram or a disk or something else, Network, probably not network I.

F

I gotta believe that, with some of the stuff that's out there in the world, we're definitely gonna at some point. Someone's gonna report this bug because they have old bad hardware. That's susceptible to this. The alert might actually be like what would be the threshold like if this comes up so roll this out would be learnin. We put the metric in and then everybody treats more than zero as a critical alert.

F

How long until we see the first one, you know, given the rough you know probably say: there's like five six million cube clusters out there. How long before we hit it yeah.

D

Yeah that seems reasonable to estimate its importance by how many times people.

D

Report it to us, we can even like put this issue number in the in the metric name or something yeah.

F

And- and it worse like from that, like we can estimate the actual occurrence rate based on the probability of like we just look at the average set of messages. How many that bit flips would cause a D, serialization error and we can get an upper bound on what the truthfully impact is yeah.

D

We can yeah that we can measure empirically yeah, I, think it's so I think a DC realization error may not happen that often, but it seems like a validation. Error is very probable like, especially if it's on a pod or something.

D

You might make the object might become unreadable even if it is still be serializable. So.

F

We would have a metric, so we already tracked validation errors in the request count field.

C

You can't really have a metric on I mean you could, but it would be meaningful. Well.

F

I know I mean I would might argue. There's a good like I'm. Just think about this, like I, could go to the Prometheus mix in right now and say. If you have a rate of validation errors that exceeds this threshold, something is fundamentally wrong. Either you got a controller out of control, but the.

D

The checking for validation, errors of the old object on Rights would detect other problems too, but it would probably detect this as.

F

A symptom validation failures of above a certain rate are not human inflicted and thus are a broken cluster or broken controller that somebody has to go deal with, but.

D

If eclis, this would cause the old object to fail to pass validation in an update sure which shouldn't be possible if we're doing our job correctly. Yeah.

F

It's good to know that, anyway, that's actually a good man, yeah yeah! That's my.

D

F

Exactly are there any other classes of metric here that we don't track that we should which are? We haven't some evidence that we have screwed up, because we've definitely screwed up on adding new fields so like the metric you just proposed Dan would have actually failed with the dual stack issues going on: that's actually how it sampled. For us, the old object was actually non-validated old. Dual stack was turned on and you came in so are there any other impossible things that really track.

F

They certainly I'm removing the last finalizer as removing finalizer just come up enough. That's another class of problem. That's somewhat like this, even though it's not a corruption case. What.

C

Do you mean by removing funnel is.

F

Something that is sat around with the finalizer in the delete state for far too long I.

D

Think that is, but that's a symptom if the old object doesn't pass validation right. You can't.

H

Well, what's far2 longer, what's far too.

C

H

C

Would probably say if it takes five minutes, that's weird! If.

H

It makes an hour it's well.

C

You're making assumptions again, those of us who use.

G

Kuba API machinery to build other systems. Some of us use finalized errs.

I

H

I

Kinds of consistency, so that's.

H

F

So, like I would say that the fair point but I would say the 99% of cute clusters actually want that alert and if there's one percent that are annoyed by it, they should be able to remove it. But I wouldn't argue that, like, for instance, pods sitting around in termination beyond the grace period should be an alert today we don't actually have it I, don't think. That's right! That's like an example. It.

C

Is they should.

I

Be type out Pacific.

C

Yeah I mean I, don't know it's pretty unusual to have a system where you it takes an exceptionally long period of time to issue a delete.

C

I

Know I'm thinking of scenarios.

C

Want to have an alert to let you know that it's down right, like there's an action you need to take when an aggregate of API servers down and it's not bad.

J

Or administrator doesn't have like all the visibility and it doesn't really care about.

J

H

Have examples here, a regular kubernetes right think about pod in PVC, so one can delete the PVC Walla pods using it. The pod could keep using it for a week. I.

C

H

I would also know it's.

C

An exceptional condition that you want to know about right like under which conditions would you not want to know if you have to take an action you want to know if your data is at risk of being corrupted, you want to know.

F

This probably is a discussion move to the. We should probably actually go to the monitoring sig on this one and queue up a discussion, because I think that this is all stuff that probably are good guidelines, that what I've noticed is that the monitoring sig has a good like broad domain expertise, but they really often need specific domain expertise of folks, like API machinery and no team to go and say, hey, there's, no alert on there's no alert or recommendation of alerting on how many cubelet evictions are happening.

F

It won't apply to everybody, but most of the time. So maybe we should put a whole subsection on that and talk there.

D

Yeah as long as the finalizar isn't a like, it doesn't prevent deletion it just at least as long as as long as it's being used like that. I think it is reasonable for some objects to take a really long time to delete, but I also think that that's exceptional for most users and about it.

J

It also affects the garbage collection performance. Just keeps on going yeah.

D

Yeah, it's a good point.

F

And the garbage collector should have alerts of it like there's, certainly and honestly, we're talking about two different types of alerts here, so I think the one we started with was a fundamental like this is just broken and then the second class is the broader. This is something that most cute people would care about, but the domain expertise is here. We couldn't dictate that everyone has to and and and to be clear, kubernetes.

D

Exports metrics it's up to system administrators to turn those into alerts.

E

C

I mean the monitoring stack, does have recommendations as I recall and.

F

Honestly, I, don't know how many people outside of the monitoring stack, but my experience is. There are a vast number of alerts in the Prometheus mix in used by a large subset of people, not not a complete majority, but like a large subset. They are vastly under cover and that's a good plate like it would be better to have at least a canonical place. We'd be like hey. This is weird with a reason coming from a domain expert in one or two places than in zero places, which is the way it is now. What.

H

Is this Prometheus.

I

I

H

Fyi I don't know about this Prometheus it makes it was this big.

F

Instrumentation maintains with the community a recommended set of Prometheus alerts that provide general cute guidance and as a place for people who have general cube expertise to go. Recommend rules, certainly there's a lot like the NCD maken. Maybe the EDD mix-ins for a cute cluster recommended number of tube issues and they're not like perfect, but it's certainly better than nothing, which is what most people go to when they look for the run books, yeah.

I

Okay, it told me helpful.

F

If you're actually using for me, though, and every way else, can go, look at those Prometheus metrics and go do it to whatever system they want. It's just the argument of it's better to have a place rather than no places and I think the alternative is no place right now.

I

F

K

So, in addition to the matrix we're planning to write, so we have enough margin when this is your annotation areas,.

D

K

D

K

See based tracing tracing errors, around serialization position.

B

C

I also have errors and gives back at decent, I'll, say decent. If you knew cube, the message is not horrible.

D

You know we probably should like this it's worth talking when our object can't be deserialized.

D

Okay, shall we move on from this topic? Anybody want to pay more. My ticket is machine-like, be reasonably discussed. This yeah.

B

Yeah I think so I mean I think that what we really like the yeah metric would be really great, especially if we get back for it to a few older versions so that, like it's, it's easy to ingest. That would be super helpful. Yes,.

D

Yeah, it's something you could you could take on or for fine folks to take on yeah.

B

I'd be happy to do it. I would probably need some someone self to look know exactly where to look. Is my personal data today I'm sure it's.

C

Down in API server storage.

G

H

I ask one question: following up on something that was touched on about sort of the status and expectation. If you used to just see our DS I know, we've been on a general programming of making them generally useful. I. Think I heard a remark earlier says: they're still, not there and people shouldn't plan on using them, as some people would say in earnest or in anger,.

D

Your interpretation of what they said, yeah.

C

I think Clayton said that the vast majority of clusters we actually see are mostly made up of core resources that there are some CRTs that have a lot of instances. But just when you look at an overall fleet of cube clusters,.

H

So I raised specifically the idea of building a system using cube, API machinery and CR DS. You know so it's our own system, it's not exactly kubernetes and the colleagues were planning to use a lot of CR DS at a high rate and Clayton said ah they're crazy. If I press call correctly yeah.

C

So there's some storage issues like if you have high volumes I, don't know that I'd want to do it in a non binary format. There are versioning issues, we're versioning does work, but it is fairly slow right since there's a wet book. It gives you the power, but it is not super fast. You.

F

Can do anything you want it's just gonna. It's gonna have a lower boundary than the existing infrastructure.

D

F

I

D

Two outstanding issues on CR DS that we should consider addressing one is that there's no binary format, which makes them heavier than they need to be. The other is that there's no way to reference other types in the system, which means that the definition is very verbose when people can't put as much in there as they want to I agree.

C

On the former and I'm not sure on the ladder, okay,.

H

This is exploding I. Should let me open a discussion on the email it made me agendize it for the next meeting. Yeah.

D

D

Let's see, let's go on to the next topic and I'll try to keep this short since we're low on time. You try to keep this to like 10 minutes at the last City architecture meeting there was a big discussion around guidance for adding skills to existing api's and I said I would take it up here and didn't have a chance to do my homework like I should have so you get this in insufficient item and the agenda here.

D

Tim Hawken added a or suggested a change to the API update guidance, which I think we're not going with. If you look at the comments that Jordan and I have left there. um Basically, where do you say this.

H

D

This is in the API changes or API conventions, API changes, markdown document, but.

H

Where is Tim's suggestion? It's.

D

Been a PR I can I can find a link. Thank.

C

D

I think there's there's at least two high-level problems. One is our documentation is not arranged in a user-friendly format. I.

A

D

What people need is a flow chart. That's like I want to make this kind of change to the API and they follow a flow chart and it tells them what they do at the end. That's not at all how our documentation is arranged. It's a list of rules, so you have to read the entire twenty page document to figure out what applies to your situation and what doesn't? So? That's that's one problem. The other problem is that the documentation is all written in terms of the built in types, but more and more in the future.

D

People are writing series and evolving securities, so we need documentation that is clearly applicable to both kinds of api's.

D

So that's the problem, statement and I think the what we're what would be nice is, if somebody has lots of time to like fix the documentation and make it easy to easy to to read and understand. So if anyone wants to volunteer for that minor tasks, let me know we'll.

H

Need reviewers for documentation PRS, yes,.

D

Yeah we'll see talk amongst yourselves for a moment, while I go find Tim's PR.

D

Of course, I can't find it in my email right now. Does anybody have thoughts on how we can make this documentation? Well? Does anybody think our documentation is correct or clear? Yes,.

C

Are you are you looking at this to try to find in and make automated checks for all the things we can make automated checks for, or are you saying well, CR T's? We don't really control the schemas, for we should provide some sort of give us an old to give us a new, and we tell you whether we think it's good or not, or are you looking at it for I.

C

Can see, or are you just looking to like, update, Docs and hope people will read them.

D

C

Don't object to updating Doc's home people. Read them I mean I helped write some of these, so I I.

D

Think there's some tribal knowledge in this fig that other people don't have and like it's, not obvious. Yeah.

H

I'm sure of it being one of those other people.

D

Like in Tim's PR, we pointed out that there's now a dry run, you can do a try running to create, which you don't know about it, and if you don't know, this is a new feature and it mostly solves problem that Houston thought.

D

The problem in that PR is a controller, is taking a subsection of a specific group like a knob extent, Park hashing it using that hash value to detect whether something changes in the future on the problem is. If field is added that doesn't change the intent that still makes a difference in the hash, and then we do an X and unnecessary rollout deployment, which is just this is a did. A few past conference, people I've had this conversation with people for the past several years, I've.

C

Seen several times where you'll upgrade levels do a storage, migration and every deployment in the cluster kicks off its a.

F

C

Experience I have helped people who are writing controllers to be able to explain like what the generation field is supposed to mean yeah. But even we don't always get that right. We get it right most of the time, but we don't always.

D

Get it there's also the problem where the generation covers the entire spec all needs to be done if you change the template, so the generation isn't quite exactly what you want close.

C

Yeah I think we came to the conclusion that yeah when it happened, it wasn't ideal, but the systems would eventually resettle it wasn't like they ended up in a loop unless you ended up with action taken against an API server, and then you switched API servers to an older API server and then like weird and bad things happened because fields would get removed. The hash changes again, yeah.

D

There's also like, if you're running, in a split version because you're doing a rolling upgrade of API servers, then there's the possibility for several oscillations with this yeah.

H

I mean yeah: what about stateful sets? Does this happen with them with them, too? I.

C

Don't know that I've seen one specifically for stateful sets conceptually maybe.

H

It would be more right, since it's it's more painful to you, know, unnecessarily to restart a stateful set. It.

D

Probably is a possibility, it would depend on how the controller is written. So it's confusing here because I think, like you, can already handle this correctly today by writing your controller very carefully in practice. Nobody writes the controller carefully enough.

D

Tim's solution is like a new rule about what sorts of changes you're allowed to make to api's I. Don't think that is, it was like I do actually don't think that goes far enough to solve this problem.

D

H

A little bit leery of saying you can't do this, because some people, some people, might run a controller. That's too sloppy I think for a team that that controls both the type and the controller they should have. The freedom to you know, write the controller correctly and you know solve hard evolution problems. You know.

D

That that's actually a really good point in that today we do something questionable which is reused by pods back as the pond template in the like deployment object. So if somebody, somebody is adding a feature to pods and they add a field to the pod spec that also affects the deployment spec, which is you.

C

D

Not correct you.

C

Know how I said that I didn't quite agree with you about being able to embed other types inside of your series. This example is exactly why it makes you not the owner of your schema, and that makes some things very very easy, and it makes other things less easy. Yeah.

D

I completely agree that the entire difficulty of that features making this problems face right, like just mechanically. Embedding your references, not that hard. It's all be like what happens if the things that you refer to changes, that's that makes it hard.

D

The situation today worked. Everybody copied everything in C or DS is maybe okay.

D

Well, it's wrong! You know it's.

C

Wrong in other ways, right I mean I can accept that it is wrong. Another way, sometimes you really do mean just opaquely whatever this is that's the schema yeah, and sometimes you don't all.

D

Right, I, don't think we're gonna solve this in this meeting, but I wanted to open the conversation about, like maybe it's time to revisit and or rewrite and or to just define for ourselves how our api's are supposed to evolve. I do.

C

Like the idea of being able to ride something that can take a like a CRD, manifest a and a CRD, manifest B and tell you whether it thinks the two different schemas are a good idea. Like did you change the meet the scheme of you, 1 beta 1 to the new V 1 beta 1, in a way that we think is a good idea? I can see a lot of value in that tool.

D

Yeah, that sounds awesome, but it's applicable not just this year to use like we could run it on on the API.

C

Spec, yes,.

D

Exactly exactly sure and actually for a server-side apply, we might be doing something similar in in terms of evaluating whether changing.

A

D

The list associativity is safe or not, so maybe there's just a general tool that we could write. That would help people here all right.

D

Hey do you want to finish up yeah.

A

It's just regular reminders. I, probably everybody knows coop con Europe was postponed. So if you kind of talk you know you may be contacted. We had to also postpone our deep type session for it, a missionary I guess we will regroup and see who can make it to the news. Eight's reminder number two: we do open for everybody. Triage is PR / issue triage twice a week come on join us and, finally, the reminder. The agenda for this meeting closes the day before so yesterday afternoon. Est so look forward to see you in two weeks.

A

I

A

Thank you, everyone. Thank you. Everybody.