Kubernetes SIG Instrumentation, 20 Jan 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG Instrumentation 20220120

Description

SIG Instrumentation Bi-Weekly Meeting Jan 20th 2022

A

So welcome everyone to today's edition of sega instrumentation meeting um today is the 20th of january, um and I will remind everyone that the meeting is recorded so be nice to each other, and I think we can kick off the meeting. um Should I share my screen to share the agenda. Maybe.

B

Feel free, I dropped the agenda in the chat.

B

Oh, maybe our first order of business should be an announcement. We have a new tech lead.

B

You want to do the intros.

C

Yeah uh damien is our new tech lead. Everyone give damian a round of applause, warm.

B

Round of applause and welcome.

C

B

uh As his reward for becoming tech lead, we're making him run today's meeting so.

A

Sneezy was completely regified.

B

Yeah, as we can see, do we even have francesco. I feel like he put an item on the agenda. That's uh f romani. He put an item on the agenda and then I don't know that he's been able to attend uh so I've just been kind of punted.

A

Yeah, it's he's not.

B

A

On the list yeah, so I guess we can go to david point.

D

E

Alrighty, so um I'm still trying to decide what my opinion on this is so uh jordan and wojciech both um they're, making some edits to deprecation policies for api versions. But they came across the deprecation policy for stable metrics and um are suggesting that maybe we as a group should reconsider our current deprecation policy, which is, I think, it's.

D

E

For a few releases and then it can be removed, so um I just wanted to raise this because I don't think everyone was cc'd on here um and yeah get people's opinions. I I think we should stick with the status quo unless we feel strongly that it should change, but um I.

C

B

Want to be, I want to be able to.

C

B

Metrics, stable, metrics, yeah, stable metrics are not stable apis. This is specifically talking about apis. Like you know, for example, apps v1 kind of thing does.

E

B

Do the same thing as metrics.

E

No, I I agree they're suggesting that we should follow in their footsteps for metrics and treat them.

C

Yes, they want, they want us to treat this as permanent apis.

C

I don't think that that necessary. Well, I don't think I don't think it's necessary because we're going to end up with blank metrics that end up not being used if some code becomes obsolete. That doesn't functionally do anything but we're locked into keeping it around forever, because we have a policy.

B

So one thing that I would maybe add- because this came up in a call earlier in the week- um is like what do we do if, for example, we find like a metric cardinality explosion, or something like that like, for example, that cbe that we had where people could put like outrageous things in for an http method, but that metric is stable like if our stability policy means we can't like do anything to it, then doesn't that mean that, like we're stuck with a cve like, I think we need to be able to like patch that sort of thing or our guarantees need to be able to handle that sort of thing.

B

So I was wondering if we should also have a policy that uh stablemetrics must have like strictly bounded cardinality or something like that.

A

But at the end of the day like even if the problem occurs, um like the unbounded, generality will only be on a label, not the whole metric.

A

So I don't think we would have to remove the metric completely like, although maybe it will lose uh lots of explicitness if we remove a label yeah.

B

But I think our policy says we can't change like we can't like blank out a label of a stable metric either.

E

A

Yeah but like the use case, I've loved is that if we ever find a cv, we would have to patch it down to three versions of kilobytes.

C

It's it's not a problem. This is not a problem like if we have unbounded cardinality, we have mechanisms for bounding that, like we can, we can do an allow list approach which caps, the the label values.

B

But that's currently a requirement for stable metrics. That's why I'm saying maybe we should make it one.

C

Yeah yeah, but I'm saying that we don't yeah for sure no for stable metrics. This is like literally the only thing we can do, because we have a problem.

B

Because otherwise, we can't touch them uh other than like removing them. After.

C

No, I mean that that's what I'm saying like right now because of our policy with stable apis. We cannot remove the label. Therefore, what I am suggesting is the only thing we can do and it is an acceptable solution for the cbds is what I'm saying.

B

Yeah well, but I'm saying if we ensure that that is a requirement going in like if we ensure that they have bounded cardinality, we don't have to worry about like retroactively, adding that to them or something like that.

B

um But I mean in any case, I think part of the problem here is like what we've previously discussed many times with, uh like our metric stability policy, where, like it kind of loosely, follows api stability but like we don't have a beta and uh like alpha and stable, are kind of misleading because they don't really mean the same things as alpha beta ga for like either features or for apis.

B

So maybe we want to like rename them and proceed with our proposal to like change the way that metrics stability works, uh because it's confusing everyone.

B

Like I agree, I think we want to be able to remove metrics because otherwise, like they're much more of an implementation detail, uh I mean- and maybe we say that, like you can't add stable metrics that like where code potentially could be removed like so long as we have an api server survey request, we're always going to have request duration right so, like that's, not really a risk, we could realistically say that we keep that forever, uh but I can think of maybe other things that that.

C

Web web books go away or something yeah.

B

If web hooks go away.

C

Yeah, if web books go away, then basically like the web hook stable web metrics, don't make sense anymore.

B

Did you raise that as a possible thing in the thread.

C

No, I I haven't been able to respond to it. I've been bouncing around, uh but I I have been thinking about it and I will respond to it.

A

I did see the email like possibility. There is also another possibility. I see when I went through the code of like the registry that we are using uh for like the gpa stability, and it is that we can hide some metric. um We.

C

Can deprecate? No, they they get deprecated and they get hidden and then and then you delete them so like you can enable them. So like the reason why we have this. This escape hatch right is because basically, we are giving people a one release window for them to re-enable a metric, unbreak their ingestion pipeline and then migrate to the next metric.

C

Right so like we auto, hide it, and you can explicitly re-enable that deprecated metric it will, but that causes you to consciously turn it back on knowing it will be gone next. Release that that is. That is the reason why it exists right, like yeah, but.

A

You're shopping to do something similar similar to what the api um live stream like what they're doing for the api like if a metric is stable and um like we want to duplicate it or whatever, we just hide it forever and then, if any users need it at some point, they can always turn it on, uh and we like, as you don't need to care about it anymore, like they just can uh allow the use of this material.

B

To delete the code paths, if we did that and yeah.

C

You need to what do you like and you you basically have to have this like skeleton of a metric forever. That makes no sense right like if web books disappear, we're going to retain the web hook metric and then it's just going to be ex like well like it's going to be broken anyway because like if it's not instrumenting anything, it's actually not going to appear in the public api.

C

So, like your scrapes, will still break right like like that's an argument for for actually removing stable, metrics right because like if they actually become unused, they will not appear and they will break ingestion anyway. Well,.

B

Han perhaps what sig arch is suggesting is those things like anything that is possibly going to be removed from kubernetes should never become stable, because that violates our stable guarantees so like, if there's any possibility of like you know, this feature might go away, meaning that these metrics will go away. Then they can't be a stable metric.

E

Or, should we just add an exception clause like the only case in which metric can be removed, as if the behavior it's describing no longer occurs,.

E

Like I can't tell you how long this thing took, because we're not doing that anymore, um it still seems useful to be able to describe something as stable and clearly there is some confusion over using the word stable to describe something that isn't actually that stable, but maybe stable.

C

A stable, stable, I mean stable, occurs, it's a spectrum of stability. Right, like you have things that are not stable, and then you have things that are like more stable. You can rely on something like the amount that you can rely on. Something is not like.

C

uh You know a a polar thing like like you can rely on something more. It's like it's a qualitative attribute. You can rely on something more or less than something else right like like.

E

I get that and.

C

It's not binary is what I'm saying right. It's not like a true false, like, um in my opinion,.

E

I I I agree um it. I guess we've defined our spectrum as what alpha and stable and so yeah.

C

Yeah yeah and we we.

E

Don't have a notion of super stable which is sort of what they're asking for perhaps.

B

Yeah, I mean really, it maybe was a mistake to not have a beta, but we sort of were like oh well. We don't know if we're gonna need it.

C

I have yeah, I have the it to do kept for that. But yes.

B

I I think the the to do cap is possibly where we want to like to raise this stuff, or at least come up. I mean, if we're just making a policy, it doesn't necessarily have to be cap per se, although I guess, if we're making changes to.

C

It's going to be code changes right because we actually need like static analysis for beta and we need like yeah well.

B

I don't know that it'll be that much more work beyond like what we would be doing in a policy uh in terms of like writing it down. So I think everybody's going to have to like kind of duke.

A

B

Out in terms of.

C

Yeah I mean we.

B

Basically got signed.

C

Off from arch for the beta stuff they're down with it, um we circulated the idea and then basically, I ran out of bed with to do it.

E

So to resolve this, should we just link to or say that we're going to tackle this.

C

E

As part of the discussion on the cup.

C

Let me let me let me let me circle around to it. I will I will reply. I have been meaning to reply, but yeah. I will reply today after this meeting.

C

C

C

Is there anything else on the agenda or is that is that primarily it.

B

I think that was the only item that we had. uh I also uh I I forgot to remind us about the upcoming release dates. uh If people want me to do that.

E

Is this the last meeting before the kept freeze.

B

uh That's a good.

E

B

I think it technically is because kept freeze is like the same day as our next meeting, so.

B

We super quickly go over um uh like the the list of caps that we pulled in for the release. I can share my screen if you want, because I I was wanting to put them in the.

A

B

A

Yeah yeah, please do it.

B

Okay, ah sorry, one second.

B

What are you doing, zoom in hands.

F

B

F

Question about the stable matrix, so yeah so currently, I think there are two options: one is to delete some matrix, another one to modify or update some matrix. So currently we are, I think, for delete metrics. We hope we can delete some measures later, but for update or like modifying some metrics. So it's it's it's supported or not imported. If the um metric status is stable,.

B

F

Don't understand.

B

The question uh yeah.

F

So I'm saying that, so if the metric is stable in the future, we might have two kinds of options: one to delete it and also to change the matrix name for.

C

Example, the metrics name is deleting it okay right because you are effectively removing that metric. Yes,.

B

Okay, can everybody see enhancements.

C

B

uh So I put in five enhancements for us for this release. uh Basically everything with a question mark I didn't add, uh and let me show you what I pulled and then you can tell me if your favorite cap is missing.

B

For example, we may decide that we want to do something with the metric stability stuff because we need to so um here are the five that I hold so api server tracing uh going to beta kubernetes system components, log sanitization, uh it's being deprecated, uh open, telemetry tracing is hopefully going to alpha uh deprecate, uh k, log, specific flags and kubernetes components, hopefully going to beta and contextual logging, which is a new thing from patrick, uh I think is maybe going to alpha, uh assuming that we do it for this release, I'm not sure if there's gonna be consensus for that one.

B

um Is there anything else we had like? We discussed a lot of things, um but a lot of them were like question marks. So, like I don't know, if we we committed to uh metric cardinality enforcement.

A

Yeah this one: yes.

B

A

Is no need to update the cap, it's just like some implementation. That needs to be done.

B

Okay um yeah, so uh I uh have not heard from clayton on this one. um I think that this is being superseded by patrick's cap. Perhaps.

E

That's the contextual.

B

Logging, you don't believe so.

E

So we originally had their cup and it involved both a contextual logging piece and a trace id propagation piece, and we ask them to tackle only the trace id propagation piece and not the contextual login piece. I see oh, so it's separate okay yeah, um they did respond to me. I need to get back to them. Okay,.

F

E

Doubt that it'll make that we'll be able to reach uh consensus by the deadline, but I'll keep working with them for now,.

B

Okay, um so structured logging, it's not going anywhere.

B

uh So yeah metric stability classes was one question mark and the other thing that I had a question about was these the graduation of these metrics apis? um This seems to have been moved to 125, and I just wanted to confirm that.

A

Yeah, okay at the beginning, like shui, uh planned it for 125., okay,.

D

A

I've seen okay.

A

Cause like we don't.

B

A

Time to listen to that.

B

Yeah I suggested uh I mean, since these are all like just sub domains of metrics.kates.io. I think we should just do it as one cap like a single cap, because it's all basically the same thing uh rather than trying to have to track three different things.

D

It's gonna make it much harder for us. Yes,.

A

Like I'm, gonna have to go.

B

And review three different things: instead of one thing which.

A

Is basically the same, I personally think that it's a bad idea because, like even though, like we have two captive, follow if we merge all these initiatives together, like each api, has specific change. That needs to be done that isn't related to the other. I'm.

C

A

C

Oh wait, I think elena is making this suggestion, because the process overhead of each individual cap is extreme.

B

It's not just that, like I don't I don't see what the difference would be in any of these cases. Right like if we're graduating these to ga all it means is we're taking a bunch of files and moving them from like beta to v1 like is there anything else that needs to be done to graduate these.

A

Yeah some like maybe we don't want to graduate to ga all of them because, like there are changes that needs to be done to the apis like, for example, the metrics api, like the risk matrix api, we wanted to add a new feel to it to kind of expose um the like the information from c groups, because, like you, don't have access yet to like for code matrix, you don't have access to all the data from the board itself, you're on the other wow like the resource situation, memory, usage, yeah.

B

That'll make sense to me, but I wouldn't expect that to happen when we're graduating it. I would expect us to add that field and then once it's stable, then we just basically take the beta api and and rename it as v1 right like because otherwise we're stuck with the stable deprecation policy where we can never remove the thing. So I would definitely say we should not be adding fields going into ga uh like when we want to ga the thing it should be done, like it's ready to go.

B

E

Also add fields after ga right sorry,.

B

Yeah, we, you can add.

E

B

After ga and they can be versioned with the feature flag and all that, so that is that is also.

A

Possible, so what would be the best like? Should we graduate to ga and then do the change or um I don't know, create a new version like let's say, v1 beta, two or whatever? I.

B

Think it really depends.

A

And then graduate to j, it.

B

Depends on what the change is, there's no reason to add a new api version on if it's a compatible change, so uh I don't know what the changes that you're suggesting are. So I can't tell you up top my head if it's compatible change, but typically adding a new field like a completely new field, is always a compatible change.

B

So you don't need to do a new api version to add a field.

D

B

D

Is to like with graduating is graduating means that you have tested this api enough, that you trust that you will not remove fields like and until you want to take overhead of releasing version two.

D

So if you have an idea about changes, you should test them in beta and only like wait enough releases for feedback and to only then be be sure that you're not making changes. So the idea original about making to ga was okay. We no one requested any changes. We can ga at this at this stage, but with changes we need to. We need to like cook them verify that everything is like. It makes sense.

A

So the best would actually be to like do the change without going through a process and then, like, let's say, have a six months a year cycle and then graduate to ga.

D

Yeah happy review just happy review and.

B

Yeah, the the api review uh is often separate from kep review but like if you're adding any substantial new fields, you probably want a cap because you're going to need a feature flag to track. The graduation of that field, like I think that we currently have the policy that, like any new field in an api, should go through the kep process, uh just because, like it can be really really difficult.

B

uh Otherwise, to like go through the whole graduation process like that, helps us go through the steps and ensure that, like any operational things that we need to do, get answered and like that we're very confident in the design and whatnot, we typically don't go and just like change apis without caps. There are some exceptions to that like, for example, uh uh some fields were added in the past release um from some of the work that patrick was doing and one of them like.

B

Basically, the the field was the wrong type, uh and so uh patrick came back and he's like. Can I change the type of the field, and we said no, you can't do that. That's a breaking change!

B

It's like a beta api that will break the api so like we can't do that, but, like I think that we could do is we could immediately deprecate the field that has the type issue and add a new field that we want people to use going forward and write documentation for that, um and so like that, that sort of thing that would not have to necessarily go through like we wouldn't need a whole kept for that.

B

um But if we're like, I want to add this new field, and uh here is what it's going to do and like here's, how we want to test it and here's how we're going to evaluate how the field is working that should have a cap.

A

Yeah, I think we should.

D

A

Then rename all these skips into like what we really want to do like, for example, for the risk matrix api. We want to extend it to also expose information about the c groups uh and for the others, like it's more like about. Actually um the user experience, while using the gps.

D

So the problem with those metrics is that they predecess cap process. So we don't even have anything that we can discuss and track the changes through release process. So yeah like, I think the idea is if we are planning to to to add some fields, let's create a cap in beta.

D

Let's say that will propose beta 2 or those changes, and then at some po and at least have it as part of release process and be tracked, and at some point this cap will be revisited when we will want to go to ga and like specific, like decisions about changes in each api.

D

Don't like don't make it make sense like we. Don't not specific for specific changes or specific api cap doesn't make sense for me, because all the changes are in the same code by like in the same code locations with the similar users with similar goals like like here, goals being better service, like I s, for example, better serving uh auto scaling. So you need same users or same reviewers, so I don't think there is a like any reason to diff like different audience.

D

Like one case, caps could be different types for if there would be different audience to read them.

B

Okay, uh sorry to cut you off, but we are currently at time. uh Do we have any last quick things follow-ups we can do on slot anything.

D

When you think uh the contextual login cap currently is is not a trees, but the main blocker in discussion is performance, so I wanted to ping david, because contextual logging is related to tracing. So if we are want to invest in either of those, we should have similar front for sick arch to to bring that the instrumentation here is worth the cost of propagating the propagating the the context or like changing the context.

D

Okay, so it's just.

E

How can I help do you want me to just review.

D

E

D

In fact, at the contextual logging and like there is discussion started, I think, by legit about the performance. So if you can easier to descend I'll.

B

Take a look at that one too, I think I'm the prr reviewer uh and then uh one last thing before we drop uh sorry for going a couple minutes over, uh but I just wanted to remind everyone of deadlines. We have a soft prr freeze next week in one week from now on the 27th.

B

So that means please have your production readiness review questionnaire done by that point, it's kind of like the sla for the prr team just to ensure that they have enough time to get through all the reviews, because it's a small team uh and our enhancements freeze is going to be february 3rd, and that is when we're next going to meet. So if you need any help with your enhancements, uh let us sink asynchronously in the slack channel or on the mailing list.

B

Thanks for joining us, everyone today.

B