Kubernetes SIG Instrumentation, 16 Apr 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG Instrumentation 20200416

Description

SIG Instrumentation Meeting - April 16th 2020

A

All right welcome everyone. This is sick. Instrumentation today is April the 16th, and what do we have on the agenda today? A couple.

B

Things you want.

A

B

Yeah the first one is ok, so I read this thing yesterday, so excuse the listener. Keith garbage I was very tired. I will I will rewrite it, but it is basically what we discussed and lasted meeting for.

B

Being able to dynamically restrain the cardinality of the metric value, metric label, values.

B

So if people have time to look at it or people have read it and they have thoughts, I would be. Would love to hear feedback about it, get some ideas.

A

Actually, lily and I were briefly talking about it today.

A

One thing I think yeah I, think Lily I mean she can speak for herself as well, but I think we were talking about there's like one PR out there about reducing cardinality of bucket metrics or like buckets the where we were wondering if we could potentially solve the same thing with that as its densed I. Think no but I wonder if there is a possibility where we could do something like that. No.

B

Way yeah: this would work with that right bucket. Our labels.

A

They are but the the way that I understood your proposal is that we any label that we try to set that is not part of the white list would go into some other bucket right and that wouldn't work for a histogram at least no.

B

It would because okay, so the way I think about labels is a little bit different from the fundamental Prometheus model for the Prometheus model. Labels are not actually tied to a metric right like they're like like the underlying data structure is like you have this label it's some string and there are n values of that label and it doesn't matter which metric it came from.

B

However, practically speaking, that's not really true. That's not how people interact with metrics. ah Labels are scoped right, so I'm not see.

C

A

The unique combination of a set of labels makes the identity of the time series right.

B

Well, the new combination, the label values plus the metric name- makes a unique time series. The metric name is just meta label, daddy. That is true, that is true and and so like, yes, and so, but the way that so the way my proposal sort of abstracts. That thing is: if a label is considered nested, it's already going to be scoped to that metric label. Name right, like.

A

B

A

I'm saying yeah you're said: you're I understand it.

B

Would solve the problem that you are describing? It would absolutely solve the problem that you are discovering, because I am NOT doing this generically for a label I'm doing this to a metric and label yeah.

A

I think we, the the only part that I wasn't sure about, is I. Think you have like the the fallback mechanism, basically where, if you try to set some label to some value that is not whitelisted, it will go in some non value or something like that right and so I'm, not sure that would be okay for a histogram I mean. Obviously it's not that big of a deal, because the histogram or I don't know I. Think.

D

B

Would be nonsensical for the histogram, but we can we can hard code this right, Elias. We know what Elly is yeah.

A

Yeah exactly like that, that was kind of the idea of or the discussion that lily and I were having. Where.

B

I meant and or yeah I mean like make a poll I'll merge it into the thing. It was just a.

A

Like site conversation that we happen to have because of the bucket thing, not so much proposal, I actually.

B

Did not did not think about that thing, but III do remember that issue where people wanted to get rid of some of the dimensions from one of the metrics, because there's tiny buckets but yeah yeah, yeah sure I think yeah. That would be super easy to do. Yeah yeah. So if anyone wants to help out with the cat or whatever just you know, feel free to make any modifications or whatever send a pull request and yourself as a go out there, whatever yeah.

A

And I think I think in general it's the dates for.

B

Released and so we have I guess we have less than a month three weeks.

A

B

A

B

Writing up for everyone else's help. Please! Oh yeah,.

C

D

C

A

Yeah hon, we can't hear you I think you were talking about important dates. Oh yeah, can you hear me a little bit better yeah.

E

I think Khan was saying that uh oh sorry go ahead. Oh yeah.

B

Sorry, I just yeah, better okay, great yeah I was just saying we have a deadline coming up and if people want to get their caps in there's three weeks. So probably if you want to get a cap in this release cycle, you would need to have it written and probably have some people looking at it by the next next big meeting.

E

Realistically, there are only two working weeks before the cap freeze, so.

B

Yeah but I imagine if someone has a cap that they get a draft of by the next meeting, then we could probably get it in I can probably get it in.

A

So yeah I I think it depends on the size and everything but I'm sure I think if anyone wants to realistically get anything- and it probably needs to be open by the next meeting so that we can at least discuss it a little bit and then obviously do reviews and potentially merge it. But uh what I wanted to say about like a closing note, but the first topic I to me. It doesn't seem all that controversial I.

A

Think the demo, as always, is in the detail, but yeah the limiting label values doesn't seem all that controversial I think there was some talk about it in coop site, metrics already, as well, potentially also with label values.

A

Sorry with like more dynamic metric names as well, but yeah. We can talk about that and on the cap as well. Oh.

B

Yeah yeah yeah there's lots of carnality.

A

Okay, I think. The next point that we have is should we plan on bringing metric stability to 2ga in this release. I.

B

Would like to start marking metrics is stable because we are treating them as such and we should probably be auditing changes to these things. I think.

A

Before we can do any of that, I think I really think you're right, but before we can do that, have we actually completed all the things that we wanted to do for the metric stability kept in itself because I, don't think so. I think there are a.

B

Couple of CIA there's a couple ci tracks and there's.

B

There's we need to be able to turn off metrics you ah so yeah I'd say those.

E

Things are relatively big. We probably can't take that to GA without those things.

B

Actually build, we just need the flag. Yes,.

E

Yes, I think if we'd get all of that stuff done before enhancement for use, then it should be fine, but like we can work on it all simultaneously as well trying to identify that list of metrics. We want to be stable, but I don't think we should set any of them to stable until we have the rest of that stuff agreed.

A

B

I mean there are: there are a subset of metrics, which I really want to mark a stable like the requests ones, because people use some press ellos. All right you, you really can't change those like the request lanes. You risk request. Count they're, like it's kind of special I.

A

Tend to agree, but we just changed them so I'm, not we changed yeah. So what do we do? We, the.

B

D

A

D

A

Do but like I, don't feel too confident having just changed them that we should be marking them as stable right away. So.

B

Alternatively, I okay, so if we don't want to do that, then I actually don't like the metric and I think we should create a better version of it like right. So if that's really the way you feel about it, then we should wipe the slate clean and get a good metric like I should have yes, yes, do.

C

You prefer that, because yes, thing.

B

Is like it drives me? Nuts? Yes, I, agree: okay! Okay! Oh that one! It's going to be a little bit trickier, but okay, I, don't.

A

Disagree at all we, we also use this for SLO tracking.

D

A

We yes I, definitely agree. We can improve this a lot. Yes,.

E

Dumb question would the list of metrics we want to be stable, like should that be a cap, so folks can comment on it before we go like as the code change is trivial, but like picking that list is maybe worth discussing. It's.

B

Going to be it's going to be up to component owners because they are the ones who are agreeing to the contract of maintaining those metrics. And there are api's.

A

Plus, like plus, it gets sent into our directory right, which we are all dressed.

D

A

So yeah I think.

B

A

The very least for the for the first ones that we're doing we should probably actually discuss them in this round, like I'm I'm happy to like start reviewing the metric, see what we would want to change and do those changes and kind of go about it in that way, and then once we actually reviewed them a couple of times, maybe for one or two releases, we've been happy with uh with the actual estate.

A

That's where I think something can probably become stable, because it's a pretty tough commitment, in my opinion, to make it to make a metric, stable.

E

Should we be sending up some communication about this like on kubernetes day or something like that, like yo, you know, propose some metrics to be stable. Let's talk about them. This release I. Think if we're making this, if we're aiming on taking this GA, this release we're going to have to do that because I just don't see another way like if there's not a cap or some sort of communication, the rest of the project I'm, not sure how we're going to like do this in some blanket form.

B

Well, the API machinery ones are a little bit easier for me because I'm active in the API missionary. Yes,.

E

But like it's not just API machinery, we care about we care about everyone, yeah.

B

Yeah for sure, David's, probably good point of contact for the node ones, yeah.

E

And I'm not trying to come up with a list of contacts, I'm saying we need to inform everyone if we're taking this GA, so somebody should draft a communication and send that out to kubernetes dev or whatever be like hey think about this talk to us, etc. Here's the plan.

A

Yeah I think that's also very much why we need to actually finish our cap completely before we can like mark any of them as stable, because at the end of the day, basically, we haven't done finished all our work, but we're asking other folks to do work more or less. With that framework, I, don't think I think it's gonna get only messier if we invite people to already start using it. It's.

B

Not it's not that bad right I mean like these metrics exist. People depend on them and, like you can't like I know that would it's an alpha but like really, if that one disappeared, like the request once disappeared tomorrow, like you would hear about it, no, you would break it literally. Everyone in the world, like so I, mean like not marking it as evil. It's like, okay, we're just okay with breaking you for now, because we don't have a contract. It's like, but.

A

That's why we took this whole thing. We did.

B

A

Tend to agree that, like for very sensitive, sensitive metrics like this one, we can talk about a a more sensitive migration strategy, but yeah I, like do all.

B

Tool, metrics or something yeah yeah.

A

Yeah but I think this is a good one to take up for our next meeting that, let's, let's look at like the two most basic metrics that we have, which is request, count and late movies and just review them see what we find odd, maybe more violations to our guidelines, which I'm sure there there are maybe not within those two but I'm, pretty sure they're they exist, so yeah, yeah content type is camel.

B

Cased label: it's the only camel taste label. Let's dress me nuts and.

A

Probably something that shouldn't exist at all yeah.

B

B

Yeah, so we have okay, we have. We have a couple of issues for the bug scrub. We have two two issues there. There.

A

Were a couple questions and chat. People cannot also feel free to speak up there. One question, but, as I said you, you can feel free to speak up as well.

D

I'm, this is my first meeting so I. Probably these are very naive questions and you know.

E

D

Did but I just I've been asking around and I'm finally getting to the point where I realized that if I I haven't gotten an answer anywhere else, so I have to like go to the source and if I can't find it here, then I'm going to have to be part of the solution. Cool yeah.

A

Perfect, do what general just want to ask you a question. Oh yeah,.

D

So um so, in the context of this particular discussion, I'm just curious of whether there's already some thinking around having like a maturity index for a particular metric. So if people see it, maybe they'll see a value that will be a reflection of like how long metric has been in use. How frequently it churns how many dependencies it might have something like.

E

Things there are not much there's not much other than like. What's in sort of like source control, the thing we were just discussing the metric stability framework is supposed to provide some of that, but it is not yet ga-oh.

D

So you're talking about the tool that would actually do that. Okay! Thank you. Sorry, yeah.

B

Yeah there's a cap right here. Let me see if I can do it up if.

E

Someone can link up in a chat that would be great. Oh.

A

Thanks yeah so.

E

A

To explain the topic, though, that basically, we've introduced a framework where we can mark individual metrics as alphabet astable, and alpha is essentially, we can break anything at any point in time. We're making no guarantees around it. I think I think we never actually finished the entire discussion around beta metrics, but stable is essentially we need to go through a very specific deprecation model to remove something.

A

So you basically the idea is you can rely on these metrics, but in the same way- and this is kind of the discussion that we were having- we need to make sure that the metric that we're marking stable, because it's a large commitment we need to actually ensure that it's high quality and that we're not going to change it anytime soon, because that would mean we need to go through the deprecation process. Again.

A

But you're totally alright, there's like things like churn or stuff like that is a very valid concern and something that we need to take into it very much into account when we mark something as stable, yeah.

D

Because I think that in general, mmm-hmm I've seen this movie several times before I'm just relatively new to the kubernetes community. And so what tends to happen is we have certain habits around how we approach unknown new novel problems and in sometimes it's useful to simply know how much is known or how confident we are in the information that's available, and then each individual consumer can make the decision about how dependent they want to be on that resource and that allows for kind of an ongoing, continuous improvement in some respects.

D

The the the scale of like alpha beta release is sort of more of like an old waterfall --is-- approach where one has like an artifact and then the artifact it's done, and then we simply move on to the next thing, whereas this is I think a little bit more like a traditional manufacturing QA process, where you know, eventually you get to like errors per million, and then you get down to like errors per billion.

D

So the precision keeps sliding- and so maybe I'm just wondering if there might be a way of reading some that thinking so independently of the other categorization I'm, not saying that we should undo that, but but instrumentation you know the metrology, a.

E

Great idea to me sorry, just keeping an eye on time, because they only have six minutes left in the community meeting is right. After this one, we have I think three things left on the agenda, bug scrub, a bug that sera theist mentioned and fixing the calendar. Invite. Can somebody quickly, like I, don't know who put fixing the calendar invite on the agenda?

E

Can somebody quickly speak up to that, because I think there's there currently some issues we're trying to work through with who owns what in order to get the fix? Yes,.

B

The calendar invite yeah. That was me because we can't it really bothers me that we keep getting a link to the old meeting notes and then we have. We have like a pointer to the new meeting notes. So that's me: okay,.

E

I think that that is probably something that leaders should work to fix offline I. Don't think anybody else in the meeting can fix that I. Don't think somebody right.

B

How like how do people get the meeting invites? Is it just by registering to the group.

A

And the problem is the meeting. Invite itself was created with a chorus Google account that does not exist anymore yeah and it's quite it's kind of a mess. I'm not exactly sure how to resolve it. I think we said that potentially someone someone somewhere within Retta might be able to have admin access to all X chorus accounts.

A

Can we just make a second one that.

B

Somebody else owns I think.

E

We will get rid of the first one. We should probably discuss this offline. This is not something we need to talk to the whole sake with I. Don't think we're gonna have time to do a full bug scrub mark. Do you want to talk about this issue that you linked in the in the agenda? I.

F

Think that the to sting by hand but I, don't know I, don't have anything to disgust. It's just the idea that how we can ensure that all the changes to metrics are we someone from sig instrumentation, so I just started it currently talking with some people from from prowl and God looking for their feedback. What are their ideas, how to implement it? That's all.

B

Well, yeah and then we can have a more effective bug scrub because we don't get tagged on instrumentation related stuff. It's very annoying.

B

Okay, then I guess we can adjourn until the next meeting unless somebody has I.

A

Just wanted to say, I I do think we should continue this discussion that we had about the maturity model I'm, either happy to continue that discussion in the next meeting, or also happy to do it synchronously and slack, because I think this could be at the very least.

A

The things that you mentioned I think are fantastic criteria to mark something in whatever maturity. We we wanna. We want it to be or say like it must have these requirements or something like that doesn't doesn't have to be a hard rule right, but at least thinking about these requirements a little bit more. It's definitely something that we haven't done enough. I think so I'd love to continue this discussion, yeah.

E

And just a reminder for folks and for some folks who this is their first meeting, welcome if you want to discuss stuff in the cig meetings, just make sure that you throw them on the agenda as long as you're a member of the Google group, you should be able to edit that and add that there.

E

Anything else from anyone else.

F

Maybe we should put it on the dock that people should add themselves, because it's not obvious made for people.

A

That's a good idea. Yes, I'm.

E

Pressin to take the initiative.

E

D

E

Think we can adjourn.

A

In February have a wonderful local.

B