Kubernetes SIG Instrumentation, 15 Oct 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG Instrumentation 20201015

Description

SIG Instrumentation October 15th 2020

A

Hello, everyone- this is sick. Instrumentation today is october 15th, and um we have one topic on the agenda um from david: take it away.

B

Okay, um so there isn't necessarily any action items here, but I just wanted to raise it early and um because I suspect that this will come back to haunt us someday, which is that we have our own set of standards for how people should name labels and such the most important ones are things like container pod node.

B

That sort of thing well open, telemetry, which I've started to become involved in a little bit, has also added their own standards for how to name things and in particular, they've prefixed, all of the kubernetes specific names like pod and container with kate's dot, so they use kate's dot pod, instead of just pod.

B

The reason why this matters is because um if you want to do any sort of trace metric correlation or even now that we have structured logging trace log correlation, that's much easier to do when they are using the same keys.

B

Thankfully, it's actually pretty easy to rename labels uh in both systems like prometheus, obviously has label mappings and open. Telemetry also has the ability to rename labels. So I think this can be pretty easily mitigated, but I just wanted to raise it as like there. There are now two standards that that differ in how to refer to things.

C

So, uh just what's up clarification, I'm a little bit confused like what is actually being prefixed with kate's underscore pod.

B

uh So, for example, in a prometheus metric for um container cpu usage, we use the label container and we use the label pod right if you were to uh trace something that referred to a container or a pod today, that would come out as with labels, they're not exactly the same, but they're, essentially, labels, kate's, dot, pod and kate's dot container.

C

Where does the prefix come from.

B

uh It's part of their semantic conventions, so they say that when you refer to containers and pods, you should use the semantic conventions.

D

Any reason- and they can stick to our naming or.

A

It it seems a bit wild to me that this will ever work that there are globally consistent uh label namings. I feel like.

C

A

Should be like another layer on top of this to do correlations well,.

B

They're, it's it's pretty much the same as prometheus. Right like people sometimes refer to things with different labels and have to do label remappings. So it's not.

D

B

End of the world right, it's just a call out that I guess especially in relation to the tracing cap. Maybe we might want to do something, because if we're uh instrumenting things within kubernetes, maybe we want to choose between the default. If I understand it is to use the open telemetry one, so maybe we want to choose the um having traces from kubernetes components, tagged in ways that match the rest of our um guidelines.

C

Wait are our dots even permissible in the previous.

B

So, if you're using prometheus, uh I think open telemetry will end up substituting underscores for your dots. I I would need to confirm that that.

A

Would make sense, that's the typical thing that we do when we convert to prometheus labels.

C

That's weird, though, right like the default, behavior isn't consistent with prometheus labeling, like that's a little sketch.

A

Yeah, I I find that weird as well.

B

I can also look into that and report back if that's helpful on why they.

D

B

Instead of underscores, but I guess that's a separate topic.

A

Yeah, so I think this was like a name spacing topic in open telemetry. I think this topic has come up uh in various forms in various projects.

A

um I guess it's just a little bit unfortunate that, like prometheus being one of the first or one of the only open standards that are being supported by open telemetry that something was chosen, that's not prometheus compatible.

C

So this, I was really happy that you put this on the on the agenda because there's another aspect of this that I wanted to ask about, which is basically, um I have talked to some people about this offline, but um I kind of want to start passing context to.

C

For for uh for labels similar to like the way open census does this, um so this is relevant to this particular conversation, because uh this has to do with the way that we're passing along metadata right for instrumentation, and currently we have um well. I mean we just do lots of things depending on where the metrics are in the code base.

C

But uh generally this comes in the form of like you have a function and it takes like n number of arguments and you basically have to plumb all of these arguments uh all the way through which can result in errors, because people sometimes swap arguments, um and it is really ugly to pass it all the way through. If we just fund context- and we did this it would, it would make a lot of things- a lot nicer and then we would also have consistent access to this metadata uh for stuff like this right.

C

So this is a a different aspect of the metadata stuff, but I mean: does anyone object to like passing contacts.

B

I guess my question would be other than attributes of objects. What are you thinking about storing in there? Because it seems like most of the time right now we're uh we're using, or maybe I'm misunderstanding when this would be really.

C

Requests like, for instance, request cycles, have a number like. You have many many layers uh like filters, and you just accumulate all of these values. uh You can either accumulate them in the context or you can like literally take them and then pass it to the next function. Until you get to the point where the metric is actually recorded.

C

So then, basically you're, just like accumulating all of this craft until until you're ready to record that metric and that metric ends up having like 12 arguments, and you know like uh it can be kind of a mess right like and we've had, we've had instances where, basically we're passing along all of these arguments, but then, at the very end we do some sort of transformation to to make like a resource, a canonical resource or a verb, a canonical verb or whatever, and then basically like.

C

If you don't do that transformation at the very end or whatever, then, basically, you end up with multiple metrics having different things, and this has happened to us several times right so like if we were to just like funnel this into one place and have like this. Is this is this? Is the metadata for this request? You can record whatever you would like right like uh if you have some tracing thing use this thing right, like everybody, gets the same data right like.

A

C

It's just cleaner.

A

Yeah, I think this makes sense, like uh other other uh things, that I can imagine that I've seen in other projects that it wasn't necessarily done through context. But it's totally doable through context is like wrapping of the metrics registry to say, like everything that I passed down here, um like all the metrics, get this label or get this prefix um or stuff like that.

C

Yeah I mean we can use the prometheus curry. With I mean it's basically.

A

C

Thing except the problem is it's like I. I don't want this isolated to the metrics, because now we're talking about like intersecting, yeah yeah.

A

I was just saying that that's the parallel to what exists in metrics already but like wrapped in context essentially, and um this could help us also um with uh plumbing through the metrics registry, like we wanted to right instead of having the global one. Hopefully that should like I would. I would hope that somehow, um like binding calls to metrics to the context would be possible, um and then we can have this um refactoring, hopefully like as a side effect kind of.

C

Yeah I mean this: this is completely backwards compatible right. We just add a new thing to our our component-based code to take the context, and we just like do contact stuff to do for now, and then we just start implementing it yeah. So it should be super easy. um Okay, then, if no one has any objections, then uh I don't even think this requires a cap. We can just probably just add contacts, because it's plumbing context right like this, is just like.

A

But but we're we're we're kind of proposing to put something put a value into this context. No.

C

uh We can we don't have to right, like you can support like first you have you expose two apis, one which doesn't take the context. The label values the regular way and also it takes the context initially, that context will be a contact set to do and won't do anything until component owners implement passing label values right. So.

A

Like you're you're, not even suggesting yet to put anything into the context, just literally adding uh plumbing of context like, I don't think you need to kept for that. Yes,.

C

Exactly I think, when we get to the point.

A

Of saying we want to embed something into this context, then probably we should have a cap for that. Okay,.

C

Yeah sounds great awesome all right. I will add this.

C

Okay, yeah, that that was it for me.

A

um David was there anything else that you wanted to discuss on your point, or uh were you gonna circle back with the with open telemetry, see.

B

I will circle back. There also aren't the action items for this. It's more just like a a heads up that there are different standards out there. Yeah.

A

As I said, I I I assume like I not that it's necessarily wishful thinking, but, like I, I assume, there's always going to be um inconsistencies, um no matter what standards are uh agreed upon, even in kubernetes right like um there are other projects that integrate with kubernetes and expose metrics that don't necessarily follow the scheme. So I think we all um pretty much agree.

C

So but open telemetry is nascent right like it's new and their standards are changing. Prometheus is like kind of a defective standard now so like is it?

C

Would it be possible for us to like ask open telemetry if this is like set in stone? If this is amenable to change or like because like if it's colliding already that's like kind of not the most positive sign.

B

Yes, I would say: they're not ga, yet and they're, making a whole slew of breaking changes right now. So if there ever was a time it would be now um I don't know, I think it seems like they've agreed to do name spacing, but I don't know how much thought they put into the name spacing character and if that was like, I don't know necessarily why they did that. So I can at least look into that point.

A

Yeah that that would be really awesome um like if somehow we could find something that would be somewhat compatible with prometheus.

A

Like I don't know just off the top of my head, like double underscore or something to um to to signify, there's a new namespace, there's a namespace ending here or something I don't know, it's just a thought right like um it would be neat. If we could uh provide our input here, I guess there I would. I would think I would hope that they're happy for our input as well.

A

um Okay, our other topic is lily. Coop state metrics thanks.

D

Hun for the reminder um we just released uh another pre-release candidate, essentially um after some bug reports, but we're going towards v2 quite nicely um we're working with six scalability soon um to do some scale, testing and yeah our we just want people to use it more to get more bug reports essentially but yeah. That's all.

C

I will mention that in the community uh update today, awesome nice thanks nice.

C

Okay, uh is that is it different, that's it for the internet. Do we do triage after this or how does that work? Triage.

A

Is next week.

A

C

A

B

Yeah there's not too much to do every week, so it's usually just one or two pr's one or two issues and we get them doled out.

A

Yeah, but it's good to have this scheduled because it makes us actually do it, which is definitely helpful. um There have been a couple of things where nobody had replied until the point where we did triage. So that's good.

A

Okay um looks like we don't have anything else for the agenda. Is there anything else that anybody would like to talk about? Otherwise, I guess we can give everyone 10 minutes of their time back, go once and twice all right, then have a wonderful local time and see you next.

B