Kubernetes SIG Instrumentation, 4 Aug 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG Instrumentation 20220804

Description

SIG Instrumentation Bi-Weekly Meeting Aug 4th 2022

A

Hey everyone today is August 4th 2022 and welcome to the seek instrumentation bi-weekly. We have I think one or two items on the agenda, although it doesn't look like we have a an owner for the first item. So does anyone want to talk about the metrics API API server.

B

Who wrote this? Does anyone know.

C

um We talked about it last week and I think this is a follow-up. um It was uh was here last week, oh yeah, here.

C

Damien I think and I had a discussion about it last well, not last week, two weeks ago,.

B

Yeah: okay, what exactly did you want to follow up about?

B

Thank you, like AP, like stability for for metrics apis for.

C

um I think it was I. Remember it now yeah the mostly I think my concern of. Can we stabilize metrics API when we know that metric server implementing the current API doesn't scale well enough past a certain size.

C

So do we want to finish an API, but if we later on discover that we need to restructure it because otherwise we can't scale it um we're kind of stuck with it and, what's kind of my question, I think.

D

I, don't think we'll be State necessarily like, um even if we stabilize like we go ga with this Matrix API like we can always comment with a new one later on, like that, wouldn't be an issue um and also like. We have to consider that Matrix server is not the only implementer of this API like we have many projects, that's actually implementing this API um and yeah. Maybe the problem means in itself is that metric Server doesn't scale, um but maybe also project might scale, depending on their implementation and I.

D

Don't think that the API itself is the reason why we are having these issues in electric server.

B

C

Well, one thing: I I would like to clearly Define before we stabilize the API. Is the exact feature coverage like right now there? There is no the API definition like most kubernetes API definitions is the types it returns, but we don't impose any like requirements or guarantees of like it supports latest lectures.

C

It supports field selectors, you can watch, you can't watch uh um which, for me, is an integral part of an API definition, but right now it's not there and and defining these again puts limit on what we might be able to change in the future.

C

I'm not I, don't know if I'm expecting Too Much from the API I didn't finish in here.

D

Yeah I'm not sure we will need that that much to be honest, like I I, don't see like the actual use case, uh that we could have right now for this kind of future.

C

It's like if, if you're a cloud provider or like somebody who wants to implement metrics API provider and metrics apis I, think one of the few apis, except for the cloud provider. Things were there's one API and many or possibly more implementers that we need to come up with some feature parrot or something like the API can't just be a type definition.

C

We need to be able to tell people hey so most things like Auto scaling, Works in you're, close to using your metrics API, you need to support label selectors or you don't need to so negotiate a contract between the most users of the metrics API and the ones providing it.

D

We already have some contracts like the basis of the API, is a contract by itself, like the only difference is how you handle.

D

The data um like with the serving of the data is always the same um like it's have to then be implemented to serve it and to access the metrics wherever they want to access them, um but like at the end of the day, it's what we were trying to do when the API was created because, like you, can have many um internet platform um as the backend of your cluster um and the API is just here, the middleware to uh selling back to metrics from wherever you want to uh back to your cluster back to kubernetes and your auto scatter I I, don't think like.

D

We need a particular contract in terms of how um the data itself should be sent back to kubernetes.

C

Well, but that's right now, the only thing we have right now. The only thing we have is: if we send a request to this API, you get back a Json response in this format. Our type definition, but we don't have any like definition of like people can well it doesn't support metric Server, doesn't support watch but kubernetes apis in general support which people might expect that it works or rely on it in one implementation, and then it doesn't work on the other one same way, like I, think Auto Skiller relies on label selectors.

B

Metrics API is an aggregated API server right, yeah, yeah I mean look. uh Endpoints.Supporting watch is a thing in kubernetes like you can't watch Discovery Docs.

C

B

I mean so it's it's like you, you can't really expect every every endpoint to have watch I think that's. That would be just a mistake in general, no.

C

But what I want is if we graduate the API that we should be as it's an API that is not implemented by kubernetes itself, but can be implemented by Cloud providers and Biometrics certain, but other components that we specify at least some. If you implement this API, you should support this. You should not support. Watch you don't need to. You should support labeled selectors, because everybody needs for this API yeah.

B

I see I see what you're saying see. Yeah yeah I mean make it explicit what one should should implement.

C

Api, how to be conformant with the API.

B

A

There was a bunch of like efforts to do conformance tests for all of the types and like to be really explicit about this. Did they ever come to our Sig? Do you remember they came to signode a whole bunch.

B

So there's a funny thing about the metrics API thing, which is like it's almost orphaned at this point: how many people actually like work on it like um Merrick, used to.

D

Be yeah on history.

B

Yeah Merrick used to be one of the big contributors to it and I think he's basically dropping it. So it's starting to get into it like an orphan State, I I would love to see it graduated, but we need like unders for it like.

A

B

A

The metric server.

B

It's a metric server I mean like if we want like stuff, like conformance to us, yeah for sure right.

C

Yeah kind of both and like we're checking, if you know, and we can have some more people who contribute and own it more and it's where, like these ideas, come from that we don't want to graduate an API that might constrain Us in fixing metric server or.

B

C

B

Happy to have contributors to the project right like we would be happy to get anyone get it into a much nicer, State like especially like performance. That's obviously, a good idea right, like just yeah I, think it's on like not even deniable so like. If, if you or anyone you know wants to jump in on this, like we're happy to, like you know, add people to approvers or to escalate that process.

B

Just because, like we don't want this to get in an orphan State and we would like owners for it and we would like it to be healthy.

D

Yeah, do you think, like we have? An even bigger issue is like the other API like the crystal metrics API, and the external Matrix API like I'm, the only one maintaining them and I'm barely looking at them right now, um and they have essentially like the same. You should have been better since ages,.

C

D

Have many people relying on them but like no one actually maintained the actual apis.

C

You mean summary API, for example, or no.

D

No, no, the, like you have the resource Matrix API, which is the one that is used by Metric server um like CPU and memory and stuff like that, and then you have like histometrics API um that serve um like any kind of metrics about a particular kubernetes objects. So it's like namespace scoped and then you have the external Matrix API that serves any kind of metric and.

E

D

Just three Matrix API that we currently uh serve and um like supports, but we don't have like that. Many maintenance for those.

C

Resource metrics was the one from couplet in Prometheus format,.

A

Yeah I, don't I think so.

B

Wait which one we're.

C

Talking resource metrics is the one in the Prometheus format, from couplet or.

B

Yeah yeah yeah yeah, yeah, yeah, yeah, yeah yeah, it's the summary stats, API yeah.

C

B

Okay, yeah, it's a it's a similar problem.

C

Yeah so I, don't know like for those who don't know like from context of my team is GK metrics, so we're scraping all of this stuff and it's where we're like. Okay, we should help maintaining it and we're checking like how we can yes.

B

That would be great contribute. Yeah I mean uh yeah, like jump in, if you guys want like, if you guys need approvers or whatever we're happy to like escalate that process and make it faster than the normal timeline. I think the situation right now uh basically warrants that so yeah I mean like we're happy to work with you on him. Yeah.

C

And I think one one question for mine is also kind of on like Justice League have any plans or like I mean yeah, we talk about eventually metrics AP, but is there any anything on the road you would explicitly want help with or need help with or like any or is it mostly maintenance state.

B

Damian would know because Damian has been on the hook for graduating the apis.

D

Only like the main thing that we should solve um is to regroup all the API in the same repository like currently, we have like the Matrix API that is implemented in metric server um and the two other apis are implemented in, like a library that we have in kubernetes six, uh and it would be great to at least merge all of them into one repository that we could use as a library everywhere and like that, would remove dependency. uh Two metrics servers by like basically all the project that currently wants to support the resource.

D

Matrix API have to import metric server, which is not something that's going, because it's not a library, and it's also prevents like changes in metric server, because, like metric service, sometimes breaks all its consumer because, like it's not meant to be consumed in any way.

D

um So that's one of the major issue that I would think uh need to be solved uh and beside that yeah. What we really need help with currently is to graduate all these apis, or at least like see an actual bus of update, either like it's by improving the apis or just graduating. The current one interest start thinking about a new API that will like improve the current state.

D

But we really need to do something about the apis because, like it's been like almost like 20 releases or something where, since it's still better and we need to like- have a bus forward. At least.

C

Yeah, uh do we have any like document or.

B

Yeah there's some issues. There's some there's some issues. Yeah.

C

Okay, GitHub project or like stuff that wants to be worked on, that we can pass around and see if we find somebody.

D

Yeah I will send them to you thanks.

B

Okay, uh I guess we have time for the last subject: I just it's a quick update, um yeah I'm planning on getting the the extension for um metric stability in by the next release. uh I have actually been improving the stability framework with extending the static analysis, tooling. It supports summaries now.

B

um So, basically, by extending the static analysis coverage, we can more adequately cover the entire metrics code base, which is going to be necessary if we are going to start linting against Alpha Beta And stable metrics, which is going to be larger. A number and more diverse, set of metrics than than currently exists.

B

So I've started working on that just wanted to update people. um It's it's gotten.

D

Are you planning to submit the cap for the next release, then.

B

Yeah the next release, so I am going to start working on it right now. Do you need help with.

D

That because I started looking at it like when we discussed that topic but like.

B

Yeah I'll I'll, yeah, I'll, I'll I'll I'll, share I'll share what I I'll show what I got? Okay.

D

Yeah sounds good. I can help you with that.

A

David, okay, I snuck, another topic on I, just for anyone who wasn't here at the very beginning, um cubelet tracing got in so um big thanks to Sally um for her work on that took almost a year for that, PR um got a.

B

uh So now we have API server, node and theoretically, a TD. If we ever version bump.

A

And theoretically container so container runtimes already have implemented some tracing so so.

B

Now we have like multiple layers of of tracing, should we start working on guides on how people can utilize this.

A

I think so I think that um hopefully should be part of the documentation um for cubelet tracing there. There are guides I, think the.

A

I think one of the issues is the older version. We're on means that it uses an old version of the format. So we probably don't want to publish it too widely until we're able to update the open, Telemetry libraries.

B

Okay, so we probably can't advertise this until one two six then.

A

Yeah um yeah we're still waiting on the SCD.

B

I, don't think we can rely on SE to version bump anytime, soon, okay, the state of the LTD communities. uh It's it's a really bad state right now,.

A

Do you think it would be worth it for us then, to ask them to update one of the older release trains like at CD, 3.4 or whatever we're on.

B

That would be a big patch right.

A

It would just be a dependency bump for open telemetry.

B

Four three: four.

A

Whatever another option is, instead of waiting for another minor release, to ask them to do a dependency bump in a patch release and then yeah.

B

But they don't, they don't have tracing implemented in three four. So you would need to backboard that patch right.

A

I thought it wasn't three four.

A

Either way, three four three: five: whatever kubernetes.

B

Yeah well, the question is: if the if the cubelet tracing stuff is in three four, then, yes, we can do your suggestion by asking for a version bump for open Telemetry and that that's fine, um but if the change isn't in three four I think we're going to be hard-pressed to get them to backboard tracing changes into three four.

A

um So it is it's three five do we know? If are we on three five of the FCD client in kubernetes, we're.

B

On the 3500 CD client, but the STD server needs to be on three five. No.

A

So I'm talking about specifically getting the go mod version in kubernetes kubernetes of open Telemetry, updated. That's the important piece for graduating these features to base.

B

Oh, that we should be able to do that. We should be able to do that because we're using W5, client, STD client.

A

I believe but yeah. If they're willing to accept that update, then we can get past this because we've been stuck for a few releases.

B

Yeah I think we should be able to do that.

A

And then, at that point the protocol is stable and uh we won't have to worry about um telling people to use specific versions of The, Collector or anything.

E

One question so I have mentioned the. There are two versions: the client version and server version for nsat, so before in my class, because I'm using three four, so I just change the ICD on the port version to a newer version, so it doesn't mean I need to change other thing to to support HD tracing or something it's enough. I just changed the sad part version from three four to three five.

A

So three five does have tracing you have to enable it via command line Flags. uh The only issue is that it doesn't let you control the sampling rate.

E

Okay- oh that's not.

A

Good you'll sample 100 of requests.

B

That's terrible that.

A

Is terrible yep, so I don't consider it usable in 3.5. Unfortunately, we.

B

Can't fix that either backboard.

A

It's a new feature, I mean we could ask again, but.

B

um We should we should fix that because that's basically a bug.

E

B

Sampling 100 of the time is a bug I.

A

Think perhaps what we could ask for is to change it from 100 to zero percent without making any other changes, because that means that only incoming sampled requests will end up being um yeah.

B

That that sounds like a bug fix that sounds like a bug fix so.

A

That, maybe is potentially another way we could address this. Okay.

B

I, like that I like that idea,.

A

I'll make notes um and Catherine do you want to collaborate together on working on these? Yes,.

E

Yeah because I have ever used increasing in my classroom now is it's amazing.

B

E

Yeah and currently I'm working on end of tracing for webhook for the uh mutate mutating and uh some of my Hawks here, because I want to see the latency for each webhook and find the web Hub, which are the slowest, are very slow.

B

I mean you can also use metrics for that, because we have. We have a metrics.

E

E

But I so I can know okay, so there are several different kinds of uh API requests, for example, watch and the other cans, so they have different uh values for words. It's very big, so I can see very clear to understand the distribution from this Trace data from for metrics I'm here. To think why some values are very big foreign.

B

Hooks for everything should be the same.

E

Easier to for visualization to understand the distribution for for the latency.

B

For the entire request, latency Maybe.

E

Yeah, maybe I can ensure a screenshot later to to say yeah I.

A

E

So currently, yeah even I, don't know the details about API servers. So I know there are different kinds of Apso requests, so I can stay very clear from the tracing graph, so there are one several cans of changing it's. It's then its value is about over 10 over one minute and other than this data in less than one. Second.

B

Yeah I mean I mean you have lists and watches I mean yeah those take a long time. But but what.

C

I'm saying is yeah yeah.

B

But but if you're looking at strictly the web hook, latency metric the web hook, latency metrics should be reliable for you to get the distribution of web hook lenses right because it's web applications aren't going to vary depending on a watch. Requester list request right like the web. Hooks are web Hooks and you don't watch a web hook. Request right. You, your Web book request, always just occurs: yeah yeah, it's just a synchronous request, yeah. uh Basically,.

C

If your web request takes long because it's a watch- and it's not because of the watch, it's because the web Pro is broken exactly.

B

But you can tell that from the web hook. Metric is what I'm saying yeah like you. Can just look at the web hook metrics and be like. Oh this one's hitting the top of the distribution. It's probably a massive web hook.

E

Yeah so I think there are several ways yeah. If we can correlate this thing together, it will be better, so we can first see the choosing graph to know. Okay, where is have some issue, and then we can say different, see the logs and measures in that point, as at some point.

B

E

B

I mean I personally, disagree like I've. Looked at these metrics a lot and I personally. Think if you just look at the web hook metrics and look for values that are top of the distribution bucket you're, basically going to be able to tell if you're going to have a problematic request.

B

C

Me tracing sounds more complicated to look at yeah.

B

C

Thank you. Thank you.

D

B

Yes, I I personally think that the simpler thing is just to look at the web hook metric. But that is me.

B

But yeah I mean tracing is nice to look at overall request flow yeah, but um but if you're looking for problematic Web book, metrics I would look at the web hook. Metric.

B

Okay, I think I think that's it for today, right I.

A

Think that's it for today thanks everyone for joining lots of good discussion. Awesome thanks.

B

A lot everyone.

A