Kubernetes SIG Instrumentation, 21 Jan 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG instrumentation 20210121

Description

SIG Instrumentation Meeting January 21st, 2021

A

Welcome to today's edition of sig instrumentation, it is thursday january 21st 2021 uh frederick, do you want to kick us off.

B

uh Sure, let me actually go backwards on our um on our agenda, because this one's just an an uh an announcement, I guess um so uh prometheus kubernetes adapter has. Finally, uh I don't know exactly what the.

A

B

It's been accepted for it's been accepted for a long time, but there were some legal issues in terms of the contributor license agreement or whatever it's called yeah.

A

B

um And yeah those legal issues have been resolved, so uh that's really awesome. This is, I think, we've had this uh in like pending state for three months or so so it's good to have this kind of closed. Finally, and I believe our next point is actually somewhat related, because I think marek did a pretty cool proof of concept using it is that right.

C

uh Yeah yeah: uh do you want to do those go for a second point with elena.

A

uh You're, you need to present right I'll. Give you a co-host.

D

A

Okay, you should be able to present now. Okay.

D

Oh and to present full screen- that's not very nice, but okay, maybe uh is it okay.

C

Yeah we can see okay, uh so let me discuss so. Let me talk about the demo uh that I did so hey to those who don't know me. I'm mark I currently own the metrix server, I'm one of the owners of magic server. So one of the common things and common question is: why cube ctl top? Is that way.

C

That yeah- and it only shows me and memory and nothing else. Why can it be more and all those questions uh so usually our answer was it shows that way, because we, which it's a way for us to display your core metrics, which are metrics, that we use for auto scaling and we don't expose anymore and after the last uh last week, when I got the same question about a gpu metrics on some, I think exactly premium adapter issue.

C

uh I finally got it why we cannot do this because no one like started work on it and no one like just did it. So what is the overall idea uh gets uh implement the new new those are just uh temporary like uh commands, but you implement cube ctl top metrics name can be changed. That gives you a list of all metrics that you have defined.

C

uh Those definitions already exist in kubernetes apis. Those are just the custom metrics that I think you need to to have them. You need to have a prometheus adapter that is configured or any other adapter that serves a custom metrics. So the only thing that I have uh is uh yeah. This is third but yeah. You have this api and that's all um so you get like uh all the information. So currently I have configured my adapter to serve some custom metrics.

C

Let me know if yeah increasing it will help and it adds like metrics about bots. So, let's see there are some based ones like uh pod cpu, but I I don't know I could display threats. Threats are imported, so cuba, ctl top usually has some has a way to define your custom columns. So you could technically get any any object in kubernetes and define your own columns and take any field. So I did something similar for the for cube still top, and so you define cube steel top normally gets you.

C

Oh those are not important because it's a demo and you specify and.

C

My demo will break.

D

C

Sorry, I can't remember and.

C

Okay, this should work, or I made a mistake.

B

And then the output says custom metrics and options.

D

uh What do you mean yeah.

B

So you've got options and they're the flag name is custom dash metrics. I think.

C

Oh okay, so yeah they're, pretty yeah thanks for.

A

C

I think the name doesn't make more sense. So with that you can, you can do things like that and.

D

Those are just.

C

This like this is nothing. This is the the result that we would expect with normal columns, just extend it to any metric that you defined, and I think that's all for the demo like I have a pr I can link, but not from this computer, and this pr, like is the poc.

C

If someone wants it like, it's amazing.

B

I think that's really cool.

E

Yeah yeah. That's that's amazing.

B

One one question that I that I, that I have- um or I mean uh so I think, last time we talked about coop ctl top. um There was somewhat of a consensus to do only server side, printing and just do so that we end up doing something like goop ctl, get uh pod metrics and then just use server side printing. For this I think this was even implemented uh so it like. I, I have no problem with changing changing our direction.

B

It's just a side note that we we had previously thought of a different direction if we can think of a way to do server-side. Printing with this, I would be super happy about that, but I'm not um married to that thought so.

A

Awesome so just to clarify uh like to ask the question, so this is all implemented at the uh like in the cube code. Client, uh it's it's on the client side, not okay,.

C

No no changes in like underneath, I can use normal ah sorry, I start presenting so underneath. I just have a prometheus operator with the adapter that I added some edited the configuration that is the default configuration um so that maybe one problem server separating that one difference in top is that it's the only really custom command that we have in qctl, which is when compared to get delete. That's the result of this. It's really custom uh and really like not touched for last four years. I will I removed last time.

C

I removed hipster implementation in it. So one thing is that having such pages requires touching with multiple apis, which can be implemented by multiple components so, like server side printing would be complicated.

C

Like integrative settings are pretty complicated, because there is no only component for this for this, because there is no single api when compared to other other methods that are because they are based on the api and not abstraction, of our metrics.

B

Yeah, I I think that makes sense. um As I said I am I'm, I don't feel too strongly about it. It was just what we had previously thought about. um I personally, I don't know how many people actively use the custom metrics api. I find that I I actually had this uh this topic in a previous uh meeting which we didn't get to but, like I find it awkwardly non-ergonomic, I tend to always just use the external metrics api because I just get to query exactly prometheus, essentially.

B

But um I think this I think this work is really cool and I think we should try to have something like this, because I really like the user experience of this.

C

Yeah so one benefit or, like the, I think, there's two major things that have a benefit for custom. Metrics first, is that I only implemented top bots, but you could have a qtl top deployment cube, ctl top uh namespaces nexus is just like. I just would need to copy copy and wk or like make it generic at some point, make it generic, uh and so this is out of the box, and second thing I think I don't know external api or, like I haven't, crafted, but uh for custom api.

C

It's enough to know about the object to build the proper query for external api. I was like my first idea was with custom. Columns was having a more uh dsl like similar to how adapter defines how it translates custom api into tremendous metrics having something like this more, but this is pretty to to to use to have user's site or, like lock, local machine, your own definitions of how you translate matrix, but this this this is somewhat problematic without yeah.

C

This is a problem because you change the cluster and it doesn't have some metrics and it doesn't work.

A

Sounds good, do you.

D

Have any other questions.

A

I open it to the floor on frederick's behalf because I have hijacked the meeting. uh I.

F

A

To do cap review if we can try to at least get some of that done, so we good okay. I will share my screen, not.

E

That one, I think this one yeah.

A

uh So for those who are following along at home, I just clicked on this open caps link, which is at the on the first page of our agenda. Talk, so we've got seven open caps and they're all in various stages, and I wanted to let people know that so previously in other release cycles. Basically the enhancements team would come to us, and so they would say, like hey, you know uh like they would check in on every single enhancement and in the 121 release.

A

That has become like a way too heavy touch for them because there are like, I think you know, there's like 200 enhancements in this repo or something like that being tracked right now. It's too much work for them, so they have asked all of the sigs to instead come up with their list of the things that the sig is going to work on and uh like the points of contact for each of those cats.

A

So we only have seven things: we're in pretty good shape, I'm also tracking the sig node caps and there's like 50 of those, so we're we're mostly okay, but we've got a bunch of stuff here that we need to go through and decide uh what we're going to get done in the 121 release, and I know that not everybody who has one of these caps is in on this meeting, and I know that these folks, for example, have been targeting alpha in 121.

A

So I don't know that we necessarily need to go over that one, uh but for all of the other caps here. Do we just want to go over them one by one? There's not that many.

E

uh Yeah, I actually have a question for something: that's not on.

A

E

A

E

Not on here, you.

A

Need to make an issue so we can track it.

E

uh My question is whether I need to make one of these. I don't know.

A

If I need to make one, what is your thing? Is it a cap.

E

Well, I don't know if it should be a cap like so the metric stability stuff is basically done, but we want to start promoting metrics to oh.

A

E

A

Think there should be a cap because we need to talk about like we need to go and poke other groups about picking stable metrics, and we need to provide guidelines for the stable, metrics and whatnot. I I think it should be a cap just because there's going to be a bunch of stuff written down, there's going to be a bunch of stuff discussed uh and.

E

So it would be better.

A

E

We've actually like we talked about one specific one last week, which was the etsy object, accounts.

A

E

Basically, I talked to apis.

A

E

Yeah, I did just make an issue for it. I.

A

E

Yeah, I did make an issue for it: uh yeah and uh you volunteer volunteered.

A

uh Oh, no, the so that that issue is in the kk repo. uh There needs to be one in k enhancements. If we're going to do a cap.

E

No, but why does that need to be a cap.

A

uh Because it's not just going to be the api server that we're picking stable metrics for right. It's going to be. We.

E

Don't choose cable, metrics, no.

A

No exactly we need to tell this, like various other cigs, how to choose stable, metrics and tell them they. You know should start doing that this release, if we don't have instructions for them to do so, like I kind of like we should have a review process for it right. I kind of don't want to just like open the floodgates and let random sigs pick things as stable, not understanding what that means, and then we get like stuck with bad metrics.

E

I I hear you except.

E

Like there isn't anything that we would be doing so like capturing right, it would be more of like a protocol as opposed to like this is what we are. It's not like we're going to build something.

A

Well, we need enforcement mechanisms for that stuff right, like do we have anyway.

F

We have, we have.

A

F

We built this they're already in place. They're done, we haven't.

A

Tested them, yes, have we tested them? We don't have any statements. They're.

E

Not their tests like we have, we have tests.

C

But we haven't added one, so, yes, first one will be still an experience.

E

I mean we, we can literally just test this. Without doing this, we can just I I I once.

A

Here's why I think that this should have a cap.

A

I think this should have a cap, because, while each component owner needs to pick their own stable metrics, we need somebody to coordinate that effort and we also need to make sure that, like we don't randomly miss something, because if it's like, oh yeah, we've got the api server and we've got like some stuff from the cubelet.

A

But, like everything else, you know there's no stable scheduler metrics, haha like that's, that's, not a good user experience, and so I think, if our goal is to go and do a bunch of stable metrics this release, we should coordinate that, in which case that has to go through sync instrumentation. So we should have a cap. It doesn't have to be super heavy weight cap, like I don't think, there's any prr review even required for this one, but we should have some sort of centralized tracking point for coordination.

A

And trying to do that piecemeal issue by issue in the kk repo is going to be really confusing for.

F

People, I guess, there's like two things. One thing is the guidance we want like. There are three things actually in my mind. One thing is like we want to give people in different component to like a guidance on which are the. What are the metrics should be considered as stable. That's the one thing, the second thing is: do we have like a centralized uh place to checking all the stable metrics if we want to promote them? The third thing is promote one specific matrix to for from.

E

Unstable one, so the centralized thing is already exists right, like basically, we have the static analysis thing and like as soon as someone bumps the metric to stable, like it will, like the the. I think,.

A

We actually have to approve those. Don't we did you set it up? Yes, okay, yes,.

E

So so, basically no one can actually commit a stable metric without instrumentation approval.

A

B

So the the two things that I think we should do is we should actually go through that process ourselves once and once we've done that we should send out um an email to kubernetes dev that this has been actually used now and people.

B

If people think there are metrics that should be stable metrics, then um they can come to sick, instrumentation and present them or something, and then we can work uh towards um towards adding them.

B

My my guess is that even like with um lcd object, counts, um we're probably going to have to revise almost every metric that um that goes stable in some way right, like even api server account. uh Like request total, um we already decided there are a couple that we still want to change so that I don't know, that's just how I see um what what we should be doing, and I don't really feel like that- requires a cap.

A

Yeah, I I it's like it's not like. I mean so like there is a cap already right, but we've already marked it as closed and done, but it's kind of not done because, like we haven't actually picked any of the metrics and we've been getting a bunch of flack from other people for like. Why are these lists empty? Where are the standards.

E

Yeah and that's that's exactly what we're doing we're. We have an issue for for elevating one yeah.

A

So we don't have a tracking issue and we like need to do coordination with a bunch of other sigs, and we've not done anything like this before and like the cube way of doing. That is a cab.

B

What kind of coordination do you see happening.

A

uh We need to make sure that everybody is aware of it and we need to make sure that, like we're tracking who we want to get stable metrics from and when they are done, so that.

B

Like we can call.

A

It feature complete right.

B

The the the cap is just about the framework. The framework is complete.

A

I know the framework is complete. That's why it's done, but the problem is, we haven't actually gotten the work gone and done the work to like pick the stable metrics. So, like all the end, users are like. We don't.

B

That's a continuous thing, that's something like that's something that will always happen like yeah.

E

Like like, you can create prometheus the framework and you're done with it. But if no.

A

One uses it. I know that we're going to make sure that this is done, but the other groups who actually need to pick the stable metrics are not aware of any of this stuff. They don't have the context. They don't have the history, they don't know what.

B

I'm not disagreeing that we should be evangelizing this. I think we should. I agree, but I don't think a cap is necessary for that. Yeah.

E

And also I, I am actually evangelizing it for api machinery for the bulk of their metrics, because I had.

A

So all I'm saying is: we need to write this down somewhere, because if we don't have a oh, we need to make sure han goes to api machinery. Alana goes to node like frederick, goes to scheduling. I don't know like we don't have any.

B

That part, I don't think, needs to happen like that. That would need that would mean that we need to do this on a quarterly basis or something like it's the same.

A

B

A

Think are you adding metrics in the first place.

B

Right, like we're not evangelizing that either we have, we have documentation for this and we should have documentation.

F

How would this sound.

E

Compromise compromise compromised position a little bit of both how about this.

E

How about we do the testing, before the next sig meeting sig instrumentation meeting, to ensure that the framework does exactly what we expect it to do? And then the next sig meeting we will uh appoint delegates to sigs to go to the that sig meeting and to have like a brief announcement about what they would have to do in order to get in order to elevate their metrics disabled.

E

Then, like we can just like people can volunteer to take a six and go.

A

So, like, I think, maybe uh what I've been saying is kind of getting a little bit lost in uh in the weeds. I am like not like excited about a cap for like the purpose of going and writing a doc, or something like that. I have just done this. It is like quite heavy weight right now. That's not what I'm suggesting we do.

A

The issue is that, like from a kubernetes release and end user standpoint, if like the goal, is to make a bunch of stable metrics in this release, which there is- uh and that is my understanding of like the sig's goal for this release- is have some stable.

A

Metrics uh then like there needs to be one centralized point for tracking that so like we can deal with release notes and like the enhancements team knows who to talk to and like that kind of thing and like if we don't like you, know, there's like no other framework for tracking something like that other than like at least an issue in this repo.

E

No, but but the thing is these are are going to be auto generated like like the release. Note stuff is literally a part of the framework we have already built that that part is actually.

A

No, no, I know that I'm not talking about like the specifics of like end users, contributing I'm talking about the release team. We'll have no way to know that this is going on. If we don't have a cap, or at least an issue, it doesn't have to be even like a new thing. We could just refer back to like the this cap exists.

A

We are encouraging caps to make sure it seeks to ensure that they're picking things for this release, uh because this list is empty and we're getting a bunch of bugs filed about it, which is true.

E

Okay, I can try to draft something.