Kubernetes SIG Instrumentation, 27 Jun 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG Instrumentation 20190627

Description

https://docs.google.com/document/d/17emKiwJeqfrCsv0NZ2FtyDbenXGtTNCsDEiLbPa7x7Y/edit#heading=h.fkr3c6ckd6pp

A

Okay, welcome everybody to today's instrumentation meeting. Today is June 27, 2000, 19 I believe we actually have a relatively short vagina. So if you still have anything you want to discuss, you cannot still put it on the agenda. So without further ado. Our only agenda point is discussing metrics migration control, plane stability, yeah.

B

So Peter can make it today he's still working on the static analysis piece, but it is progressing along and the prototype is also advancing sort of in parallel with it.

B

Insofar as the migration cap, we have LT TMS from basically all of the component owners except cloud provider, but I have been talking to discipline from cloud provider about it because there are bunch of metrics and the legacy providers. It's a little bit messy I, don't know how familiar you are with the whole I'm.

A

Pretty much not not familiar with that piece of code actually is that in tree in like kubernetes hi.

B

Yeah so there's a folder called legacy providers. Basically like AWS code as your code, google code is all vendored entry. There's an effort led by sick cloud provider to extract these things out of tree.

C

B

Yeah yeah they're, gonna, they're gonna, create their own controller manager, got it okay, but it's also interesting because.

B

I'm not exactly sure what the architecture is for the cloud controller manager, but at the base level it presumes. Prometheus and you'll see a lot of Prometheus metrics in a lot of these cloud provider code bases, but when they extract this thing, I'm not sure. If that is going to be a valid assumption, ah I mean.

A

If, if they're, if they're still doing it under the permit the kubernetes project, then that's the standard way to go right like why would it be different.

B

Well, this is specifically the cloud controller manager right, so this is always going to be.

A

Clouded specific, but people still have their own opinion about this right and I mean kubernetes. Has some opinion about this and I feel like we should be consistent. I'm I think it's fair that those those provider, specific components could have additional things for their specific monitoring solutions, but I feel like we shouldn't force people into that. I.

B

Was actually thinking, maybe it might be worth doing an efforts similar to what happened with client go sure yeah that wouldn't work for sure yeah.

A

B

Because, like Glencoe obstructs away all the Prometheus detail sent in juxtapose. Similarly, if we did that, we would have two genera size, the metrics, but they actually don't have generic metrics right now. So, like the cloud provider metrics, if you looking like a super fighters, has arbitrary math metrics all over the place.

C

B

C

Those exported those on the controller manager of metrics endpoint.

B

Control the manager binary I, don't like ice more I'm, not sure what the status actually is today. I might actually just be the controller manager binary. But later there will be a cloud when you limit I can show.

C

B

Yeah, so it is currently queue, controller manager and I think it's all just entry, but they plan to take it out of tree and make another cloud controller manager binary.

B

A

That means that this becomes kind of an interface in each cloud provider implements their own Club route, controller manager, control.

B

So yeah there will be a controller manager. Yes, it a cloud controller, monitor or per side so, but each cloud will have a cloud controller manner. Time.

A

I don't know my my feeling is that Prometheus format or eventually open metrics format should always be available. That's I understand that it's definitely about clearly bound to some cloud, but allowing the portability to be a little bit. Bigger is kind of kubernetes promise right.

B

Okay, well, yes, but maybe we should attend one of the stick club with other meetings and actually ask them what their is. I should probably agreed their design doc and try to understand. What's going on because it's hard for me to say what would make sense, I think the client code thing might make sense, but again I'm not sure this is a I mean like yes, there is like kubernetes land stuff, yeah, it's hard to say it's, it's hard to say, there's like a lot of very specific cloud things do that.

A

Makes sense like I feel like there, there will always be cloud provider specific metrics, and there should be because the clouds are somewhat different, I think what you've? What you were talking about this, like maybe a kind of a skeleton for cloud controller managers right then they.

D

Only insert their own logic and maybe.

B

They'll have two metrics endpoints right, like you have the generic one and then you have specific you.

A

Should be able to hook into that.

A

B

They'd have to use a library.

B

Yeah I guess we'd have to think about it, but in terms of the migration it shouldn't affect the migration path. Mostly, we can just migrate over their metrics and things should work I'm a little bit concerned about cubelet metrics. The only there's, there's four culet metric end points.

B

Most of the cubelet metrics use non-standard collectors.

B

Which is a problem because we don't really have wrappers for non standard collectors.

B

So there's a couple different ways we can go about this one is I, can modify the framework code to register non-standard collectors and just do the normal Prometheus thing like it currently doesn't allow that we only accept you know these new metrics with the metadata, but we can, you know open up the API to accept just standard Prometheus collectors, and these will not have any metadata associated with it.

B

And yeah it won't take deprecated version, so it would just be it would just act normally, that's one way. We could do it for non-standard collectors. Why.

A

Why could a collector as a whole not be like alpha or stable.

B

Because of the way that they are defined so currently, then the metadata is associated to the OP struct right.

A

No I'm not talking about the way the code is right now, but, like fundamentally does fundamentally, we can wrap the.

B

Dispar meatiest description, yeah.

A

B

It's a little bit more involved because that code is less.

B

I think it's less written for this sort of thing.

A

It's I mean like transfers.

D

These things via channels.

A

Right and we should intercept the description and at the stability level within that know, we.

B

Can yes, we can do it, there's quite a bit of plumbing for the description, stuff, yeah.

B

And their use of internal struck description objects. It's also a little bit problematic for wrapping it. It makes it a little bit more complicated. We can do it. It's definitely possible I.

B

Mean we may want to attempt to do it, we're.

A

Primarily talking about the CEO Pfizer collector, no.

B

Well, it's not just the adviser, it's like the so people. It has four end points. It has slash, metrics, slash, metrics, probes, slash, metrics, see advisor, slash, metrics mister! Is it what is it a resource resource? It's resource, slash metric to source? That's the new one, I.

D

Don't even know about that one. What does that one expose? It's the one that David Asheville's pretend to discover: okay,.

A

C

I just liked it in the chat. Oh.

B

Yeah yeah: this is yes, I, always honestly to find this vial I, always just reference metric, slash probes, because it's one of the few places that you can find it and then it just takes you to this file.

B

But yeah, so there are four metrics and points of slash. Metrics endpoint also uses a bunch of non-standard collectors, so the only easy one is the slash metrics process.

A

Because that one just uses normal tanto metrics or engages yeah.

B

So I mean like look if I add the hook to just accept regular prometheus collectors, then we can migrate all of the qulet metrics and then we can be fur wrapping. The the description, slash collector API to handle non-standard communities collectors in the future and then my good tequila it stuff separately. I, don't know how people feel about that. I.

A

Mean like how are we thinking this to work with, like the collectors that come with Prometheus client, going like process metrics and go runtime metrics, for example, I.

D

Submitted a peer to you right, but I'm.

A

Talking about like do we want to mark those as alpha no.

B

Those are just by D they're, not wrapped at all. We don't they're, not wrapped it off, which.

A

Means they're: they.

B

Just don't have any guarantees right.

A

D

B

Out, but we have not wrapped in these alpha.

B

Those are those also use non-standard Prometheus collectors. Yes, so, basically, if we were to solve the culet problem, we could solve that problem. Yes,.

B

I would personally like to get the codebase migrated as soon as possible. My goal is to get it into 1/16 and it looks like we have the requisite permissions to do so. I have a breakdown of cats.

B

And then we can do some of this stuff like after the fact right like it should be. It should not affect component honors and they seem to be okay with this thing, so yeah I mean. Does anyone feel strongly about doing it? The other way or I.

A

I think that sounds fine.

B

Elena woody, you think, giving me thoughts about it. Her.

C

Strong opinions, yeah.

B

Okay, yeah I mean I just would like to get the code in and then it would be I think simpler to do like more surgical changes in the future is that I was thinking yeah.

A

I mean, but then what I'm hearing is. We will just not be annotating any stability foresee advisor metrics, for example. We just only find their if they've broken very significantly multiple times, but we won't even see the Alpha one right if the Alpha annotation on it so I just want to make sure that we appropriately communicate what that means. So.

B

Okay, I, okay, you know what actually it might not work to do it that way, and the reason why not might not work is when you have non-standard Prometheus collectors, it's very hard to verify.

B

The data structure ahead of time.

A

Yeah I understand what you're saying like we can. We can really review metric names, labels and so on, like it's a dynamically.

A

That's the whole point of the collector interface right, like that you can make this dynamic so.

B

Alternatively, then maybe what we want, maybe a simpler thing to do, is to wrap custom descriptions so that we just annotate alpha on all of these custom names, because if we were able to do that, then any custom Prometheus collector in our registry was just always alpha, like would always be alpha, and then people would know that they don't have any stability guarantees for these permits.

A

And the way I see it, is it's sufficient to do this just for like to just intercept the description and just pass through for the actual collector stage right? It's like there is.

B

A pass through.

A

The clock stage you just intercept the description too, and the the only the only thing that happens. Is we prepend alpha to the description right, yeah yeah? That's! It should be pretty easy right, yeah that I don't see a problem with that really.

B

Yeah, okay, that should they should probably be fine.

A

Okay, yeah, let's and ultimately it would see adviser D. We want to remove that from the code base, anyways right and then it wouldn't fall under these guarantees anymore anyways like it. It should have its own stability guarantees, but they're, probably gonna, be distinct from these.

B

Yeah but the other Kubla metrics also have non-standard collectors, so even if we get rid of C advisor, it's not we're not we're not rid of this problem and people. People do this stuff all over the place where they don't even need to do. It I've seen non-standard collectors where people can be using gauges so like as a part of this migration, I'm, probably going to be turning these into just normal Prometheus metrics, because.

A

There's one that I can think of where I'm not sure how to solve it. There's a collector for log sizes that looks at container large sizes and exposes those as I'm not sure, as gauges or counters, probably gauge.

A

Yeah and basically I think it pretty much literally does like a DF or D you on the on the directory of the container logs. But can you just use j j-- funk.

B

Because I mean like, if you rap, because the thing about using gauge funks and counter funks is that you can use the OP struct and then this this funk gets called at collect time. So you can do all this this. This, like you, know, fancy Prometheus non standard collection, stuff, and you still have the metadata that you're used to yeah. You actually like ya, know you're, probably right yeah, so I just put a bunch of these things. We can just change them to be compliant yeah.

B

So long as the the metric metadata itself is not dynamically inserted. Yeah.

A

So, which see advisor does do right.

B

Yeah, because the container stuff is obviously I mean yeah short-lived, whatever yeah so yeah anyway, that was the status update. Sorry I took up so much time. I mean.

D

We don't, we don't have anything else.

B

Or did anyone add something? Sorry I did want to bring something up, though.

B

Okay, so I added something to the agenda.

B

On the top, so.

B

So 116 has opened and I have put the enhancements freeze and the code freeze dates. So basically for this cat. We need to get it into implementable state by this date, but it sounds like we were there already the static analysis piece, so we have like a month so I will let Peter know if there seems to be other people in this chat.

B

As for and Pablo and Elena and hi.

C

B

And so, if anybody is interested in taking doing something for this release cycle, there are things to do, and I am happy to talk to you about potential things. If you are interested in contributing, there are many metrics problems and it would be great to fix many metrics problems. So we have these important dates coming up, basically a month until the Henson story, so there would be one month to work on a cap if you wanted to get something in this release and then two months to get the code out.

C

What's the status of deprecating like direct access to the sea advisor metrics.

D

A

Think we are still stuck at having some replacement right. Is it resource posted beautiful resources was supposed to do it, but it is far from graduating right so like we need I think before we can actually remove it. We need something stable that everybody regards us being stable, yeah.

C

Agreed I guess that means I think that I haven't actually gone ahead and put in a PR to remove the old labels on some of the sea advisor metrics. So if they're not getting ripped out this release, I should probably do that, because that did not happen for 115 I. Believe.

A

We kind of said that that would happen for 1.17 right we're. Removing the lid innately.

C

No, no so there were there's duplicate labels right now and a bunch of the sea advisor metrics, there's container and container name. They previously were just like container name in pod name, which didn't match anything else you couldn't do joins on like the sea advisor metrics and the cube state metrics metrics. So that was a pain in the butt. So I added like labels to match the instrumentation guidelines and they're just duplicated right now, and we said we would deprecated them in a future release but which future release wasn't.

B

Break people I mean like what, if somebody's behind the other one, yes yeah,.

C

So we gave them like a long period to to migrate, so we added the duplicate labels I. Think in 1:14, with a changelog release. Note saying like this is changing. Please migrate. I saw that sumo logic had already put in a PR that, like it's changing this for them, so like people are aware of it. Yeah.

D

Open shift has changed as well. Let.

B

Me talk to the know team on our side, because I want to make sure that they have migrated to the correct labels. um Oh.

C

Yeah, so that got changed so that's the duplicate. Labels were available, I think in both 114 and 115 and I just wasn't sure when to submit the thing to remove them. If we want to do that in 160, in ER, yeah.

A

So, like the rest of the I think there's a thinner line here than the other things in the metrics overall, where we have actually two metrics right for those we said we will disable the deprecated ones by default, for 117 I. Think is what we landed on right, huh but we can't do that with our Esley.

B

This one, this one is actually a backwards, incompatible change, yeah.

A

C

I mean folks I've had now two releases worth of warning, so I just don't know when it's reasonable to cut the cord. At this point.

B

But I mean so heavy. Maybe we should attend a Sigma note meeting I! Think that's reasonable.

B

You know mostly because of the deprecated metrics. We are caught a lot of people off-guard and I.

B

It probably would be nice to be a little bit more proactive it so that people yeah.

A

I mean the kind of the feedback last time when we actually tried to merge. Removing everything in 115 was, like everybody, jumped up and immediately said no right when we actually wanted to do it. When we planned everything- and we told everything in like we told it publicly in the community meeting so clearly, people have chances to hear it. I, don't think everybody goes to those meetings. Well at least cichlids should know what's happening, even if they just read up on the notes or something right but yeah, but know.

A

Basically what you're saying is true people don't do that and so I think for this particular one I think attending a signal meeting and telling everyone and making them aware. Let's say we're going for 116 that that's a hypothetical right like let's say that we go into their meeting and we actually make sure that they know about this. Then I think it would be and they agree, then we can go ahead and do it for 116, okay,.

B

I think that sounds reasonable awesome, but yes, we're running out of time. Yes,.

C

B

Wants to work on extra projects before they kept deadline. Please reach out to me I'm, happy to yeah.

A

That's maybe maybe hon and right now we can chat offline to see what to do with sick node. So we can move forward on that because, ultimately, that's an improvement for everyone, because we can actually do the joints without doing label rename people.

C

Can do that now? It's just yeah! It's, let's get rid of the old things at some point. It's.

A

A lot of network bytes transferred very unnecessary.

B

Yeah I'm just worried about breaking ingestion pipelines, yeah.

A

No, no I get it it's the same story: yes, yeah yeah everything else: yeah, yeah, okay, yeah we're already one minute overtime. Sorry, everybody for me being late and I will try to be earlier next time. Yeah. Okay, then have a wonderful local time and see you next time all right.