Kubernetes SIG Instrumentation, 18 Mar 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG Instrumentation 20210318

Description

Bi-weekly SIG Instrumentation Meeting, recorded on March 18th 2021.

A

Hello, everybody and welcome to today's edition of sig instrumentation. It is march 18 2021.. We have some items on the agenda today and I shared a link in the chat. uh Let's see, what do we got going on a reminder? Test freeze is march 24th. That is next wednesday. So in less than one week from now uh as well another announcement, we have the 2021 contributor survey, uh which uh sig contribex has asked us to send to everybody.

A

uh So uh we have not sent this to all kubernetes users, because we don't want to hear from users. We specifically want to hear from contributors.

A

So uh if, if you are here, you are a contributor and this survey is for you, uh so please check out the link fill it out if you haven't had a chance to already or very excited to hear from our wider contributor community.

A

Next item is han and a logging working group come on. Take it away.

B

Did I put it in here.

A

I put it in here on your behalf.

B

Oh yeah yeah, so uh so, basically, we already have kind of an informal logging working group, and I was thinking that it would make sense to make a more formal, structured logging working group uh because it's been sort of a gigantic effort and elena and merrick have been working quite diligently at it with a lot of contributors. So I I feel like it would be. You already have the the triage board so should should I just like look into form formalizing. It.

A

Yeah, so I agree with all of the above, and I would add to that that uh I mean we have a like. You know this is kind of uh having talked to a few folks. This is kind of the ideal case for a like chartering, a working group, so we have like a fixed effort. It's got like a target end date. It requires, like you, know, additional participation above and beyond the sig. It's not necessarily like needs to be shared.

A

Every update uh in the sig's day-to-day business uh so uh and the the key thing is, and when we charter a working group, we could bring in a bunch of like new leadership and folks who, like have basically volunteered to steward the whole thing from start to end. uh So I think it would be really great to put out that call. uh I know I've spoken with some folks at companies that, like previously haven't necessarily been big, instrumentation contributors uh but say they would be interested.

A

If, like you know, we we have such a group in like helping out with that. So I think for like visibility's sake, it would be really good to spin that off. So I am like strong, plus one and there's a whole process. We have to go through to charter it, so so.

B

Yeah I I'm down, I think it makes sense, but uh I think like in terms of a lead like the people who come naturally, to my mind, are either elena or merrick, but I I I suspect, elena.

A

I don't know I will have time to like own this end to end, so uh I I would like to see you know like, let's put out a call, let's see what happens um and I mean that's all part of the chartering process. uh If you know, if nobody is willing to share it, uh then that would be bad, uh but I don't think that I have time to own this end to end because, like logging is not part of my day, job.

B

So merrick I hate to put you on the spot, but is this something that you are interested in.

C

Yeah, like I was I'm currently getting out of one uh uh one effort and hoping to to to invest more in structure logging, because definitely it's something that needed something to mature, but we, I think we should, and I I want to get ready.

D

To like to fully deliver it to okay,.

A

And I don't think we need to put anyone on the spot like we can uh like. Certainly, I think we're gonna need more than one lead of the group.

E

A

It sounds like you know: I have interest in it existing, although I don't want to chair it. uh Merrick also has interest in it existing and may have time in the future. I think that's probably enough to at least sounds good start investigating so yeah.

A

Is there anybody.

C

F

Else is interested in.

A

Like participating, helping uh we'll make sure to send all the info to the mailing list and.

B

Also, like we have attendees right now, uh people are filling them out. So if you are interested, uh maybe denote by your ldap that you are interested in this working group and we can send out invites or targeted information once we get the logistics worked out.

B

How does that sound good yay.

B

A

Cool uh next is me: um I got pinged on a pr for the website uh for stackdriver documentation and I was like. I don't think we support this. We ripped out the end-to-end tests, uh but somebody was like just trying to update the page to make it more accurate and I think contribex or some other group came back and we're like uh you've stuck sig instrumentation.

A

um So do we want to support this? Should we remove the page? uh Like I mean some very kind? Community individual has gone to update it, but I think we removed our end-to-end test, so I don't think it's actually supported by sig instrumentation and it would be better for that to not exist in the official kubernetes stocks.

C

I think that original idea was that uh when the class or kubernetes cluster exists, there was a place for add-ons and first-party integration. To start uh just ended up with two solutions. One stack drivers, elastic search stack driver is mostly like not maintained. I think there's still two people that maintain elasticsearch, but with the application of cluster and like not really um huge focus on vendor documentation, and then I don't think there is a big uh yeah.

A

I mean generally.

C

Like my concern is.

E

A

Is almost 100 vendor-specific documentation at this point, like people are adding links to cloud.google.com?

A

If that's the case, they should probably live in the google docs. It should not be in the kubernetes stocks.

C

Yeah, like half of this, I think I drove it through mentions, like other integrations or other solutions and they're not killed, because I think elasticsearch was not completed or documented. So it's uh yeah, it looks like google or like driver anything else, but there is no anything else.

A

Yeah, so what do we want to do about these docs.

A

I think we should remove them, because I don't think this is supported.

B

C

I think merrick is the right person to to answer that, so I talked with the person who altered it, travels and like he would prefer to have like some references to how like, uh but if it like, he understands that, if that's uh like vendor specific things shouldn't be good. He's. Okay, with removing.

C

A

I'm just I'm trying to like look through uh for for those who are following along, but maybe not looking at the agenda uh here is the.

A

A

And yeah like there's an entire file logging stackdriver.

A

And it's basically like.

A

Pretty google specific already and the update is making it more google specific- and I don't know why we would have this in our community docs, and this is a very old document. This is like written from folks who haven't been active for like two or three years. I think we.

B

We actually have a lot of uh cloud provider. Specific docs like there are sections of kubernetes documentation which is devoted to like azure logging as well as stackdriver. There there's like basically flavors for all of them, so I.

B

I'm not sure like. If there are other logging ones, then I would say that removing it doesn't make sense, but unless we remove all of them.

A

Yeah, so specifically, the comment that we got from the docs team is, I believe we are trying to exclude vendor-specific or dual-hosted content on the website, and this is all pretty vendor specific. I.

B

See then, then we should get rid of it right.

A

That is what I am calling for.

G

Yeah, I think that makes sense, if possible, is there a way for us to leave like to remove all of the page but link to somewhere else for people who still rely on links to that doc, or something like that.

A

uh I can put a question to the docs team about that: okay,.

B

Oh, what about why don't we just throw it into our instrumentation repo? Under a doc's thing, we can have a doc, slash cloud provider.

A

Because then someone would have to maintain it.

H

Yeah, I was just gonna say I I think actually, the um the owner of these is the cloud provider specific sig. No, like I actually kind of kind of wonder why we were pinked on this. In the first place,.

A

We were pinged on this. I think, because there's some like metadata on the page that says it's us or was us at some point, uh because it's.

H

A

H

Because I think the the e2e test, or whatever it was, um we did transfer a bunch of things to the um like google cloud sig or whatever it's called. I think there's one for every major cloud right, yeah.

A

I think we transferred they're no longer separate sigs but like they are there's like, I think, they're like you, know, sub projects within the sig, so there is like a google cloud sub project within sync cloud provider. So what I will do is I will respond to the docs person who says we should be removing the vendor specific stuff saying yes, we are okay with this at a sig level. We don't think this is us. We think this is cloud provider. They should weigh in. How does that.

H

Yes, I think that makes sense, yeah, perfect.

A

Cool, that's, I think all I need for that one. uh I can put a thing saying I will follow up and then next on the agenda, we have d grisoni.

A

Damian the floor of yours.

F

Yeah, can you hear me so I wanted to start a discussion basically around pod matrix. For those who don't know, prometrics are like another community resource uh behind a cheap, catalog pod and we've been reported uh two days ago.

F

I think that uh the values that were reported by pod uh or some kind of drifting from um the matrix uh from the advisor for the whole pod and the reason behind this is that uh first uh for prometheus adapter, we were not taking into account the post container that he started into every pod and also um like clayton mentioned that in the board.

F

There is also like some processes that are started and all this tasks like memory and cpu usage of these processes are charged into the c groups, but we never really took them into account uh in the phone matrix, so they are not really really taken into account.

F

Whenever we see them, we skip it on the pod or even for autoscaling, and the current problem we have with the parametric resource is that it there is only a way to describe uh container based metrics and not like all the other metrics that we can have like the one for the other processes so yeah. My my question is like.

F

Would we want to update the resource type to include some kind of higher level metric for the old pod, uh in addition to the one for the containers because, like we are, we already have like these metrics either coming from the advisor for premise, adapter or in the case of matrix server? We have them under the api resources, endpoint of cubelet, so yeah. I wonder if we want to update that and how we would do that.

H

Let's say we added new fields to the api. I don't remember exactly what the api looks like, but it feels like adding new fields that describe the pot as a whole would not be a breaking change. It would be forward compatible.

H

um So that seems to me, like the right thing, to do to have exactly what you said like a representation of the entire pot and then the individual containers, as we have them today.

H

That sounds reasonable to me, but yeah go ahead.

C

Okay, I I would say that the reason that we only monitor or the metrics core metrics api only lists containers is that the use cases that we have for this api don't really care about the overhead of running the bots so that slip like yeah any container some processes and a slip container, and that holds the network stuck.

C

uh The the reason is that those core matrix api is going to be used for up skating so like when you auto scale- and you know memory of the pod- your requests, you don't care about the overhead because you just want to adjust the the container itself.

C

So here we go go more into what would be that benefit to adding this like to expand this api. If this api, like mine, use cases are, uh are auto scaling or like I, I treat cuba ctl as a after thought to to be able to look into auto scaling pipeline, not us like main dedicated.

C

Thing to check what are popular resources, because uh you can always look at grafana or any like your monitoring solution to check. What is your overhead.

H

I I get what you're saying, but I think that's not as obvious to end users right like I, like you and I know and understand this, um but or like everyone in this call understands this right but, um like I feel like I understand where people are coming from. Like I don't know the entire context here, but my assumption is that someone opened a bug against openshift or something and said wait a minute. This pod actually uses more resources than kubernetes. Tells me, and I and I kind of understand that confusion right.

H

A

Would any of them be addressed by the pod overhead cap? That's going on. I think it's in beta.

G

So I want to clarify one thing as well, because um I was confused about this. The the pod level metrics already exist in the summary api and the cubelet resource metrics api. This is about um the actual cluster level resource metrics, api.

C

Yeah yeah so, for example, like I think, stackdriver already at some point, definitely differentiated those two values. It's for more for prometheus and instagram like to monitor your yeah, to build a monitoring system, not and not something that concludes into the basically metric server like this is like the amazing metric server output and auto screening um further going into it. Do you think from uh to comment on your on? What you mentioned is that I noticed there is definitely lack of good the documentation to keep ctl top.

C

uh At some point, I I think I one of the my first contributions was trying to add something there, but I don't think anyone from cli maintains that well and it's hard to do it, and it was not changed for the last like three at least years to to to make cube ctl prop.

C

Like explain what this does where, like it's not extinct, what this does where the data is, why we collect why it is this data, so I think there is some improvements to be done. There.

H

Yeah, I I believe the the last contribution might might have been even me changing from uh heapster to these apis and I think back then like 60 li or I don't. I don't know if this sick even exists um said that they wanted us to move away from group ctl top to make this a generic thing to use like server-side printing eventually, so I don't know what um what that sig's idea is today.

A

So, just a quick time check, we have eight minutes left in the meeting and we have a demo on the agenda, so I want to make sure there's enough time, for that is this something that should spill over like? Should we continue this discussion in the next meeting? Can we maybe continue this in uh slack, so it's a little bit more visible as well.

H

Yeah, I I think of an issue. Inevitably, we need to kind of involve um six cli anyways so, like we probably just move in circles here so yeah, I think let's go ahead with the demo and move this to slack for now.

A

Awesome: okay, thanks folks, uh brian the floor is yours. Are you gonna need to share your screen.

E

A

Please, oh han, I think you have to make him co-host, because I cannot.

B

E

I could start talking, uh while, while we figure out the screen thing, um that's good, so yeah. I want to show you a thing which I wrote called caseband.

E

The uh so the basic idea is that I I like distributed tracing uh like jager and uh I spent a bit of time trying to put some tracing into various bits of kubernetes and then this occur. This idea occurred to me that kubernetes is already emitting events and a lot of the times. It emits an event at roughly the same point where I would want to span so. Can we do that? um So? Can I share my screen? Yes, I can share my screen.

E

Okay, uh so for the purpose of doing a demo, I have a little um deployment here. uh Can you can you see that? Let me uh you might.

A

Want to make a little bit bigger, but it's good! You.

E

Want it bigger.

G

Okay, let's go bigger.

E

I was trying to get rid of everyone's faces because I know you hate looking at your own face. Well, I do anyway, but um uh okay, so uh deployment's got two pods in it and I'm just using this thing: uh we're not even gonna. Look at the pods, I'm just gonna change the version to make it generate. Events um and I've been doing a few of these before the these are the kind of events that we get. You know we get.

E

This uh created container scaled up scaled down whatever um and like that coming out of cubekiddleget.

E

Well, a they come out in the wrong order uh b, they're impossible to understand, and anyway so, let's um make a change, uh so that would have generated some events and if my program's working uh yeah, we have some spans coming in to jager. So I got a few runs that I did earlier half an hour ago. Let's just update that okay.

E

uh So uh this is what it does it. uh It takes all those events and turns each one into a span.

E

The parent-child relationships are derived from the owner refs, so each object each event has an involved object and the involved object may have an owner, ref, owner, f1 or f. All the way up here is that big enough do I need to make that bigger.

A

I think that's pretty good, bigger is also good, and also this is awesome. Thank you for demoing this.

B

You're using my.

A

B

Or something excuse me you're, using under owners as a heuristic for for connecting the spans.

E

Yes, uh I would go as far as to say I'm using a ton of heuristics.

B

Yeah, it's super interesting.

E

um So we can click on these things and I I pulled out some stuff. I mean notably the uh the message.

E

uh Yeah, so I am doing a talk about this kubecon.

E

uh Thank you very much to those of you who are on the selection committee, um but I did want to. I did want. To put it, I mean what I really wanted to do was put this in front of a couple of people quietly and get some gentle feedback and so on. But I was persuaded to come to your meeting and just show you it fully. Do you have a repo? I I just want to take a look at the code.

A

I think in the meeting minutes I have a question which is events are best effort. So what happens if one doesn't show up? How do you.

E

Yeah well yeah best efforts and heuristics are or yeah um yeah it's it's it's not it's not an exact same yeah there. I could talk for quite a long time about what what doesn't work so well in the event model- and you know I'm kind of I'm kind of taking lossy information and trying to put detail back in and I feel dirty, but it looks awesome.

E

No, it's it. It's.

B

Yeah I mean this is amazing: you you're you're building something given some very, very narrow constraints and it's very creative. I have to say just very.

E

B

E

B

E

There is almost no knowledge of deployments and replica sets and so on in the in the program.

E

um I I'm also testing it for cluster api, but they don't really issue enough events, so I'm having to get a little bit more creative there, uh but, for instance I mean I I never I never ever even tested it on a stateful set, it works, it works fine.

E

It basically works on anything that issues events and has a involved, object and owner ref.

A

Yeah for me, if more from, like the you know, sort of traditional operations or sre perspective, uh one of the things that is really challenging is like trying to make sense of events, and this gives like a really nice narrative. Yes,.

B

Yes, it finally makes events somewhat usable like there. It's it's been this functionality, that's been in need of a use case because it doesn't satisfy any of the things it was originally designed for, but now it's possibly useful, which is much better than I have no idea why this exists.

B

Wow, this is terrific. Thank you.

A

So we have one minute left: do we have any questions uh for brian.

B

I just think it's awesome. Yeah super cool.

G

B

C

G

Almost no specific knowledge about controllers. What's the little bit of knowledge you had to sprinkle in to allow to.

D

G

E

Yeah is it oh? Okay? I I don't even know my own program.

E

Basically, the yeah, so we these these events come with the wrong involved object. Can you read that the the um uh these things that are like scaled up replica skill whatever um so we we need to correct? We need to kind of we we. The thing that we actually want to hear about. Is this thing, but the event is issued on the parent as the involved object, um and so it's not even entirely like. Should that be fixed possible yeah, I I you know it's a it's. It's one person's bug is another person's breaking change.

E

So uh well I mean yeah. It doesn't have to be like hard quite as hardcoded as this you, you can sort of recognize the pat you know. If you, you sort of see these names, then that's the one you're interested in I don't know, um but that that's the th, that's the only place in the code where we kind of hard code deployment and stateful set.

A

Cool well, we are a little bit over time, uh thanks so much for demoing. This is super great. This is very exciting. uh It sounds like uh community interest is also here. uh Everybody is like very excited about this and thanks everybody for attending today uh it was great seeing everyone's faces and I hope to see you all in two weeks. Good luck with test freeze next week.

B