Kubernetes SIG Instrumentation, 28 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG Instrumentation 20211028

Description

SIG Instrumentation Bi-Weekly Meeting Oct 28th 2021

A

Hello, everyone- and today is the october 28 2021 edition of sig instrumentation. We have a few items on the agenda today. uh We will start with uh action, item review and announcements.

A

The announcement is code, freeze is coming up. uh It is november 16th uh only like three weeks away. So don't let that catch you by surprise uh action. Item review. um I had been tasked with uh an ask to see if we can disable prs or issues in repos like kubernetes, metrics or other things staged out of the main kubernetes repo. So people will stop filing issues there because we found a bunch there.

A

There are currently no issues there from what I can see. I did talk to contrib x and they basically said uh it's not possible to disable prs. That's not apparently a feature of github. uh We could turn off issues, but that's not something that they currently support with their uh like get ops, configs or something like that. So uh there's a bit of a discussion about it, uh but like no obvious solutions that we could turn on today. So uh that's that any questions.

A

Han, you said you had a couple of items for the agenda. Oh I see it's on there. Now cool bunch of mentors are getting promoted, stable, awesome, uh we'll come back to that later. uh Next, we have sally with cubelet, open telemetry tracing cap.

B

Yeah, so I wanted to bring attention to the um the cap. I'll just share my screen just so I but yeah the enhancements up um for 120. It would be 124 right, which is a while away, so I'm not in danger of missing it right, the the enhancement freeze. I always okay.

A

No danger at all the investment will be in like january of next year, yeah.

B

Oh yeah, cool, okay and can can I share my screen. It says I'm disabled. Let.

A

Me give you a co-host all right, cool.

B

Because I want to show you I have um I did hook up so recently. um Tracing was added to cryo and I I started a cube admin cluster with cryo enabled, and then it was at cd and api server because they also, you know, have experimental tracing but like the chain was broken because cubelet. So that's really why I wanted to add cubelet. I just wanted to see um the chain getting growing. I guess- and uh so I was able to to enable that in okay here we go. um Let me find the right tab.

B

It is not showing what I want to. Let's see, sometimes it just shows up. If I move things around just a sec.

B

um So I have some choices that I want. I would like to show you guys yeah. I guess this might work.

B

Do you see the sig instrumentation notes or no yes, and also your calendar? Yes, and you see jager now.

B

Actually, let me just move it over then I'm going to stop sharing.

B

And I'll move it to my other monitor, I had the same problem yesterday. Well, it's not a problem. I just I think now you can just see eager yeah.

A

B

Cool um so yeah, just I have it running, but I'll I'll pull up the um the cap.

B

It's 28 31.. It has been reviewed a bit, uh but if you guys can take a look, uh I definitely I'm not like an expert with kubernetes architecture. um I think you guys know more about that than I do so. I have questions like. Where does it make sense? Alana I I did um elena, I'm sorry.

B

I did um take your feedback and the api review too. So that's that's in there um pushed now um so yeah and then, along with this, I also have the pr and that has been reviewed more than the cup, and so you can find it easily it's through the cup um and okay.

B

Let me show you some traces and then I want to talk about. I can just go into the second thing on my agenda too right.

B

Yes, yeah yeah yeah, so you can see with um api server cryo at cd and cubelet. uh Like I say you know you want to look at this one. You can see. Oh if there was an issue it looks like you know: cuba is taking a lot longer than cryo like this is what I need you guys to help with that.

B

This group- um I don't have like the context for like why is this gonna, you know change the world like what problems is this gonna solve at kubecon, I watched the correlation, the the correlating data um presentation, if, if you guys saw that it's really great there's a few um applications where they showed some issues and how they solved them with correlating traces with metrics, and that was great but like is there anywhere that we can go to get more of that like um who's using this, I guess is my question and why did we as as a group like land on open telemetry?

B

Is it because that's what's there in the community right now? That's what has the momentum like or was there like? A more informed decision about open, telemetry tracing is a really good fit for kubernetes. These are the questions that are spinning through my mind. That's.

A

A good question for david, I think a little bit, um but I would also add to that, like there were at least two competing standards. Previously there was open census and something else and those got.

B

Emerged into open, telemetry.

A

Or open tracing right, so uh I think that in part it's just this is the thing that the community's kind of centralized on and backed, and that is why also open telemetry is a cncf project, so in general kubernetes, it's like why prometheus as well. Well, it's a cncf project and in general, kubernetes has sort of focused on integrating with cncf supported technologies, so yeah.

B

All right that that's that's, that's what I know too, um and it makes tons of sense to me, um but so I'm I'm also a member of this other group called operate first and there's a telemetry working group there and it's a collaboration with some research people from like tufts and harvard and bu, and then it's red hat's, a a supporter of it too. um But it's really supposed to be out in the open. It's just new and it hasn't been yet.

B

But the researcher was asking me yesterday um about this question like why are we tracing asynchronous or synchronous transactions? And you know um what what is this going to be used for and and all of these questions that I couldn't answer, so I want to to kind of just get a discussion going here, um maybe in another meeting, but something to think about, and then with that with that group operate first, I have the opportunity to stand up a long-running kubernetes cluster, with tracing enabled the way that I have it here.

B

This is just a single node cube admin cluster, but is there anyone that would like to help me by like helping me set it up or getting some varied workflows running on it like the research people, they really want data and they want to evaluate like how we're using tracing and then get back to us and be like. Oh you could these people have like phds and tracing specifically, you know like.

B

Oh, you could use it this way, and this would make a lot more sense or if you, you know, if you configure your sampler this way, you can get all this more information. So that's something that I I want to set up and I'm wondering if that sounds interesting to anyone or helpful or yeah.

B

So those those are the two topics that I wanted to bring up. At the same time today,.

C

So so personally, I think tracing is pretty broadly uh useful, mostly because kubernetes is, uh you know pretty distributed, um and basically uh the cardinality of events is just too high to be digging around in logs for different.

B

C

Yeah and you just need um connective tissue, sometimes in order to debug a problem um and.

B

C

Yeah yeah and yeah and basically tracing gives you that uh and metrics only give you metrics per component. um So uh that aspect is really only solved by by distributed tracing.

B

And and don't you think that the real benefit of tracing is going to be like, like that kubecon talk was talking about just correlating, you know looking at looking at hints in the metrics and then being like. Oh that's where I need to look, and these are the traces and then and then the the logs are stamped with the trace, and so you know correlating those things together, and that is what is possible with open telemetry, that's not possible. You know if you use different tools for different things, so yeah.

C

That's like stage two yeah yeah.

A

Exactly can you can you tell us a little bit more about the trace that we're looking at right now, because yeah.

B

To me, this doesn't.

A

B

A

We're using all.

B

Right so yeah, so in my in this cluster I have it's just a straight up, nothing running except for a test deployment and it's a it's a mutating admission web hook. um I I can just throw the links in the docs, so you can they're very reproducible.

B

um So I figured this oh and I can let me go back as you can see like the api server. I can scale it too to create some activity that I I set this up a couple of hours ago. Yeah. Let me go here and scale it, but um so yeah. My idea was that would create a lot of activity with um lcd and api server right.

B

I figured with a mutating emission, my box, so that's why I chose, but that's the only thing that's running here. It's just.

B

A

Yeah, when I think of like cubelet tracing there are like specific.

C

A

Loops in cubelet that I would care to find information about, and nothing that I could see was there like. I'm not sure what this container status is. I'm not sure what the entry point of code is.

B

Yeah, let me see what I.

A

Have with people.

B

That's so I have.

A

B

So I because I started with cryo I tried to so here's the cubelet and here here are the operations um with keyboard and I just wanted to see cubelet with cryo. So I have the grpc calls like with the the runtime service like what is status, for example, um sam pod sandbox status. Let me oh no.

A

There was one that was just called status well,.

B

A

B

A

Yeah, so that's the first one that we looked at and that one is like quite fast from what I could see. Yeah.

B

A

With some like slower tail ones, there was one that was just called status. I don't know if that was like full pod status.

B

A

It was um under cubelet, you had just scrolled past it under keep it. Oh okay, all right. Let's see it's like closer to the bottom. Oh okay status, yep, yep yep got it.

A

So these are like basically sub millisecond spans. I'm not necessarily sure how I see.

B

What you're saying those are going to.

A

Be because, like okay code is running fast,.

D

So if I can chime in and talk a little bit, uh one of the things that I think would greatly improve, this in terms of usability is connecting these grpc calls as part of a higher level flow right. So um when I previously worked on something similar to this, I guess a few years ago now, one of the things that was useful to me was to put a parent span at the entry point of some something the cubit is doing so. Okay, like one thing the cubelet might do, is to start a pod right.

B

D

To update something right like there's some entry point somewhere where it's doing.

D

Maybe five or six or seven grpc calls to the container runtime as part of that, and so okay.

B

D

You generally want to do is start a span at the beginning of that longer running operation to get a single parent context. That's then used for all the child operations of that and that way.

A

D

Like the idea with tracing is, I can have a general problem that I know about like. Oh, my, my pod is taking a long time to start and then I can see the sequence of events right. Oh, it went and called create pod sandbox, and then it went and called this other thing pull image and that took 20 minutes and like that way you can. Then you can also.

B

D

Back a particular grpc call.

B

D

A particular pod startup and that's where.

B

So you'd have the grpc calls like um together in one sp in one span, yeah.

A

I mean what we're getting here, probably isn't that much more interesting than the grpc calls themselves from like the cryo side, like I don't think we're getting.

B

A

More information right now by like instrumenting from the cubelet side. I strongly agree with david that, like what we're missing is the context like how do these things all piece together, because right now, we can't see like what does the entire life cycle of a pod? Look like, uh like you know, to for creating a pod, or this kind of thing like it requires more intimate knowledge in terms of like how does the cubelet schedule pods.

B

A

Think, right now, uh like the the information that we are getting here, is not there's not much benefit. Above and beyond um the the cryo traced stuff like we can see.

A

Okay, this thing is taking a long time, but like not like this thing is taking a long time in the context of this, or in this case most of these are taking a very short time, uh and so that doesn't like, if there's a lag, for example like say, there's like a three second delay between x and y yeah, that's probably more what we care about than the fact that this grpc call was very fast.

B

Yeah and I would love to um have a an application that I could tweak to make like to break something or to make latency like add it um add like a timeout or a lock or something so I just haven't gotten there yet and then that's really good information, because now I'm going to go back through the code and look for those spots.

B

The thing is, you guys know the code so much more than me. So if there is something that comes to mind where you'd be like that, would you know if you could just reach out to me and or in the pr you know nicely.

A

Node owns this code, so people on this call don't necessarily but.

B

A

Us that intersect with sig node, we know this code. uh That's probably me and david.

D

All right, some of the more interesting things when I did this a few years ago, were image. Pulling, can be really long.

A

And it's really easy.

D

To make that be really long by having finding a giant like many gigabytes container image, if you want to.

B

Show how someone.

D

Right, if you want to show how someone would go from hey my pod's, taking a long time to start to oh image, pulling is taking forever. It's really easy to see. If you get that kind of trace. The other thing that can be a fun problem to debug is the cubelet needs to report status for pods, and that can sometimes be the long tail of seeing a pod become ready, and if you are spamming, the api server with other junk or if you rate limit the cubelet.

D

I think it has like a certain number of updates. It's allowed to do per second, I want to say elena might know better, but um if you lower that to something really unreasonable, you can see a big gap in there.

D

As elena was saying. Sometimes the gaps tell you just as much as the spans themselves.

A

Yeah, the other thing I would add, is if you're looking to come to speed for to get a little bit more context on this. We like in the last release, totally refactored or maybe not totally, but we did a very large refactor in terms of how the pod life cycle currently works. We did a little bit of unification uh and I spoke about that at kubecon. So if you go.

B

A

Watch the sig node recording as.

B

Opposed to the instrumentation.

A

Recording uh there's like a short section on that uh that talks about the.

B

A

And where, in which files in the code, you should look.

B

Cool all right that gives me a lot to work on and thank you thank you guys for that awesome.

A

Thanks so much for coming and demoing and working on this, this is very exciting and it's.

B

A

Especially exciting that this is coming to us like so early, so we can be prepared for the next uh cool.

B

Cool development cycle yeah and quickly, one thing that david suggested was um rather than now that we have two components is to move the tracing configuration to the component base repo, so that, then you know if the scheduler wants to add tracing- or you know the controller manager, then that will be. It will be a unified tracing configuration so that in the in the in the pr that we're doing that too.

A

B

We may want to.

A

Do that as a separate refactor, uh but I think everybody from the api review group was happy about that so david. I want to take a look at that.

D

Yeah, we can definitely. We should at least add it to component base with your change, and then we can change the uh that's. What I.

B

Was thinking yep, okay, great.

A

Thanks again, awesome thanks so much sally uh han you wanna, take it away with a bunch of metrics getting promoted to stable.

C

uh Yeah, I just wanted to throw it on the radar. uh Basically, we have like three different components. I think that are getting proposed uh scheduler, I don't know of controller manager, api server and there's three pr. There are three issues I think hold on. Let me.

A

Do you want to show your screen.

C

uh Hold on, let me isolate my windows.

C

This is weird because I think this is actually.

C

Scheduler too, I don't know why this one is separate. um Okay,.

C

A

C

Yeah, sorry, I lost the window for a second. Let me share.

C

It's this one, can you guys wonder oops.

A

I could see it, but then you got rid of it.

C

There we go yes.

A

C

Yeah, so we have three cls, uh scheduler metrics, there's a few that they want to promote. I don't know I I think I had an objection to one of them, but.

C

um Yeah, I just wanted to throw it on the radar that we're we're getting a bunch of um requests to promote disabled now.

A

C

Yeah yeah, so so, uh if you guys want to take a look at these, in you know, offline, uh probably a bunch of eyes before we commit a bunch of metrics to stable, would be a good thing.

E

C

Remember correctly,.

E

um There was like one condition for the migration from alpha to stable at first. That was supposed to be different. Well, like a metric needed to be used somewhere, I guess to be used widely before being promoted to stable. um Do you know how we can measure that.

C

uh I think we measure that by big time.

E

Okay yeah, but what? If what? If the matrix is not used anywhere, then we have.

A

It's been there for two years.

E

C

I I know that internally at google, we use these metrics.

A

So yeah, I think at this point it's mostly just going to be based on people self-reporting and saying yes, we would like these metrics to be stable, like kubernetes, can't call back to people's clusters and ask hey: are you using this metric in a meaningful way, and I mean even like that would be useless in this case, because of course the metric exists? If it's there, it's going to get scraped, but that doesn't necessarily mean somebody's querying against it. So.

C

And these api server metrics, I am pretty sure that we use all these and possibly alert on these two.

E

uh The schedule.

C

One just asking.

E

For, like the general purpose like what, if someone was just asking for a metric- and we are not sure that it is used widely or not,.

C

We we should definitely ask people from the components I mean like so my comment here was basically this um saying. I don't know if you guys can see this, but the component owners own them. They commit to to the stability requirements to adhering to the stability requirements. So I mean if they believe that the metrics should be stable.

C

I mean they presumably have a reason for this. We only own the infrastructure, so for us, the things that we should be doing is making sure they had proper bake time that they follow metric conventions.

C

um You know uh that sort of stuff, but in terms of like actual usage, uh I don't think we're, as I said, we're in a position to to know.

A

Or as a project, I think uh yeah, I would say you know, ask around at your uh place of work. uh If I, I don't think we have any objections to making these stable, I mean it's it's on the component owners to either object or uh accept, uh since they will be involved in the approval uh it looks like we are at time.

A

I don't want to keep people late, um but thanks john for bringing this to the group- and uh I guess for the newcomers uh you can find us between our meetings on the pound, sig instrumentation channel on the kubernetes slack, uh if you're not yet signed up. I believe it is slack.katesk8s.io.

A

And we also have a triage meeting at the same time slot every other week, so we may see you at triage next week thanks everyone for joining cool, bye.

C

A

C