Kubernetes SIG Instrumentation, 3 Feb 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG Instrumentation 20220203

Description

SIG Instrumentation Bi-Weekly Meeting Feb 3rd 2022

A

Hello, everybody and welcome to today's edition of sig instrumentation. It is february 3rd 2022., david ashbold kicks off.

B

Sure um so for announcements today is the final day for cap review, get your caps merged. I just lgtm the cubelet tracing one, but um maybe at the end, if we have time we'll review additional caps.

A

Yay, okay I'll check the spreadsheet and have that ready.

B

Okay, uh damian did you want to talk about.

C

Yeah I wanted to bring a topic to this meeting because, like recently, I've been seeing more and more than like pr that get merged without their approval, even though, like the archangel metrics and whatsoever and like whenever. Basically I'm finding like some weird metrics, that I've been modified recently. Every time when I go to the pr it's been approved by and like someone from the sikh team from this particular component and kind of bothered me that we could have uh like give some insight to them like on how to improve that properly.

C

And I would have.

D

An idea for this, if we could.

C

Yeah, if we could bring some tooling or anything really to uh make sure that at least someone from c instrumentation either say it looks good to me or just to prove the pr yeah.

D

I I get what you're saying so. The problem there's a couple problems, one the way that the labels work relies off of regular expression off of file names that we can't really change. What we can change is, we can add a static analysis tool and a hack verify script that ensures if a component-based metric is defined somewhere, that that file name must be suffixed with underscoremetrics.com.

D

Yeah, that is probably the easiest way to ensure the thing that you want.

C

uh Yeah, just one comment about that and what I've seen also so far is that um the file were named properly, so the label uh sig instrumentation was present, uh but I guess it might have slipped like in between um our uh triage sessions. So we kind of missed it and to have a proof before like we had a chance to really review it. um So it was more.

D

C

Really make sure that we are either reviewers or approve those so in this particular file up here.

D

D

To become approvers is a big.

C

D

C

Yeah the the idea that I had was to basically ensure that all the metrics that go file are in their own directory, where we have an owner file that is owned by instrumentation, um so that I guess well.

A

I don't think that that's open.

C

Will be the first assignees.

A

I don't think we should do that. I will tell you why that's.

D

Outside the scope of our thing, it's outside.

A

The scope of our sig, our job, is not to maintain the metrics for the components. That's the responsibility of each component owner our job is to set standards. So, like the reason that we have, I think han wrote a special thing such that for, like any metrics.go file, it gets a sig instrumentation label and then we can review it during our our reviews.

A

Like we get because the label gets on there, then we can get pinged as reviewers for those files, but we're not in the owner's files for those things, because we don't want to be uh and so like. If those component owners aren't like requesting review and they're merging, like metrics with issues, then that sounds to me like more of a social thing, that we need to like talk to other groups and tell them to make sure they're pinging us uh when they want metrics, reviewed or they're making metrics changes.

D

Well, alternatively, if you like, depending on how much this is bothering you david uh metrics, adding beta stability class to metric stability framework, will solve a huge percentage of these problems, because there will be many metrics in beta and those metrics in beta will have to generate the metric schema file in our directory, which we have to approve.

C

Yeah yeah so yeah that that would fix the issue um so basically like. When the metric goes from alpha to beta, then we would have always to review it and approve it right.

D

C

Yeah that would kind of solve the issue like the only thing is that that is kind of still bothering me is that anyone can really like introduce their own metric with their like with issues. uh I don't know like a label that might cause uh cardinality explosion, and in that case it becomes our problem and not theirs.

A

Oh no, no, it is their problem like if, uh for example, there was like a cve or something like the owners of that component would be the people on the hook to fix it, um but like for that reason, it's why we have things like the metrics cardinality proposal in order to like clamp these kubernetes wide. So we don't have to worry about somebody doing something like that. The problem is, we just haven't made much progress on it. So no.

D

No that's implemented.

A

It's implemented as an alpha. It's.

C

Still not graduated yeah yeah yeah. You have to do this for this.

A

So uh yeah there's.

C

Stuff that we can do but like.

A

In theory, someone should be, I mean the ideal in my mind, would be someone should be able to add a metric, even a bad metric to kubernetes safely and like we have provided sufficient tooling such that, like we will catch them from shooting themselves to the foot too badly.

C

Yeah definitely, and that that's why recently like I've been working uh with the people from client golden to introduce like a new mechanism that would help prevent cardinality explosion, but from the client side. That would be like kind of an opt-in flag to add to the registry, so that, like any metric that has more than I don't know like three like 5000 value for a particular dimension, we would just remove the dimension. So I'm working around that but yeah.

C

I guess like the thing that han said about reviewing all the metric that goes from alpha to beta um it's kind of good to me, I guess would be like a great improvement, at least.

D

If you want to take up the beta stability work, I have this work in progress. Draft.

C

D

As a short term,.

B

Fixed should we just rename files with metrics in them, so that.

D

B

D

B

D

B

Right so that we get notified for more of the changes.

D

That that doesn't solve the problem. There's a couple problems. One is they're, not always in the metrics file. The second problem is that, like the triage accepted is once so, multiple sigs getting tagged, which is always the case with civ instrumentation and if they have a triage they're going to remove it from our queue.

D

A

Well, no, but we look at we look at things without assignees too right, I mean the issue. I guess.

B

Is that they may also have someone who's, not us.

B

This is what the like all instrumentation prs is supposed to catch, but we.

D

Don't always spend a ton of time there? Maybe we should. We should have a list of assignees amend that query instead of triage accepted just do where assignees is not in this group of instrumentation reviewers.

B

That could work yeah.

C

Yeah yeah definitely.

B

Okay, is that all for that topic.

E

B

Why don't we move on to whoops um jin haas? Are you.

E

Here, uh yeah uh mg house: this is uh my topic.

B

Would you like to present or yeah.

E

Yeah, I would like to presentation or email some documents, so I would like to share my screen and explain it.

A

Give me one second to give you the ability to share.

E

Go ahead: okay,.

E

Change your screen, uh can you see my screen.

D

E

A

Yes, it does say that this is confidential, so if there's anything we shouldn't be seeing, I just wanted to mention that, since this is a recorded meeting.

E

uh Sorry uh this this is, uh I can publish that uh this document, so please don't mind. uh It's just just being extra safe yeah, just.

D

Don't want to get in trouble.

E

Please don't mind this design. Okay, I will start uh I've shared ideas, I'm currently thinking about that kp. So I would like to hear feedback on the concept of my idea.

E

At first, I explained objective of this therapy distribute proposes to propagate trace, contrast across components and across a series of related kubernetes objects originating from a user request, and what I want to do this to do is track rows, so this kp raise the foundation for enhancing relevant but strategy with sid as a common identifier,.

E

Next, I explain connect our rough situation. Currently, we can trace in only qbap server and x3 by using a psm training feature which is developed by david, and we can use trace id for road tracking. By using this feature, specifically, we can acquire dress id from trace, compilers and adding trace id into each logo in kb server, and we can easily extract the api server log, which is called superior request by filtering the api sub log by trace id.

E

However, we can propagate x-rays countries across each component's component, mainly control site, so I would like to solve this problem, but currently we can't complete design to solve it. Yet uh the figure on the lower left is the range that can be traced. Now it may survive. The city and cube rate will be covered in the future.

E

E

The video of the lower uh right, it's a range of the rock tracking that I want to achieve to be processed. It means range we want to trace the rock, so not the range we want to propagate in that respond list.

E

This is initially our idea. I and david thought that we stress context to annotation in an object so that we can propagate context to each controller.

E

Specifically, a sensory consult function in controller has a key of the object to be reconciled as the argument, so it is possible to acquire the trace context of the annotation, with object from the key and propagate it in reversal. Function.

E

To explain in the video below vista stress context in annotation, and then this can be differenced by using key and most reconcile functions have a key as an argument, so we can use that key to get respondents.

E

However, I noticed that this method has a program. In certain cases, we can't provide correct address quantities.

D

Multiple objects can can actually uh actuate against the same object, so you might lose the trace context and get a different one.

D

Right like there are multiple controllers which act on the same objects.

D

A

Sorry, do you want to give a very boring example of that sure, uh so a very boring example of multiple uh controllers acting on one object, is on the pod right. Like initially you know, uh a client puts a pod into the api server uh and then uh you will have the scheduler uh go and like potentially put a note on there, you might have an ingress controller go and put an ip on there.

A

You might have uh you know uh any other number of controllers that might be touching, that pod object and then eventually like uh at the node being. Maybe the most obvious thing will act on that pod object and actually go and do things with workload. So.

D

So so, I'm not sure exactly like what this trace context will get you, since you don't know which trace context. It is you'll get like some random controller, which last acted on it, which may not give you the story that you are looking for right, which is like.

E

Yes, I think so so.

B

Yes, this is the initial thing that we had in the api server tracing cap as well. So this was this is the initial idea. I think they're going to describe.

D

Oh there's a different idea.

B

There's a different idea: right: okay, there's going to be another one.

D

D

E

Go ahead, yeah.

D

E

uh This this is problem or initial idea. We can propagate with this context. If suffering object types are different. uh Here I use two types: a and b, so, regarding type a I I define object which triggers contraction as a and then I define object, which is reconciled by controller as b at this time. For example, user updates, the dataset object in this case type a is replica set after this use option. Deployment control does reconstruct action.

E

In this case, type b is deployment. This situation means the deployment contract tries to reverse the changes in the replica set. In this case, deployment control gets deployment key in argument of reversal function and then propagates press context. However, this reconcile was truly not by the biggest exchanging, so we should propagate race contrast which contained in the precacet, but we.

E

Can't this is explained in the video below uh when the user updates, the replica set uh actress context is saved inside the duplicate object and which goes into direct 54 inside of the informer, and then the preset object is passed to each handler.

E

At this time, the replica set control has not only the requested hundreds, but also the deployment handler and the deployment handler receives the replicas object and then deployment handlers, such as for the deployment object that becomes the owner of the replica set, object and end user owner's deployment, object key to the deployment controller's rock2 and when starting the reconcile function of the deployment controller.

E

What is obtained from the argument? Key of the function is the deployment object, so only trace context of the deployment can be obtained with the reconcile function. This is what I consider to be a problem.

E

Next, I will explain the concept of the solution that I'm currently thinking about, and please note that this is different from the idea. I recently told david. It ought to require information to the log files in the same way, but uses different information.

E

Main concept of the solution is using commands, object, uid and its version in lcd. Current components does not use objective version, but it is stored in the cd.

E

If we look directly inside lcd, we will see that each object has a version and badge means object version and the objective version is incremented by one, but the object changes such as populating specs or status updates, for example, if the version of the deployment object is two and the contents of the objective united states such as spec and status, are updated, the version will be incremented to three.

E

This solution provides method that drag operations, microscopic components by tracking changes in the components objective browser, so it doesn't use tourist context and doesn't propagate it. So it's a completely different approach to the original idea.

D

Do you know what the difference is between the version and the generation.

D

So it's slightly just speaking, uh do you know what the difference is between the version and the generation.

E

uh Is different from generation.

D

Yeah yeah, but do you know what the difference is.

A

uh I don't know what the difference is han so could you explain it to me.

D

No, I I can't recall the difference, but I'm just wondering if one of them is more appropriate for this than the other, because I I can't remember, I.

A

D

uh The okay, so the generation is, is okay from the docs. It says a sequence number representing the specific generation of the desired state.

B

Would it just exclude status updates.

D

B

That would be a nice feature for this.

D

Yeah, isn't that possibly better.

B

But I think we can maybe explore the general idea of using a uh version number.

D

B

I'm just saying yeah.

D

D

E

D

Yeah, I'm sorry to interrupt. I was just.

E

E

And uh we can get usually objective version. He addressed a at the following timing.

E

First, when abs server receives the request, this means that we can get the version of the object that the request wants to change and second, her red handle, gets object and includes object, t into rock2.

E

This means we can get the version's object, which is passed to the handler, and we can also get budget of the object to put ins workly from the handler and third, when starting a console. This means we can get the version of the object from the key.

D

Oh so you're talking about rv here, you're talking about resource version.

E

uh I I I think, uh resolution.

D

You are talking about resource version here.

E

Yeah: okay, okay, cool.

E

E

For example, or by outputting following information to logo files, I think we can track the processing flow, but this is still under consideration, but I'm considering without information, which should not request regarding a quest api server, for example in case of user output. Sorry.

E

For example, in case of user request, in this case, uh trigger is crib. Control uh in case of request from controller uh trigger is uuid and object version which is acquired from the controller, and uh I also want to output tonight information uh turn. It means that the object, which is changed by the request so, for example, uh in case of user updates, typical set uh in this case uh trigger, is created, control and the turnitin, because it's usually id and object browser, and then I want to output.

E

Information which is acquired in android site uh in case of hundred uh trigger, is usually id and objective browser which is passed into this handler and turn it its usual id and object version which is entered into rock2.

E

So in this case, if user changes replica set and then deployment controller starts reconciling in this case a trigger is replica, set user id and object to browser, and, let's turn it is deployment uid and object. Browser.

D

Interesting, so you can make you can make like a linked list between uh the object and the rv yeah.

E

Interesting, yes, and.

E

This uh this allows us to track the processing. That was a problem with uh page four. uh I see I think uh this is program, uh so I want to solve this problem and connect my idea. uh We are good.

D

We're running out of time, but can I be frank with you like? I don't mean to like this- is going to involve many cigs you're going to have to get buy-in from, like basically every sig and right, because.

A

Yeah this touches api machinery. It touches apps and.

D

A

uh Maybe touches node. uh Definitely you know it's under the scope of sig instrumentation uh there'll be a lot of people that we would have to convince is a good idea, because we we can't just go and do this on our own. We need all those people to agree.

D

This like, if there's like a more incremental way to to implement this, I think that would be your best bet like if you could target like one component, where you could just do this and convince that sig, hey. Let's add this thing for debugability.

D

I feel like that, has a higher probability of success than like presenting this entire solution to every single sig like there's going to be some amount of information overload, um it's interesting to me. I'm not sure I'd have to think about this more because it's like this is a lot right. You guys have obviously put a lot of thought into this.

D

But from a practical side, like you're gonna run into some resistance from just the fact that there are like five sigs that you're gonna have to talk to, including probably cigar conduction, who you would have to convince. If you wanted to do this because yeah this is like a cross. Sig thing.

A

So what do next steps look like here.

A

Because the thing is that this is we're suggesting that we want to do this via log changes right.

A

We currently have a working group, that's working on structured logging, migration and, like that, that effort alone, like just transitioning, to structured logs, that's taking an entire working group. um We had uh another uh a proposal that came in this release for adding some contextual logging to some components where we could propagate like sort of automatically more key value pairs in the structured logs. uh But it's not touching any of the the api machinery or adding any excuse me annotations to any of the uh objects that are being acted upon.

A

uh It's just using information, that's already available in pre-existing contexts uh rather than the the global logger. So it's a bit different uh than what's happening here. uh We sort of gave them. I was the approver on that because it seemed limited scope and because there were like on off switches, um we did give them the sort of tentative go-ahead like cautious go ahead. It's approved, see, see how it works for something like this, I would say we probably need a similar plan.

A

You'd need um a sig that you have buy-in from uh that will will approve changes uh to any logging code that you've made um and that will support your sort of poc.

D

Yeah, so you would have I I would say: you'd have to go to api machinery and get them to sign off on this, and it should be a much slimmer version than this, because if you present these slides, I I I know the api yeah it just. It would have to be a much simpler proposal uh for api machinery.

D

Specifically, that's how I would that's how I would go for it and if you can get them to buy off on it, then, like your chance of doing all of this increases tenfold, I have to go because I have another meeting. Yeah.

A

I think we all do uh we really want to see.

D

A

This is a very.

D

A

Scope proposal so we're just trying to help you scope it down so that we have a hope of getting it landed uh because in its current state it will be very difficult to get all the necessary approvals.

B

um How about people if they have any comments on the proposal itself um and if you can link it, can we do that in the agenda doc, so please pass a link to or somehow so that people can view it and then, if people have technical feedback on it, please leave that in the agenda doc as well feel.

D

Free to come to the next triage meeting, we can discuss it there if we have time more yeah, okay,.

E

I will share this comment.

A

Awesome yeah yeah plea, please, if you can share a public copy of this presentation as well. That would be very helpful. Okay cheers everybody bye-bye.

E

Thank you maybe.