Kubernetes SIG Instrumentation, 14 Apr 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG Instrumentation 20220414

Description

SIG Instrumentation Bi-Weekly Meeting April 14th 2022

A

B

This meeting is.

A

Being recorded and welcome to today's edition of Sig instrumentation, it is Thursday April, 14th 2022.. We have uh a pretty sparse agenda for today, uh but I wanted to pull in at least one announcement and possibly raise some sort of longer term discussions bubble, those down from sort of kubernetes wide uh chairs and leads meetings.

A

So first I know on the agenda announcement. The release has been delayed. uh It was supposed to be happening next, Tuesday, the 19th. uh It's been pushed two weeks primarily due to a golang 118 bug uh that still hasn't been fixed, I think the release Branch still hasn't been cut, so uh yeah we're definitely not releasing next week uh and I linked the pr in the Sig release repo that updates the dates.

A

Any questions about that.

A

I, don't actually think that this affects us I, don't think that Stig instrumentation has anything left in the Milestone, uh but if I'm wrong feel free to correct me um and then I wanted to sort of bubble down some discussions from the chairs and leads meetings.

A

So uh this month we've been talking about three things. uh The first thing, I think is maybe a little less applicable to us, because I don't think that we have. We don't own a lot of tests, uh so our tests don't tend to like flake in a release blocking way, but uh the I think uh like Sig release uh and the CI signal team have been having some difficulty uh with ensuring that chairs and leads and, like other responsible folks, are being responsive to release blocking flakes.

A

uh So you know we have our triage meeting I. Don't think that we are in any like bad shape with that one I'm, pretty sure that we and when we get flakes we deal with them in a timely fashion. As we do any other things. We don't get a lot of volume there, uh but I wanted to raise that to folks attention, because apparently uh this is a problem for some steaks, so I'm curious. If we even have any flakes under instrumentation right now,.

B

We had the like metrics gatherer ones, for a while, but I think.

A

Yeah uh yeah, like good news. uh We we currently have zero open flakes. So uh that's that's not a problem for us, but uh you know. If this happens, we should be responsive. uh Han and I know this uh as chairs. It's our responsibility uh to make sure that those things are getting handled so um I don't know if there are any anything to add on that one, but wanted to let everyone know that that's a thing that the release team is having difficulty with. So please help them out. If you see that happening.

A

uh The next topic uh is about improving reliability. I think there's like a giant cap which includes like a power to block the release um or other, like quite uh I, mean I guess compared to like kubernetes developments uh previously like somewhat drastic measures um and there's a lot of like means in which we were talking about like how reliability and kubernetes is a problem and how this is like slowing down development, because a lot of parts of the code don't have a lot of good tests.

A

And then you change something you're like well Ci's going to catch that right. But then it turns out. There was no coverage for it. uh So something ends up being broken and either it doesn't get caught in tests at all, and we end up getting a user bug which is like probably the worst case scenario or uh still a bad case scenario. Is it like causes a race condition or something like that?

A

And it starts flaking on a test, but then go back to item one the flake doesn't get addressed and people don't realize that there was a regression.

A

um So there's some proposals being discussed about how we might better uh handle this sort of thing. I just wanted to make sure that folks were aware that that was like kind of floating around, in particular like the idea that like if we are having serious reliability issues, uh some people are talking about having the power to block a release uh or to block features for inclusion in a release uh and to give everybody a chance to like weigh in on that or possibly add to the discussion there. If you're not aware that that's happening.

C

No I mean I think stricter test makes sense. We don't want to be emerging in bad code, I think.

A

Yeah I mean the problem is that we don't have the tests uh and I'm definitely curious, like what the incentives are here. um uh I suppose that the the chairs and leads meeting sort of like going to the Chatham House Rules, so like I'm, not going to point any specific things that, like anybody raised in that meeting. uh But if you look at some of the public comments uh there are concerns of like you know like chairs, and leads sort of have limited powers like they can't make.

A

Anyone do anything, um and so like is this? Actually the situation better, um which I think is a very fair point. um So I don't know if other folks have. It's certainly like the idea of like being able to block a release.

A

I just want folks to know that, like that's only being proposed, that's uh not necessarily accepted, um but if you haven't looked at it and you want to weigh in uh you should probably do that.

B

How's, this different from just regular release blocking tests.

A

That's a good question: uh I think the idea it's been a while since I've read over the cap uh but like I, think the idea would be they can say. We just won't accept your features um like in addition to like they're, going to block the release to make sure the thing get sticks, but they can also say we're just not going to allow you to merge any of this stuff um which is like I, don't even know what the enforcement mechanism would be.

B

Yes, previously.

A

That's been like Milestone maintainers kind of thing, so maybe the idea is they just don't take Milestone protection off uh and, like somebody has to act, every change going into kubernetes kubernetes like that's. That's me, spitball I, don't know if that's exactly how they would do it, but that is a possible enforcement mechanism.

A

I think that that would not be great um but like yeah I mean, if that's what people think needs to be done in order to uh prevent, like so many people from kind of doing heroism, to chase down regressions, then maybe that's what we do. I don't know.

C

I think it's fair to dump it on us uh generally I mean people have to propose caps in order to get information code. In so I mean we have to safeguard the stuff. At some point it does fall on us yeah.

A

Yes and no for new features, I think like we have a pretty sustainable workload generally, uh it's it's not like. You know uh like to give an example: Sig node will often get like 20 to 25 people uh like trying to propose graduating enhancements in a given release, which is a lot um whereas like we might have two or three in a cabin release.

A

So it's easier to make sure that, like everything, is getting the attention and like reviewer time that it needs, but there are certain things that, like wouldn't fall under, like you know, sort of the enhancements review um so, for example, bug fixes like what happens if the bug fix makes something worse. That could potentially happen. uh Another possibility is like what, if someone does a refactor right, they refactor something. They think it's a no-op. uh You know no test changes are required right, but then it turns out.

A

There wasn't a particular case like providing some coverage and they actually broke that case and we find out about it later, but it was refactor so like there was no release. No, like everybody thought this was safe and then it turns out it wasn't. um That's. That's the kind of things that they've been discussing.

A

Tool uh and then I guess the last topic uh to Bubble Up, uh which Han you sort of segued into nicely is uh like enhancements process and I know.

A

We've talked about this a little bit as a Sig before um a thing that I really pushed um uh Stephen uh was presenting this and I think that I really pushed him on is like ensuring that, like we know what problem we're trying to solve and that it solves that problem for everyone all of the stakeholders involved like Sig chairs and leads uh like authors of enhancements, the release team, like it shouldn't, just be targeted at one particular group.

A

um I, don't know if anybody has read through uh the receipts kept in its uh existing state, which I think is one of the things that people are being asked to comment on. I personally, think that it adds a lot of paperwork burden and I don't understand what the benefit is or what the problem it's trying to solve is so.

B

I'll give a brief summary of the cup.

A

The idea, I think, would be that each cap would have like you know how there's like a kept.yamble file right now with a bunch of metadata um I believe that it would be like kind of an expansion of that, uh and that would be like more like the data in there would be required to be more accurate and uh it's I haven't looked at it in a while again, because I feel like it was a year ago.

A

The last I looked at this, but uh it was quite uh quite a change, but ultimately, like the goal. I think was just to like use that to auto-generate a spreadsheet or something like that uh and my my hot take from that was like well honestly. I feel like this is asking everybody to do like a ton, more work and ultimately we're still going to have a spreadsheet and uh I.

A

Don't know that this will actually improve anything because now, instead of like the burden of you know, just like okay I can update a cell in a spreadsheet. Now the burden is like I have to submit APR uh and get it emerged uh and I'm worried that the turnaround for that will be very slow and I know at least right now. In my experience as especially like a prr approver people have a really difficult time, filling out the kept.yaml metadata correctly to start with and there's not very much in there that's required.

A

uh So if anything like I would Advocate like getting rid of the kept.yaml metadata or make it smaller.

C

You can you can use cup cuddle to generate the so.

A

As somebody on the prr team, I have uh been a user of cap cuddle and it frequently breaks like yeah I.

C

Know I I just fixed it I, literally just I, just fixed it I.

B

Didn't even know it existed.

A

uh Cap cuddle, yeah I've had to like do a bunch of things to to fix it because it wouldn't even work uh so I. Don't know that like kept, cuddle is in sort of like a ga state where, like.

C

It's not, but it works now. I I literally just fixed. It.

A

C

A

To you for doing this, but given every single past experience I've had with it and given how much limited time I have arounds their release deadline, the idea of every single time I need to use cap could all have to go in and try to like fix it, uh because that has been my experience. The last three times I've tried to use.

A

It is not an acceptable user experience for something that we are going to make everyone in kubernetes use, particularly also including people who aren't necessarily like writing code and are trying to help out with the release like these are people who are not necessarily like even pushing stuff to GitHub. So.

A

Like Han tell me more um because you're saying that oh well, there's just kept cuddle. uh How does cap cuddle make the enhancements process better or less work for you, because I know that you have been outspoken in the past in terms of how.

C

Because, because exactly what you're talking about um there's like a zillion fields that you need to fill out in order for it to be valid for it to pass the hack validations, this is this is the the fix. um It was weird because they were just not mostly Plumbing stuff. All the way through um you need like approvers, you need prr approver, you need an author um yeah, it was. It was mostly just Plumbing.

A

Some stuff yeah, so one of my concerns is that, like I personally, think that it is redundant and toil and a bad use of people's time to ask them to fill that stuff out in metadata, because that can automatically be generated from a git history.

B

C

A

Think that's part of why we're having a lot of difficulty getting folks to actually like fill this out accurately uh like it is a struggle for me to get people just to set the current Milestone correctly, set the historical Milestones correctly and like set the phase of the enhancement correctly.

A

That's just three Fields uh there's, like probably at least a dozen that are currently supported, and in my experience like they are almost never accurate uh or like they're, just not filled out they're just ignored in part, because you can get this information from other sources and fundamentally like the other sources, are the canonical truth like I could put an updated date in that field, but it's not necessarily going to be true as soon as I update the cap. Again, that's going to change so.

A

If I forget to update it, then that's a problem if I need to like update it every single time that I make a change. That's a terrible user experience I think uh because it's an additional thing for me to do for benefit that I, don't really see. uh I think maybe a way that this could work is, if, like we had some tool that was kind of like make update that just filled that all out automatically. So nobody has to do any work.

A

uh You know like they just like kind of typed make update in the same way that there's like a hack script to fix the table of contents and it automatically populated all this information for them.

C

Yeah I mean I think that's like. Basically, the next step is um is to have a hack script that invokes kept cuddle um like cap cuddle, fix or something so.

A

That I think that could work for the stuff. That's like automatically generatable but, like some things are not easily. uh So those are things like what the current phase is when it graduated that kind of thing and I think that mixing that in a file that is like half edited by hand, half auto-generated is also not a great user experience.

A

C

I mean we're kind of using GitHub as a database.

C

We should really just create like a cap UI with an actual database, because then you can have like validations and people can like upload stuff. So.

A

All of this is true, but Han. Let me go back to my original question. What is this solving? What is this making better for people who are writing caps, like fundamentally as a reviewer.

C

A

C

Any of this stuff.

A

Like well I love, the process takes less pain process.

C

Less painful for me, because.

A

I, don't know I just made the process better like this is what I'm saying yes.

C

A

C

We're putting a Band-Aid over uh yeah exactly exactly.

A

C

Should be yeah I.

A

Don't like processes can change fundamentally, I want to say, like what problem were you trying to solve if the process sucks? Let's fix the process, let's not write tools to paper over a sucky process.

A

um This is like my concern with this stuff. It's like well we're talking about like adding a bunch of tooling, possibly a bunch of toil. Maybe there are some ways that we can do this, but ultimately we're looking at technical problems to a process or Technical Solutions to a process. Problem and I, don't understand what this is solving other than maybe taking the existing process.

A

That I think is already too heavy weight and I've definitely gotten that feedback from other folks in the Sig and making it more painful, maybe slightly less painful, because we pulled intools, but for the people who don't know the tools, it's going to be more painful for them, particularly if the tools are funky.

C

Yeah I mean look, I just sometimes want to create a cap, and capital is the easiest thing for me to create that cap, because otherwise I have to figure out exactly what all of the required fields are, and at least at least cap cuddle has some validation, but it is built into it. So I can at least fix the plumbing foreign.

B

No um I think it's probably fine um and I would prefer if things were a bit simpler, but I feel like small incremental changes are probably better. So um I probably don't want to expand the set of fields too far, but.

A

Cool sorry, I just wanted to give you a chance to jump in I, wasn't sure.

B

If you had something like.

A

That um so Han I don't know. I've never used cap cuddle to create a cap. The readme says to copy and paste the like template folder, which is what I've always done uh and then just basically like validate it until the validations passed.

C

Yes, that's basically what I hate doing, because the template is basically like broken and then you have to like run against the verify thing and then like yeah, invite.

B

C

B

C

Yeah no cap cuddle is like a thousand times better because it auto generates the right things and then I'll validate.

A

But I do Wonder, like it just seems like so heavy and so much paperwork to do all of these things like if we're gonna reimagine like if we're gonna, make changes that are gonna affect everybody and possibly like in ways that like are gonna, affect us for many upcoming releases. I just want to think a little bigger. Like you know, the the enhancement process in its current state really I think has honestly only existed since, like after 114 and.

C

A few years yeah.

A

Like it's not been like eternally like this in all of kubernetes, it's getting more and more heavyweight and I just want to, like you know like, let's think creatively like do we need this process? Is it serving us? How do we make it better? How do we reduce the amount of work.

C

There's an easy answer to this and the answer is to move our code. Instrumentation code out of Truth.

A

But that doesn't solve the problem for enhancements for everyone. It.

C

Doesn't but I'm saying like what creative ways can we use to make the process easier and I am saying this is one way to make the process easier, which is circumventing it altogether? Is this really.

B

That much of a problem for our Sig I feel like this is a problem for lots of other sakes. But we've got like five.

C

Well, I mean like I, don't really even want to create caps, because it's such a pain in the butt and I would do it if it was not as toilsome I like there's like things, I would like to implement, but basically the process that.

A

Is the problem I think that is the underlying problem? Not tooling not yaml files like I. Don't want to do this because the process sucks. That tells me we need to fix the process and, let's bypass the process by spinning our stuff out of tree I, don't think solves that problem. It.

C

Does it solves it for us.

A

uh It may solve it for us, but we are a horizontal spanning Sig and I'm, not actually convinced it solves it for us, because anytime, we want to do something that will touch other components that are in three or back in the same situation. It's like looking at cubelet, tracing or contextual logging.

B

Everything we do is yeah gonna be like for the.

C

Other I mean there's some stuff that we could just move into uh importable vendored folder.

A

There is some of stuff, but ultimately I, don't think it's the majority. I, don't think it's going to solve the problem for us and I think we should be good stewards. We should be good citizens. You know we should be working with the rest of kubernetes, even if we could solve the problem for ourselves and like everybody else, is stuck in the boat where the process sucks this.

C

Is just not really a hill I I want to die on.

A

I, don't want to die on any Hill I. Don't think anyone should be dying. No.

C

You know it's like look, it's it's a pain. uh People are adding more processed to enhancements, not taking it away and.

A

And so that's why I'm saying like if we don't give them feedback, if we don't give them Alternatives, this is only going to get more painful and I think that we need to be clear and United in saying hey. This is hurting us. This is making it difficult for us to do important things. This is a huge barrier for new contributors, like all these things that matter a lot to the project. If we're not giving that feedback and we're just opting out, it's only going to get worse.

A

And almost like, it's like a hill, we need to die on I, don't think anybody's dying here, but, like I, don't think that it's necessarily as long we need to get into a big fight for or like something that we need to like sort of uh like draw a line in the sand or anything like that. uh I think that we just need to ensure that that feedback's getting bubbled up and that that's being considered uh in these discussions, I.

C

Mean I I. Look if, like.

C

C

I only have so much mental energy to expand where, like I, just don't feel like going in talking to the enhanced people and telling them the process sucks like it. Just you know.

A

But you're willing to do that for me and that's why I'm having this discussion so then me or you know, David or somebody else can go, take it to them and say: hey we're consistently getting this feedback for members of our Sig, especially new contributors, who are encountering a really high barrier to entry when it comes to doing this work like.

C

Yeah they should document cap cuddle, I mean right, like people don't even know it exists. Right like like, like um you know, there's also uh kappify, which turns yamls into caps. I. Think.

B

C

B

With extra fields and stuff, as long as I still run the same cap cuddle command to generate it all. I, just don't want to have to do anything additional, that's manual, otherwise they can add as much metadata as they want if they want.

C

To yeah, if you can run the cap, cuttle commands to generate the thing. It's like, not that bad um yeah, it's pre-cap, cuddle life, sucked yeah, and that's that's why I actually fixed it was because I I wanted to like this is like literally the easiest way to generate a new cap and the template.

C

Copying thing makes no sense like because it gets out of date and it doesn't actually match the parity of the hack scripts anymore and like it's always out of sync, with the hack scripts right like there should be like, and that's why cap cuddle is a little bit better. It.

A

Well, just does cap cuddle end up just copying the template file like.

B

A

Yamble but like the the.

C

It has a yaml file, so it has like a defined structure for it. For the uh expected output, it uses yeah the defined objects.

A

Okay, we are at 10 o'clock my time uh Catherine you were clapping earlier. Did you have anything to add? Are you just stretching no.

B

Because my because I'm in the meeting you so the light turn off so I need to collapse.

B

A

So you're not just really excited about this discussion. That's that's unfortunate.

B

Oh yeah I'm excited about another offers a time for the fight. You might mean.

A

I understand, uh and how are you do you have anything to add currently.

B

No just to have a look at the news.

A

Cool okay! Well, we are now officially one minute over our time, so I will give you. uh You know uh the rest of your day. Back uh it's nice to see everybody uh I will do my best to Bubble this feedback up uh as chair on behalf of the Sig uh I. Think that there's some great points made uh and thanks for attending today.

C

B

C