Kubernetes SIG Release, 21 Aug 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: k8s 1.16 - Week 8 - Release Team Meeting 20190821

Description

Release details: http://bit.ly/k8s116

A

Hello and welcome to the first midweek burned down for the 1/16 release today is Wednesday August 21st. We are in week 8 of the release. It's wonderful to see all your faces before we get into this packed agenda. I would like to remind everybody that all conversations here beneath has a lovely smile on his face over Neath you're laughing. At me, our subject of the kubernetes Code of Conduct, it's lovely to see that smile.

A

It made me happy, thank you and that's that basically states that we should treat each other excellently and not be jerks, as I've heard it eloquently put. So let's get into this packed agenda for this week. If you're, following along from home, we're in week 8 August 21st you can. The links have been added to the chat and I have taken note that the burndown has the wrong link in it. So I will get that fixed ASAP and you will all get a hundred emails. Each I swear.

A

Ok, so let's get into it. If I could have a note-taker that is always really Pleasant, so I don't have to run the agenda and take notes. So if you would love the honor of having your name in the note-taker space and taking notes, I would be much obliged and very thankful and also attendees. Please place your name in the attendees section of today's meeting minutes and agenda without further me talking, let's get into it. Kendrick with in enhancements, update, take away Kendrick, yeah.

B

What's going on y'all so right now we're looking good everything's green thirty-nine enhancements tracking one drop this past week, API unions went to milestone version 117, so right now we're looking good starting Monday is when we were going to start bugging the crap out of people to make sure that we see all the kkp ours that are going in. So we have an actual tracking mechanism going into code freeze other than that we're good.

A

Thank you mass. Why that was dropped. I haven't looked at it, the API unions. What was a specific reason just.

B

The feature owner just said: we're dropping this for now it's all. They said in the issue, so nice and clean and simple thanks.

A

Kendrick I appreciate the update, see I signal with George, so.

C

First, stop everyone say everyone in the world of CI signal: I, don't really have anything interesting to tell you all, and everything seems to be working at least a at least at this point. Hopefully, it's gonna stay like that for for a long, long time, just a just a couple, things so say signal report if we sent out a after day release meeting last Monday. So if you wanna, if you want to look at it, see a a could get the details into a couple.

C

More things are going on within the community, especially efforts cleaning a cleaning up a CI jobs, trying to a trying to organize everything a if within the release blocking in my with Intel released blocking jobs, check it out other than that. So one big thing we finally have the 116 block and informing you can see it. You can see the links to them within my update, put a 116 block in informing same as master blocking and informing everything. Since evidence is to be okay, nothing not a nothing too terribly wrong.

C

There are a couple: a hit are a couple flakes every every now and then from some age, usually, six storage, a test pills for some reason, but nothing persistent other a other than that they they there is a CA. There is a CI job from OpenStack. They confirm a series of conformance a conformance test running on OpenStack I got in touch with someone from OpenStack, but I didn't think I didn't get too many details. I just got the I just got.

C

They were working on it, so a as soon as I get more of a more of a story. I'll come back to you all and tell it order them that they only failures that we have in your dashboards air early early same, not the only failures, but you know they.

C

The only important failures, I'm, hoping that stereo peanuts and right category are the Devon rpm APL failures, the branch managers or were working on and if Stephen has any of this to give on those I'll keep the Fay I'll keep the floor, but other than that everything's cool. Yes,.

D

I was gonna, try to hack on that before this meeting, but a few more important PRS have jumped in the way. So just a note on those jobs there. They will be important eventually, but they are not important today because we didn't have them before alright. So this is just kind of like working out what the scripts do.

D

The one thing that concerns me right now that was recently filed is this and gonna pop that in too much at that I'm going to do some investigation on essentially the they're missing images for the the marker files, the CIA latest dot text marker file, I, have been poking around I.

D

Think George you've been doing some poking around to understanding exactly how the markers are generated when, where how what they were referenced, so I'm gonna keep poking, but I won't be able to get to that until after this meeting, that's priority critical, urgent, so I'm going to tackle that first and then swing back on the on the Devon rpm jobs.

C

Sounds good yeah.

C

So anybody has any questions. Comments concerns this.

E

Is a time yeah yeah, it seems like there's some I guess: jobs missing from 116 informing, particularly I. We don't have a the cudeman tests. That would normally be an informant.

E

You know their status is on those.

C

Man, this is something that I've been meaning to look. They look up for a while I'm great I'm gonna. Give you the worst possible answer that anyone can give and just say that, as it's been like this, since I've been CI signal, it will definitely make it a priority to give you a proper answer. A next.

D

Meeting so one note there, what like one thing that I think that we need to tackle as a cig overall is I kind of don't like the way that jobs end up on our boards right, it's possible to annotate it essentially now like they've made it so that we can, you can create a job control. Your job add it to appropriate boards via annotations in in the configs for and the job configs intestine.

D

For right, the the problem that I see there is someone can arbitrarily add jobs to any SIG's boards, without intervention from the sig right, I think for the master, informing the master, Baca or the star informing and blocking. That is really important, that we have some oversight, some sort of audit when jobs do and don't get added right.

D

Additionally, there is a another annotation I think called a fork per release, or something like that when a job has that annotation there is a config Forker that runs when the branches are cut, or rather there's a config Forker that you need to run when the branches are cut. That will generate new jobs for for the relevant dashboards right. So those are the jobs that land in in, like the one 16 yeah, no in tests infra.

D

So like kubernetes, config jobs, kubernetes, release jobs, I think, and then 116 yeah Moe right so between those two things, jobs can either autumn-like arbitrarily be added to our boards or not be properly forked. So it's something that we need to look into with tests infra, but yeah. That's one of those. Two were probably the reason. Yeah.

C

Yeah also also just to tack just to tackle onto that, so a or a a so originally the a way when I first came into the when I first came into the release team, my my understanding and the way the way the jobs were is assumed. I, assume, I, know reserved a 116, all the 115, old, etc, etc. They work they work in I. Believe the consensus was that people just talk whatever jobs they want to.

C

Eventually, they want to eventually learn into master in forming within a within the within the old and within they all dashboards. And if we see that those, if we the released him, see that those jobs are actually behaving there and they're being wrong, they meet the release, criteria or release blocking criteria. Then we play, then we bump them up for they for the six put a good. That was a process. He was it an intended for us in the release team and the problem is a as far as I know.

C

That has never happened for any of a for a for any a for any job. That is actually that is living a that is living within the all dashboards. Essentially, it's just a black hole were for some reason: some jobs live in there and no one really. No one really takes care of them, a normal, a and no one. Besides, the patch managers, as far as I know, actually actually look in actually look into them.

C

Then there has been more recently there's been a more active workflow from the community were six is six actively work, a work and work on CI jobs, they proposed him and they and they you know they open pay. They open PRS in a Sulha step in this is they released. Him is tag on the PR. If everything looks good and we had that job into into one of our dashboards and, for example, that was they wait all day. Cuban minion kinder jobs were actually adding into master informing a as Friday they.

C

Never they never actually went through. They dumped them into all they. If they behave, then we, then we bump them into master informing or must a master working. So it's okay, so just to tag that I guess. We also need to establish a policy, a communicated way, communique communicated with the six and- and you know just keep everyone actively looking at it, because jobs are just a kind of a thing: work.

C

Okay, it works now, a a one release a won't release after this everyone's going to forget that we thought we did this a I'm, not saying that they say that this always happens but or a lot of job say it kind of happens. For you know we work on this, we'll fix it and somebody else is gonna. Hopefully a will be able to look at it in the future, and you know it's just that we just leave it aside.

D

For future, so what I'd like to see is now that we so josh has done some awesome work to to clean up the master, informing the release master, informing blocking criteria Docs, as well as the CI signal handbook, so that P are still active. I, think it's just about in final review.

D

If anyone has seen it if anyone is interested in that stuff, please take a look review that once that merges what I'd like to see is a an issue template for for proposing a job to release, flocking or suite of jobs, release blocking or release informing or star blocking or informing right, the idea being there is. We establish an audit log right.

D

We have the conversation in one place great, and you know, because, because all that all that you've mentioned is like, when you know when the release team gets tagged, it's if they get tagged right, it's assuming that the cig is well behaved or cares that we care about these jobs right so establishing a poly you're right. We definitely need to establish a policy around that and I think that we can do that once we land Josh's dock, so.

C

E

F

E

Was just that that those did not get copied over informative, something if there was status on them the because they are in, for example, release 115 min Foreman, hmm the the other one while I just have to file an issue about is I, don't understand why the reboot test only ones against master sales, but that would be a separate issue.

E

My general perspective, a gets one, your question, one of the questions for this, which would be another discussion issue, is: is there any point in having the all boards at all I mean I, don't I mean mostly what they are is they are jobs that we abandon, because no one is maintaining yeah.

D

All all looks like our orphan board, the the current status of like the all boards they're. Basically, eighty percent red- you know like what's the point yeah.

E

They're almost like they got added in there's a annotations when they were already broken. That's.

D

That's probably exactly what it is and again we have when someone uses the annotation sad something to our boards. We have no control over them whatsoever. Well,.

E

D

That's reactive. There should be something that per you know. Maybe this is maybe this is a feature request for a a test that blocks you from adding a job without a required to prove her or something yeah.

E

Well, that would be what it would be in testing, for us is that yeah, yes, that we might need special prowl rules that actually check the annotations I, don't know how that would work, though,.

D

C

So it's only so much so much when items to follow up on this a drug, something of a policy for a for six to actually bump a proposed their jobs to be rate to be wrong within the secret leash, desperate dashboard and a.

C

Clean up the existing scripts I take existing scripts all day a knowldege, a medomak cause. They all dashboard schedules stay alive. That's unreasonable.

A

Yeah there was just the other one of that just asked about why they were missing jobs in 1/16, a foreman. It specifically looks like three cube atom jobs. If you take 115 informing as a as a reference yeah.

C

Absolutely and I'll get back to you. Do you walk with an answer on that I think the simplest answer is you know, people just assign their jobs and they just created the configuration to run against master and and we never a when you're, creating a configuration for all the other branches. Hey, but I'll get I'll double-check and give you a full answer. Yeah.

D

The short version is like if the config, if the job is not flagged to use the config runner, it will mess up every time we we do this getting config Forker. Sorry.

C

Yeah any other questions comments concerns just.

A

One other thing: I see that Aaron, quick and Berger is doing some work to break apart long-running pieces in master blocking the serial job, which seems to be something that runs for whatever n number of very long hours. So I, don't know if you saw yesterday, but he he I think he demoted the HP a jobs from master blocking and he's still working towards getting them out of cereal.

A

Mm-Hmm he's trying to do a bunch of work to break apart that cereal, one, which was the one that was forever broken for two or three weeks, because things were timing out. I just want to make sure you're. Aware of that. Did you see that conversation.

C

Yeah we've been following up on all day all day: the arts that our own SVA aaron has been doing cool thanks George. Thank you.

A

Bug triage with Marco.

G

Hey everyone, so how are we static? For today we have 76 top initialized, including non fleek and some other types. So this is four lines that we had on Monday regarding PRS to be a to over a month there, on Monday at 49 in total number of issues, including flakes, is 89 a dead spider swirl then one day, so no other special updates for Dow. We are going to start pinning everything Friday one day mod is going to be.

G

The latest fall because code freeze is starting like next Thursday, so about two are never about that. They just have a few days to finish all the PRS and all the stuff. That is not critical, so they have time to merge everything before the code free starts.

A

Thanks maka, how are you feeling with that current load 76? Is that feel doable well.

G

Yes, I, we have started pink ink, but there are some problems like when you pink issue that is created by someone who is not a maintaining or some stuff. Then you may not get three sauces of time. For example, I think we had much rather response when we started building PRS and we managed to bring that number Romer much faster like a day or two, but this is not a case for issues we are going to see. How is it going to be until the next meeting?

G

But if it doesn't change, we need to see how we can improve the way we are painting and doing this stuff.

A

Okay, thanks for the update, maka and bug triage team, Doc's uptight with tone day, hello.

H

Everyone so for today we have about 28 announcements, abducts they are good to go, but we are waiting on eleven, so I mean I've, communicated the shadows, among also tracking each other, in announcements and reaching out on slack I, told them also to reach out to the eye announcements and get like a status update on those eleven and I think we should be able to get them in before the place or that deadline. So hopefully, by end of tomorrow or early Friday, we should be great yeah.

A

That's fantastic! Thank you. Any questions for Sunday.

A

Thank You tanden, Thank, You doc, steam release, notes with Kristin hi.

F

So we're on track to address Stephens issue from Monday I. Don't have the details on that. We have the weekly release, notes, traffic proved and merged, and then we amended it I guess there was some bad output. So that's up and that's about all I have for today.

F

A

Thank You Kristin Thank You relation that steam comes update with Taylor.

I

Good mornin / afternoon / evening, everyone happy Wednesday I have a little bit beefier of an update than usual for you. We have been reaching out to a couple of the SIG's we're still having some issues getting responses from some of them. I've listed them in the agenda. I'm gonna be working with Caitlin I might need some help from the release leads in terms of reaching out to those groups.

I

We are reaching out this week to get some blog drafts, set up as well, really looking for teams to have those in by late next week, I'm also going to be starting on the 116 blog templates and running that by various teams. So we can get the ball rolling on that and then tomorrow, at 9:00 Pacific time. I did call a meeting just to review items of importance for 116, just kind of want to open that up to everyone, make sure we're on the right path.

I

Allow anyone to interject or or make sure that you know their feature or an important feature was mentioned in that in that update other than that no blockers and available for questions. If there are any.

A

Taylor so of those cigs that you haven't heard back from, how are you pinging them.

I

So far, we've reached out the slack we've reached out to Google Groups for some of them as well. I think the remaining mode of communication is email that we have not tried across the board.

A

Okay, I will reach out to you and we can divide and conquer those that list I.

D

A

D

Know also, I would say, reach out to Paris or George who do the do? The like sig leads sig chairs technical leads like need to know email, which includes so like a lot of people, may bury the email but gets sent to a group, but that email gets sent to the individual email addresses of all the chairs and tech leads. So maybe, if you need an email Avenue, that's the right way. Gosh.

I

Yeah no that'd be fantastic and I and I saw Josh your comments and in the chat as well. I think that has been a big blockers is just the PTO with with certain people it seems it is tis the season.

A

Thanks for the update, Taylor and comms team, moving on to release branch management with yang.

D

So Jung is actually not going to be on the call I'll give the update. Basically, we've got a beta coming coming soon, or rather we just did a beta. The the next beta is going to be September 4th, that's beta, 2 and Nikita will be handling some of the branch for fast forward tasks. Basically, branch fast forward can only be done by people who are a part of the release managers group. The people who are part of that group are the patch release team.

D

The branch managers, including myself and Nikita and Klaus I believe who have who have access for publishing bot reasons and other stuff that I may not be aware of. Anyhow, we require someone who who has that access to do it. So, given that Nikita is a release manager associate, we want to give her some opportunity to exercise that muscle, so I believe Hannes will also be occasionally doing that if you were part of the elicit the release, engineering meeting, I'm, forgetting meetings now I think it was a release.

D

Engineering meeting earlier this week, Hannes did a demo of a pipeline using Concours pipeline using a leveraging branch password, as well as the release notified tools that we have today pretty interesting stuff. So to test that tool, he wants to occasionally do branch fast-forward. So that's totally cool, nothing super exciting from us, but any questions. Let me know.

A

Thanks Steven that all sounds actually really exciting. So thanks for sharing and thanks for getting the the beta at early this week and the next one, as you mentioned, is in two weeks on September 4th.

D

Oh Slade's, Josh and Stephen. They have no updates. Sorry, there was one more interesting thing from the kind of release engineering side of the house. We just had a meeting in Kate's infra to talk about are rather Brandon. Phillips did a demo of argot, which is essentially a tool that allows you to gather gather packages from currently from github and validate them against a certificate transparency log which is pretty cool. So this is, you know if you've seen the the recent news about, like the ruby gems, the ruby gems repository being compromised.

D

Part of that has to do with like how we validate our supply chain right, so, like once releases get to a certain point. How do we? How do we validate the fact that, like they are released by the people? That said they release them and nothing has been edited since they've released them right. So the demo was pretty cool, we're considering using that for kubernetes, one of the first steps to doing that is and linking in zoom one of the first steps to doing that is.

D

Allowing us to publish sha, 52:56 and 512 sums to github, so the argot tool can actually pick those up. I have a PR in flight right now. That is basically done to refactor some of the release tooling, to do that. So if you want to check that out, I should put these in the dock and not just zoom. Do that in a second but yeah. Those are the issues and the PR. If you want to take a look at them,.

A

Thanks Steven emeritus Leeds update, no updates. That's still the case. Give me a thumbs up. If that's still the case. Okay, we can continue on release, lead update, so given that we're in burn down I'm going to assign a color to the release, because I think that's relevant, given all the information coming in from all the teams and I'm going to go with yellow my main concerns from what I'm hearing just to call them out is probably just in markers making sure that we can adequately get those issues addressed.

A

But happy to hear mark has got a plan, for that. Just seems like a high number in a very short amount of time, but we will keep an eye on that and obviously Taylor I'd like to get the comms team in a good position. So they have adequate information from all those things so I'm going to call it yellow but happy to CCI signal in a green state and enhancements tracking. Well, as for updates to milestones on Friday tonday mentioned, there is the PR deadline to have your placeholder Docs PR in he is looking for.

A

Eleven of those still one of is actually mine thanks ton day, so I need to get mine in tomorrow, so I have a have something to do as well, and code freeze is at the end of next week, so I will send something out on the mailing list tomorrow, just to remind all enhancements owners that they have a week to get those KK P as in and merged, and hoping that, because it's the end of August that people are coming back online and able to address these things.

A

I also wanted to mention the next burndown, so we're on an additional two meetings a week at the moment. So, in addition to the one that we have on Monday, we have one today, which is this one now and one on Friday at 9:00 a.m. and that will repeat for the next two weeks, I sent this out on the mailing list, and then we will go to one every day, so there'll be a Tuesday and a Thursday included as well.

A

That's it for release lead update. Are there any questions for me.

A

A

Okeydoke, open open, there's a lot of very a lot of facial movement this morning on from everybody open discussion, do we have anything that we need a park or anything, that's not on the agenda.

A

Excellent, well, it was lovely to see all of your faces. We will.

D

See them again hold.

A

D

J

A

Yeah, let's walk the board, walk the board. Yeah I had figured for the board walk, we can do the board walk I. Have the board up I would do it once a week rather than every burn down order? We want to do it burn down. Let's do it walk the board. I am just.

A

Hopefully can everybody see my screen.

A

Yes, yes, okay, so I'm going to take a look at the in progress issues. First, we have backporting fixes and defining exceptions. Let me get pop. The hood on this one and I will take a look at the latest here. So that's with Jim I know: Jim picked that up a week ago, no updates Jim Jim on the call I'll ping Jim on this issue.

A

K

He's going to work on it.

A

Excellent Thank, You Gwen, okay,.

A

Okay, so same Sasha has this one, which was a sign nine days ago. Sasha any updates on this.

J

Nothing red, no, no, okay,.

A

You still have it yep.

D

Okay Sasha did we land that PR for the initial update to the template? I know that it was basically approved, but I'm not sure if it was approved by the right people.

K

So another PR a similar one and we are waiting for approval from Nikita or Christopher Chris off.

A

Okay, so this one is related to the updates of the criteria for release for locking jobs. It looks currently like Aaron is making some headway here.

D

And josh has an open PR for that to redock youment some of that stuff. So, okay, Josh, would you mind linking that back to if it isn't already in the scroll back.

E

Yeah, it's already linked to that issue. Okay, it's just a lot of yeah.

A

A

So that one is still in hand released blocking job criteria, dependency management, improvements, umbrella.

D

So we hadn't mentioned this I think on the Monday call, or maybe a different call that this is probably good to close. Soon. It's kind of its kind of one of those, like definition, have done things where we've done quite a bit for the cycle already, but we don't want it to stay open forever, so chunking down whatever is remaining to do and breaking those out into separate issues and then closing this one out.

D

A

Lose temper yeah.

D

Okay, so leave, as is for now yeah and we'll we should. We should on release team it, because it's really more release engineering and it was it existed before. Release engineering was a label or really stood up.

A

Okay, so we just removed that one from this board now yeah.

D

Yeah, let's say yeah so removed the yep, remove it from the board and remove it from the remove the release team label. Okay,.

D

Is it remove remove area space, no space area, yep and.

F

D

Maybe not maybe doesn't work like that big, so area area slash release team, well space, so remove area, space area, / yeah, and that should work out you if it doesn't. If it's about.

A

So it's just release team, okay, yeah. Let's leave that there for posterity and go like that right, yeah, okay, well, I've bought debugging Zoey. There we go and the label is gone. Okay,.

A

We did this one, we did this one, so it's just this last one that's looks like it's assigned to it closed, Sundays.

C

So the last ones will be assigned to me and hold send up er today this one about this. One is just documented that you saw a labels labels thank.

A

You George in do we have membership criteria? This one is on me. I'm gonna, take a stab at it next week. Can.

D

You can you pull me from that. One then yeah.

A

Yes, I will, quite happily can I just on the site. I'll just do it. This way is that gonna break the bots, no.

J

A

Okay, okay and I had information about what qualifies for milestone. Jeff has that Jeff is out on vacation this week.

A

Yep, okay, so that's in-state! Do we want to pick up anything more from the backlog? Will walk the backlog, yeah.

D

I mean there are only nine, so, let's see if we can drag stuff in.

A

Youtube automation, request for release team.

D

Yeah so essentially, I would like to probably see yeah I'd like to see it if we can turn it on for all of the release, team meetings in or all of the cig release meetings. So for the context.

A

Of this, we should just get it done for the relates tape, so that it's automated yeah.

D

So that way, you turn on the meeting and it automatically records and when it's.

A

D

See YouTube I will.

A

Follow this one up with Paris and George cool, let me put that I can.

D

Can you I was in community right, you don't have at it. Yeah yeah, okay,.

A

This one is success, seven add release cutting issue and screenshot to release branch management. Handbook can.

D

You assign that to me, I'm I have a branch manager handbook update in progress, and now that's now that we have, which we call it so I did some updates to the branch manager handbook that removed some requirements, so that will be part of it.

D

Hana saw also has an act of PR for adding the snapshot tool to the release repo, but I should note that this isn't really a release team thing anymore, its release engineering, so we can pull it from the release team board, as well as remove the release, engineer, release team label release team board going.

A

And I now know how to do this.

A

A

Consider linking from release notes to definitions of code, slash code, freeze and enhancement. This looks like a an Aldi if there's still code slush, yeah.

D

A

Carry release team.

D

So that's currently help on it. If anyone wants to grab it.

A

Can you take us Bueller.

K

Release notes, thank you. What does that mean from release notes.

D

So from the schedule yeah.

A

Sorry they just want a glossary of terms, I guess of what these things mean. Sure I can take that yeah.

K

Sorry about that, um that's weird I, see, release, notes and I'm thinking. You know yeah.

D

Yeah and I think this is from the context from someone who isn't who's not been on the road.

K

For sure, that's.

D

Why I asked I'm.

A

Going to just put this in to do was move it in progress once it's done, we'll just pluck it off the backlog.

A

Investigate automation to facilitate tracking spreadsheets I, know I know bob has been doing some work here, I'm, not sure. If Bob's on there on the call.

A

Let me ping Barbara guarding that one released him Saturday in Britain. Sorry, I thought.

K

Bob was still out on vacation, okay,.

A

If Bob's out on vacation I will not ping him.

A

Does anybody know about this? One release team survey, improvements.

D

Yeah, that's that's on the EA's, so I think where we're at I think this one got juggled around a few times, but it should be squarely and Josh in my court now so I think that you know part of it is. We should review not just the survey, not just the survey feedback, but also like the exit interviews and all the stuff from both the shadows from last cycle and the and the leads from last cycle and turn that into an even better survey for 117.

A

Okay, it's there leave it on the backlog for now put that in to do.

A

Okay, try splitting release schedule out of release League playbook, so.

D

I feel like this popped up in a different issue and kind of an idea that I had around. It was maybe turning this into Hamill right being able to specify the and then having that pop up on a really like a micro site kind of thing, where people could just go and view it outside of the calendar.

A

Yeah, let me take a look at this. It's already assigned to us.

A

Okay, I'm going to leave it on the backlog, already have three tickets assigned to me and I want to pick this up until I had a little a head room, but I think it makes sense, I'm handling the exceptions, yeah moley files, so I I.

D

Don't know what we want to do here, this one is kind of what is the value right like? Where would we put this right so like we? We call it the we collate the exceptions right now but like who is it useful to, and where would we put? It would be the questions before doing anything here. I think this can stay on the backlog. Tim I, don't know is this that stated in here yeah, but it was like. We could write a tool to do something like I really know what it is yet.

A

Who is this useful to and where? What was the other way would review it? Yeah.

A

Okay, let me see if Kenny has any ideas, I mean Bob might have a few as well document use of milestones for issues related to CIF, Lakes failures, I think George,.

C

Yeah I can do put in my in progress. Please this one is essentially a result of misinformation and miscommunication between a lot of people in the community. A couple releases ago there were there were a lot of issues with nade know a with a lot of people, not knowing how to add a new issue or a PR to a given milestone, and that issue just has some comments on it.

A

Okay, the final one here was the shadow halo process which I opened I will take an update as necessary. There anything else. We have one in progress: okay,.

A

Excellent, any other questions regarding the board.

A

No okay, excellent! Well! That completely, are we ready to call it I.

D

I

Sorry I just thought: I just wanted to let y'all know that like I went through and kicked out the super slow tests out of the cereal chop, I, don't know if George already reported on that I posted something in the sig release, channel, saying: hey, I! Think now that we have this dashboard that shows the health of the release, jock released, blocking jobs and how well they adhere to the release blocking criteria.

I

We should act on it, so I gently, poked, Sega auto-scaling to get rid of their HPA jobs or HPA tests out of the serial job and sig note, they're gonna find other ways to bring those back on to the port, but the serial job has been passing since we did that it's gone down from its path going over its 500 minute, timeout to 300 minutes. This is something I kind of hope that this team takes on going forward.

I

I took care of the super low hanging fruit, but I still kind of think that serial job runs way too infrequently and takes way too long, and the question needs to be asked like why do these tests matter? What is what is it about them? What is it that they're exercising that makes them released blocking so.

D

So we had mentioned it earlier on the call and I wanted to ask the question to test infra. Is there a way to enforce the type of jobs that end up on a certain SIG's board? So one of the problems that I have is that people can arbitrarily add jobs to those those boards right now via the annotations.

I

So your best mechanism for enforcement right now would be the set of tests that run again. We have a couple tests spread out as you notice them. The branch manager handbook. There are tests that enforce like proud job specific conventions, and then there are tests that also enforce test grade specific conventions. So, like the test grade convention, job is the one that enforces like release.

I

Blocking jobs have to have alerts set up or that there have to be descriptions, so you could probably enforce that, like these jobs have to I, don't know, have belongs it to something or whatever. So, alternatively, you could maybe write something. That's like any job that uses this particular test grid annotation has to live in a source path that matches this regular expression so that you could gain based on like owner spots, yeah.

D

You you, you read my mind, says everything like if moving forward. If we had like you, said any job that wanted to be added to blocking or informing would have to live under cig release, blah or kubernetes sig release, blah blah blah blah blah right.

I

So that's that's my suggestion for how you can implement that. If that is a convention you wish to in horse. Thank you. Thank you. Okay, now let y'all go sorry. I just thought. I wasn't clear to me whether folks have noticed that that that happened, but that's gonna be the end of me chasing after people, for that particular thing and.

A

I

A

Yeah I was just gonna, ask Aaron, actually I think it's worth just stating aloud. Thank you for all the hard work you've done, Aaron to clean up cereal. We do appreciate it and, as you know, that had been failing for several weeks first that job the would this come to your attention via velodrome I, was on the community call and I watched your velodrome updates last week. Was it pretty much looking at this? The average of all the job runs and saying these are the three that stick out like sore thumbs.

I

Yeah yeah, don't like the this is the dashboard I was talking about like ie I feel like when I I forget, which at least I did it right, I laid out the release. Job released, walking criteria like we expect any job that's released, blocking should adhere to these criteria and then nobody implemented anything to actually measure this and I. Don't know that anybody actually acted on this policy other than some like battling back and forth to kick the scaleability jobs over to warning I.

I

Don't know that we ever like measured this and like policy is cool, but if it's not acted upon, I, don't really see the point of it and it's kind of unclear to me whether see I signal has ever like. Like again it's it's, maybe a lot of toil for humans to notice this and measure this. So this is me trying to like wrap that up and make sure that we have the tools in place to actually hold our jobs to the criteria. We said they should be measured against.

I

I, don't think we'll ever get to the point where, like a bot, can automatically open up a PR to take a job out if it's demoted, although like this, is the way that some CI systems and some large companies who shall remain nameless, work, I, think quarantine, tests or jobs that, like are misbehaving and I, think it'd be cool. If we got to that point, this is I am hoping a tool that allows the humans on this team to make said judgments about whether or not we should quarantine things and I'm there.

I

I have an issue linked at the very top of that dashboard. If people have suggestions on like how we could improve the usefulness of this, my hope is that this is something that community could use to just like drive forward and improve the health of their jobs in general.

I

It was unclear to me how much the the previous dashboard the focused on like a bunch of random jobs or a bunch of random repos, was being used by people and then just kind of continued that I guess, like the one release blocking criteria that is not represented on here. That I haven't yet figured out.

I

How to write oracle sequel to measure is how many commits how many passes in a row on the same commits does each job generate, because one of our criteria is that each job should be capable of passing three times in a row again and commit? How does anybody here keep track of that yeah I'm.

D

Not sure that anyone does so.

I

I know that was like that came out of tribally what I haven't seen the release team do in times past. You all tell me if you still do. It is like when we're getting ready for the release, to go out the door to give us comfort and to feel like we have appropriately soaked the release. We kind of wait until we see three runs of all the jobs and, if they're all green, we're happy.

I

If not all of them are green, then we kind of try and read the tea leaves based on how frequently this jobs and failing or flaking in the past and one of the issues there are, and what's the I signal spreadsheet looks like and so yeah that's where the three commits thing came from, but I don't think anybody measures it, which is why I flake grades being displayed right now as opposed to commit to the wreck. So somebody wants to help measure commits in a row.

I

That'd be super cool or the other thought I had was you can use play grade to sort of compute the probability that it that a given drop will pass me time server, yeah.

D

So I think you listed that too again. Anyone who hasn't had a chance to look at Joshua pyaare to update some of the blocking criteria in signal it Erin mentions how you would calculate the probability there and one of the review comments.

I

Right 20%, like babies, you have a 50% chance of passing few times in a row.

D

Yeah I stared at the math, that's like cool math, alright,.

A

Okay, anything else thanks Erin.

A

Okay, fantastic I'll, see you all Friday at the next bend down meeting. Thank you, ladies bye.