Kubernetes SIG Release, 30 Jul 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: k8s 1.16 - Week 5 - Release Team Meeting 20190730

Description

Release details: http://bit.ly/k8s116

A

All right, good morning, happy Monday welcome to the fifth week of the 1/16 kubernetes release cycle. This is the release team meeting the weekly one. um Our current release team lead, Lachlan Evanson is out today, so my name is Guinevere Sanger and he will be hosted by me today. This is my first time ever hosting such a meeting, so if people would be kind and remind me if I forget something that would be delightful, um I will remind everybody at this point that we have a code of conduct.

A

The TLDR of the code of conduct is this: please treat each other with excellence, be nice be kind and let's move forward. Can I have a note taker for this meeting. I personally am terrible at talking and typing, so I don't see anything in the chat and I don't see any hands.

A

We can have a very long meeting where I look down and type and I.

B

Gotcha Covered thanks.

A

Bob right, ok and then without further ado. Let's go and look at the agenda. um Jeff put a lazy link for the agenda in the chat, if you can't find it, um which is basically me every time. Every meeting cool um put yourself down in the attendees list on the release team meeting and we should be ready to go first up most important today, given that enhancements freezes tomorrow, Tuesday July 30th.

A

What is the status of our enhancements and how can we help make everything work smoothly? Kendricks, hey.

C

How's it going gonna, be so everything's going pretty well right now with enhancements we're kind of sticking by our guns and usually what we usually do during this time. So right now, there's 51 that are being tracked. This was after or initial push going through every open issue, making sure that we touch every single one of them and getting back a response from the issue owners. We we would say that if we do not get a response back from the issue owner, we are not going to do anything.

C

However, it's something somebody has said something it is now being tracked. It is in the sheet with inside of there as I wrote down, there's there's a few of them. I believe I can't do my math right now. Eleven that are quote unquote at-risk and that's because these are not.

C

These do not have a Kip or missing some kind of graduation criteria, letter associated with them, and so you can kind of see the breakdown in the sheet right now, we've got 24 alpha 16 beta, stable at 11 I didn't get a chance to get the account for the at risk. For alphas here before we started, but I love being able to see that there's 11 stable features and one at risk right now. So it's really good to see going into this type of release going forward today and tomorrow, myself, an enhancement team.

C

What we've been doing is we're gonna hit every single one of the at-risk features, talk to the owners, see what needs to be done and if they will be able to get anything resolved by into day tomorrow, making sure that we just nudge them a little bit to think so they know exactly either.

C

What's missing for their particular enhancement to move forward, the only other thing kind of really funky is we were kind of going through these and I believe Bob made a note about this was that number 508 and 1138 are both talking about ipv6 I. Believe 1138 should probably be rolled into 508.

C

I made a comment here just before the meeting started, so once we figure out that we'll probably be able to knock that one off the list, so I would feel confident right now we know what's going into and handsome fries tomorrow, I'm just putting it yellow, just because yellow but I don't feel that anything is off track. Bunny needs.

A

That sounds awesome. Are you talking to the sink, that's in charge of 508 and 1138? Do you need any help in turf with interference there? No.

C

Not yet I believe we just need to make sure that the 11:38 is conscious. That 508 exists, because all it is is talking about. 1138 is saying we need to upgrade. We kept to a beta state for ipv6, which that should probably roll back into 508, which is saying we. This is adding ipv6 support. So that's why I was saying it should probably be collapsed into it. Yeah.

A

Yeah makes sense um yeah, let us know if there's anybody being confused or needing to talk to people in charge of the sig. um How does how do you feel about the let's establish who is cutting the alpha release tomorrow? I'm actually not aware.

B

Branch manager- that's gonna, be young.

A

Okay right- and he mentioned he was gonna- be a little bit late. So let's bring that back up later in the meeting um cuz. These two are I connected. We might make sure we get everything in the alpha release. That's ready and nothing else, um excellent awesome. Does anyone have any questions cross cross, lead concerns.

C

Also shout out to Bob for making some more enhancements. I guess you could say to the enhancement tracking spreadsheet he he put some some visual basic magic in the background that well first off fetches issue numbers, but now, as we're gonna find out going into this tomorrow, if you put something as a deferred or removed from milestone, there's now a new magic button, you can hit that will automatically take out the line and move it to the other tab for you.

C

So thanks Bob for making sure that he can put some more it's a visual basic, not visual base, I'm thinking of like it's JavaScript I'm thinking of using like Excel everything Excel, you had all the visual basic stuff, but on the background, that's what I was thinking so yeah JavaScript action thanks, Bob yay.

A

Thanks Bob awesome Oh anyway. If anyone does have permissions to write to the doc, let me know and I'll figure it out for you.

B

Join the sig release. That means you're, not part of the sig release, group or you're, not part of the release team Google Group. So you should join that.

A

Thanks Steven, okay: next, let's talk about CI signal and it's looking a little bit worrisome right now, as I was watching things get filled in. We.

D

Are at a red with CI signal we have in master blocking three failing tests for flaking and seven passing 804. Thirty-Four is passing, which was a fail from last week. So that's good for failing. We have eight or seven one, seven as a new failure.

D

Six storage tests are failing with time out errors. This looks to be a token like totally new tests and basically, as soon as they were introduced, they've not been working. So I've added an issue there. Seven nine six. Six two containers continues to be failing from last week. There is a proposed solution in 806, one five, so hopefully that'll be resolved pretty quickly and eight. Oh seven, one nine is another failing issue. Signor sig network failures.

D

They look to be caused by a misconfigured test, so hopefully that gets resolved pretty quickly as well. In the flaking tests we have differing sources, have been flaking continually in GC. Igce ingress doesn't look like there's much of a resolution there he and massaman forming. We have three failing tests, actually I think there's only two failing tests, I thought: eight, oh seven, one, five Oh yep, sorry, three failing 807 one five is the build package. Depths and rpms are failing. So those are two separate tests, but both of them are just straight up. Failing, hey,.

B

Yeah so I actually merged. So that's an addition to our test suite. It's actually check that we can build the dubs and rpms what I did. There was merge the new tests before merging the PR that adds to make targets for that test, so that's fixing out that is merged, so the next run, which should be it should be already in progress, actually should fix it. Thank You Nick for catching that Thank You Hannes for approving the PR.

D

Cool and then 807 1/9 also effects master informing. So hopefully, when that resolves, we won't be having any problems in medicine. For me, anymore, that's I said a little bit of just seven, eight nine one. There are some flaky reboot s going on and then one stale, the open set conformance test, hasn't been run in three days and I'm, not sure why I couldn't see any information around that, but I just knows.

B

That and I kind of want to pull that one if no one's giving attention to it, especially now that the SIG's have have collapsed all day.

A

B

Cloud provider sakes have collapsed, so I'm gonna check for owners on that one and then maybe get rid of it. Yes,.

E

Or just a deal to your own, a OpenStack, they actually do maintain it. They are fairly responsive lists are a very good handful of times per release. They job Costel for is unknown.

B

Reasons is it a job that only triggers on on certain file changes no.

E

One's forcing a source, I know it's, it's actually one of these ones, probably at least once per day the things that they've run it on I forgot whatever on it, but they don't actually use the same structure that we use for all of the other tests. So there's always some way.

A

Sorry to interrupt um is there anyone here, maybe you George, that could um talk to OpenStack and to connect them with the CI signal team? Yes, just do that. That would be great, because then we can either we can do it. Then we can have a more informed decision. If you say they're responsive, we should make them respond and if they're not responsive, then we should pull them.

E

Yeah definitely.

A

Awesome Steven, your mic is still on. Were you done talking.

D

So, uh release miscellaneous, there's three flaking, nothing too alarming, so nothing with reporting and then for passing I'm not going to turn over to George with some more information around some upgrades downgrades q tests that are going on. Okay,.

E

A sorry for interrupting, but I'm back, and so the last update is a continuation from a week's long conversation.

E

So a printer shorts tape, a just a summary is we used to have a lot of upgrade down with a dumb retesting master, informing they move to ca to sick orphan. However, they configure the configuration for those tests is also replicated in multiple dashboards. For example, one of the a more a one of the places is the 60 line, desperate for in which they test a cubicle skills.

E

Sorry I cut off I could over there for a moment, so is so upgrade it down. Grit and cutest are relatively common out see in are relatively common configurations in industry.

E

We discover the Turner is a little bit of an issue with the configuration for all day for all those tests, for example in in the city orphan a dashboard, we have a gray up grids now are going from 114 to 116, where they should be going from 115 to 116 after couple PRS in a lot of conversation, a demo at the most important one of the most important ones during the kubernetes testing front, 13577 PR, with a with a on a turns out.

E

The third is missing state, there's a step in the release process that we've been missing since around kubernetes 1:12. So so big idea, there are five release channels that we use for is for referencing a given release, for example latest. It's always everything. That's in the master branch beta beta a better right now, it's 1:15! Then then we have stable one stable to unstable tree, which right now are about forty motor teen 112.

E

So the beta channel it was intended to test the the beta channel was originally intended to be a type of conformance test between the release.

E

In this case they released 116 branch and they master branch when we actually hit code freeze boom in so when we enter code frist, we are supposed to we're supposed to enable the way the beta channel to be the release 116 after we go out of the code, freeze, we're supposed to deactivate the beta release and we are supposed to roll forward the rest of the channels. Sort of is all stable. One is stable. One is in this for the case of this releases 116 and stable to 115 stable, 3 114.

E

If we follow that configuration, then all they operate down with a downgrade and skill tests within the organization are going to be correct, but India. Actually, that's that's the gist of the story.

A

Yeah I was just looking at it. Thank you so much for following up on that, it looks like looks like we have a. We have a proposed solution all right. Well, we have a proposed path for a solution. Yes,.

E

Exactly we don't come up proposed solution at the beyond that I actually created. It was a hack around using what we actually have right now. The actual proposed solution will be I'm guessing for the branch management team to fix it doing the managers, the channels yeah.

A

B

It's I mean it's less tooling and it's more PRS. It's I think it's I, think it's less a lack of tooling and more a missing step in a handbook, so yeah.

E

I mean we've never done brunch management, so I'm, just guessing no.

A

It sounds like it sounds like a process um discrepancy. Basically, um people did agree about naming, so we should. We should, yes is the is somebody from the we will. Let's pick this back up when the branch manager, oh hi, Steven, go ahead.

B

I'm here and I think Young is here too yeah, so I will re review some of this stuff. I think there have been comments in the last few days that I want to make sure we have our ducks in a row before doing anything, I'm also working on updates to the branch management and the patch release management handbooks to kind of bring them in sync and hopefully point out some of the discrepancies in the process through that. So that should happen this week.

A

Awesome Thank You Steven.

A

Very good: um do we have anything else, him ci signal.

D

Nope, that's it thank.

A

You so much, let's move on to the somewhat related, but not really bug triage and Sara yep.

F

Okay, so a quick update from the bug triage is that the number of open issues is at 79 right now, that's plus 4, compared to last week regarding the PRS, we have 63, which is 2 more than we had last week, and a total number of futures we have, including tests related.

F

One in Flav's is 96, and this is five more that we had last week at this time, I have went through some of the PRS and issues I have it found something that is critical at this stage, there's some other ones that have been kept. A good record of a moved from release to release and I will keep a day of the MSE. Can we get them ready for this one and are they really needed?

F

So we gotta be back to our spreadsheet. I, have made some sub projects and told about my shadows about them, so I think we will be working on approving that one for this recycle, if possible. I think this is it from my side for now looks.

A

Good, um do you overall, tent orange, yellow or not no decision at this point, yeah.

F

No decision at this points, mostly regarding date, number of issues. Okay, we are still at big five long time at the pod freeze, so numbers are going to change a lot and we still need to pick an issue so far, I think yeah.

A

Great, thank you for watching those next up on the sheet is Doc's hello,.

G

um This is a Sunday from dark, so for doc. I do we have much updates right so, but one thing out, we've been doing. These we've been tracking an announcement sheet and we've been following up on the couple minute: announcements and I know tomorrow is um and I spent free, so we should get more accurate. Vo of like you know what is going to go into the next release, but so far everything has been going.

G

Good I've been getting like a quick response from people which is very, very good and I'm going to say yeah good right now so I'm going to lean towards, like that green cuz, we are kind of like Eddie doors are go.

A

Excellent, it sounds like we are establishing good habits. Yeah awesome. Thank you too. Nick.

A

So that was that was short and sweet release, notes yeah.

H

Hi, so we all cream and we have a new release- notes draft for this week. It may look a little bit different than the last week because we are working on some some improvements like multi-line support for them, so it's waiting for approval and then the release notes website is able to display these additional documentation section and if you can't imagine how it should look like, then I create a little gift for you, and we also edit enter and test for that today, and we are planning that that we are also able to search them.

H

We are their search field and filter them as well and beside that, we are working on a plan on the automating release, notes updates for the website for the website, and we created an umbrella issue for that and the basic idea is that we create automatic POS against the release notes website. Once kubernetes release has been cut and yeah Jeff. Do you want to add anything to that? I.

I

Mean you pretty much covered it. I can go into technical details, but TLDR rewrite a little bit of bash from the rel notes. Bash script.

H

Yeah, that's it from all sides did.

A

You just say bash on camera, very good, very good I, really like that little gift- just oh god, I can't remember now, um anyway, I really like it. If somebody could pick that up and place it in a location where more people can see it, that would be delightful because I think it's really useful just better than words. So.

B

Yeah so I sent an email to kubernetes dev, maybe a few weeks back about this feature. If, if someone wants to respond to that threat at least keep it organized that'd be cool, but this does look really really awesome, so at tip I'm.

A

H

We also maybe have to mention that it's just a constructed example- and this is not an actual release- notes entry for the next release.

B

See so one question about the tooling: is it possible currently to run release notes for an arbitrary repo.

B

Yes, yeah, okay, okay, I'm.

I

Not I'm, not I'm, thinking about will to use cases behind. There were several sorry Steven there. There was actually another kubernetes sub project that was interested in using it and and I want to say to release cycles ago we I VAP, II art and support for doing different repos than Kay Kay yeah.

B

I saw I saw that and I wasn't sure what the status was, because I want to use it for cluster API as well. So the.

I

Only thing is, it still has to adhere to the standard of how we do things in KK, but you can point it at a different repo, okay, cool.

I

If not from an issue and assign it to me.

A

Very good mmm that sounds really awesome, so I'm happy to hear all that cons, Taylor good.

J

Morning, everybody happy Monday, so this week we have pretty much read up through all the features and release notes, anything that we might be able to do. You know spread communication and help out with the release. Notes cite in terms of getting those github issues tagged more than happy to jump in and help out on that front.

J

This week our team is going to be reaching out to these signals and saying, if there's anything of note, that they really want to focus on for the 1/16 release right now, we have, you know, we're starting to form an idea of what that might be, but we're gonna be in communication with all of you as well as Doc's as well as we go through that process.

J

So this week, we'll hear back from the cig leads, then we'll circle back with everyone else to make sure that we are good to go on all those fronts other than that no blockers again happy Monday, everybody.

A

Fantastic um all right now release branch management. We are trying to I, see, there's an alpha cut coming up tomorrow, so young, can we hear about yeah.

K

Yeah we are going to cut the Alpha to release tomorrow and I will be doing it, so it will be earlier, maybe a few hours earlier than us time. So the reason is, we are not very ready to. Let associates to do it. So we are. We will update the instructions in this week.

A

Good I'm, this might be, it might be a confused question, but how? How are we feeling about the state of our alpha enhancements and the alpha release branch? Is there any conflict there so.

B

There is no alpha release branch. The alphas are cut from master, we don't we don't branch off until beta, so so everything that was mentioned in the CI signal section is where we're at basically okay, lots of failures and all that good stuff. But cleaning up. Thank.

A

You okay, thank you for clarifying, so.

B

So one note about the associates: handling cuts, it's not that we are not ready to do it. It's more about! Thank you, but it's not that yeah! It's not that we're not ready to have an associate do that. It is about building trust, essentially we're handing someone the keys to the kingdom to cut kubernetes. So it's not something that we take lightly.

B

So, yes, instructions will be updated this week for the management and patch release management handbooks and, as we see the way you build that trust for the release manager associates on the call is to get involved. If you were dropping pr's, if you're talking to us you're reviewing things you're on these meetings, like that's how you build trust, so keep that in mind.

A

All right very good um sounds like we're on track and I see that our esteemed and there it is leaves have nothing to report.

A

I gave the update in the community meeting on Thursday I mentioned that we're having enhancements raised tomorrow and that we're cutting releases.

A

That's literally all I, said I said we're all trucking along just fine and everything's gonna be great great okay. So we have a couple of open discussion items or actually just one Steven, hey.

B

So we've been kind of going back and forth about some of the release. Engineering things I think we needed to lay down some of the groundwork. So now that some of the new tests are in and and we've done some analysis on the oh, my god, there's a fly, we've done some analysis on like the current work. That's been been happening and Kay release I think that we're ready to start a release, engineering sub project meeting, so that will be happening soon. Tim and I were chatting and I think that we were saying that.

B

Maybe the right thing to do is have it on the alternating week. That's the sig release meetings happen in the same time slot, but also give us a chance to change the time slot. So we were tentatively looking at 1:00 p.m. Eastern or 10:00 a.m. Pacific, so we're going to send out a doodle for that just want to make sure that people are aware on this call because you may have overlapping interests and expect that soon I'll we'll chat about it on the sig release meeting tomorrow as well.

A

Yeah, that sounds that sounds great. um Do we have actually? This is this? Is something I've been wondering about, um so the release engineering stuff project seems to be focused. A lot on. You know how? How do we coordinate the release and the tests and that sort of thing I'm wondering if there's any kind of any part of it, that also sort of focuses on automating the release cross just a little bit more there's.

B

A large portion of our release process, that is, that a CI based so a decent chunk of it is automated. There are levers that people have to pull, but some of those levers are basically like looking at something that CI has run and saying like: okay, yeah we're gonna, let this go right. So yes, so, yes, that is like. That is definitely the goal to take the humans out as much as possible. But there are certain things like signing the packages signing the Deb's and rpms and and then publishing them.

B

There are app tinium repositories that require human intervention.

A

I'm not trying to make humans obsolete soon.

A

Okay, very cool, so basically anything that kind of improves the release process would sort of thematically belong to that sub project.

B

Yes, I think we are all all sub projects of cig release are in the game of improving the release process. Release engineering specifically handles the handles a the tools and processes around the automation. Are the infrastructure that's involved in cutting the release, okay, suppose to the human aspects of it very.

A

Good I understand. Thank you for clarifying um awesome. Do we have?

A

Oh, we have another. We have another open discussion da, hey.

L

So I think we have been doing this in the past. Oh really, there's two: we have been the automation around how they release jobs are removed and new ones are added, are done together, so the timeline says we're doing it next week, but actually this it needs to be pushed along with the creation of the branch and creation of the new jobs. Another.

L

The reason I wanted to bring this out here will stand if we can modify the base template where we create the release timeline from so that we don't have to now tackle this in every release, because last two releases also it was similar where it was separated and we merged them together, but I think it's coming from the template rather than because you copy the template and it actually happens from there. So if we can modify the base template itself, where you know how we define the city's job, can you link the template I'm asking.

J

L

I'm asking for the template: I, don't know where the template is ah yeah.

F

D

L

I've been generating full requests, every release to make this change on that particular release, but I think it's time to change the template itself. Yes,.

B

So this will just I mean that's something: I need to find because that'll be part of the documentation, updates that I do okay, yeah.

A

This sounds like um definitely file an issue in sig release and tags, even on it. Okay, thanks, that's that this is a totally reasonable and obvious suggestion, so thank you bringing it up.

A

All right um any items that we touched on in previous meetings that people would like to come up come back to anything. I mentioned earlier that we could discuss later.

A

Anything that somebody managed to not get that somebody did not manage to get unto the agenda all right, whoo.

B

Yeah so one thing I forgot.

A

25 minutes I'm.

B

Also partying also partying, maybe it'll be 24 minutes. So anyone who has things to talk about please remember that, like we have a tiny, we have a tiny like meeting. Invite that says that the sake release schedule is closed or agenda is closed for the week at the end of Monday. So if you have updates- or you have topics that you want to talk about in the larger sig release meeting tomorrow, please put them on the cig release agenda.

A

Thank you for that reminder.

A

Very cool all right keep on keepin on great work. Everybody. Thank you so much for having me as your host today. um Let's see you next meeting will be in a week for week, six and I look forward to seeing all of y'all spaces.