Kubernetes SIG Release, 5 Jul 2023

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes Release Team Meeting (1.28) for 20230705

Description

Kubernetes Release Team Meeting (1.28) for 20230705

A

Right, hello, everybody and welcome to the July 5th iteration of the kubernetes 128 release team meeting. um These meetings are recorded and uploaded to YouTube. So if you don't want to be on uh feel free to keep your camera off, there'll be a time after the meeting. If you don't want to be on the meeting um and I'd like to inform everybody that these meetings are subject to the cncf uh terms of co-coduct so uh be sure to adhere to those uh for those new to these meetings.

A

Those pretty much just boiled out to be nice to everybody. um I've dropped a link to the meeting notes in the meeting chat and um if everybody who's in the meeting could add themselves to the attendees. That would be super helpful, I'll pause and give people a minute to do that.

A

All right, um it does not look like there's any open discussion meetings or any open discussions items now so we'll skip that part and move right into the uh Sig enhancement for them. Sorry,.

B

Not even hand fits.

A

uh All right uh are you here for the.

C

D

C

Yeah, so everything looks good from outside. Now the current status is green. There are no new updates and the code phrases uh in approximately two weeks from now uh the 19th of July, so yeah, that's uh the latest my stone that we need to acknowledge right, yeah, that's it.

D

A

E

A

D

E

Our current signal is yellow because we have been observing a video on the most informing board from the last six days. It has total in total sale 46 times till now, other than that we had a feeder on the master blocking board, but it was fixed in appear so we'll be observing for a few runs to make sure the fix is good and V1 1.2. It also is scheduled for tomorrow or day after tomorrow. It depends on the time zone and Mom said we'll be giving the Google signals.

E

A

Office for a couple of days so.

D

Hopefully, and to the bottom of that so sorry about that.

A

Thanks a bug trash.

D

Yes, hello, everyone back to your status in green and you don't have much updates to speak, but we have started to reach out to issues apis that we think that might be at risk as the course presence coming up. So we are constantly tracking down and that's it.

A

For us three attacks.

F

All right, yeah, our status is green and we finished reaching out to all cap owners for the next stop start line, and that's just following up on that.

A

G

B

Screen and this week we will be working on with the six to up in for future blocks and update. You know update that.

A

Okay uh release Branch management.

H

Hi uh status is yellow because one of the master blocking job is was in red and so this morning, so I'm waiting on CS now to give me the go tomorrow,.

A

D

A

I

I was I was on mute, I I, don't have any updates, um but keep in mind if you have any problems throughout the release. If you have any questions, feel free to contact me also, I already gave this update last week, but the Retro is coming up. So keep this in mind at your items to the Retro. This is a very good opportunity to improve the program, to also share your thoughts. What is going well, what is not going well so use this meeting.

I

um Looking forward to it thanks. Everyone.

A

All right great um Grace nicely.

J

Yep, um so we had our first APAC meeting on last Friday shout out to Brad for running it. um We will have a pack meeting every Friday now, um so our next one is in two days, um so we are going to do um intros into different sub team and we're going to start off with um intro to enhancement with the chart love you want to share your screen or anything from java.

A

Let me stop sharing my screen too. Then all right go ahead.

C

Let me see my screen now yep, so this is just a basic introduction of the over layout of what an announcement team looks like what responsibilities are included while tracking the announcements. So, uh okay, why isn't that's working yeah, so the enhancements threes uh generally happens on the fourth week of when the uh release starts. So until the announcements freeze, the announcements team's job is to keep track of all the issues.

C

Announcements issues that are that are being opted in I'll talk more about what announcements freeze means later on, but after announcement series is over, there is around a four week Gap approximately after which the code freeze happens and the announcements freeze generally Works between these two freezes. So all the works that that happens, that is included with the announcement stream, is generally between announcements, freeze and code fees. So what is an enhancement?

C

So announcement is basically a feature that the user or the developers wants to uh what is hoping to uh inculcate in the kubernetes project. So the uh I have mentioned a bunch of bullet points, but what the general idea of what an announcement is that something that might uh affect the uh kubernetes Point kubernetes project in general, which may be the kubernetes uh core core components like node or some some, uh the architecture or the API components of the kubernetes so uh and which might also requires a graduation criteria.

C

That means a feature which might need uh which is affecting the users, which will require stages like Alpha, graduating to Alpha graduating to Beta or stable. So uh this is basically an idea of what an enhancement is. So what is what not is an announcement? So announcements as I said announcement is just a feature which might affect the kubernetes core components, but the basic. uh Sometimes the announcements uh might not be related to something: that's just a basic uh fixing of some basic fuel lines of code or which is just tweaking some uh talking.

C

Some changes to make the tests uh pass as something that is implemented by the crds or something that is creating new crd. So this is what not an announcement is so so what so? So we now know what an announcement is. So how do a developer? How do user propose an enhancement, so the process for the kubernetes to release cycle is uh introducing or proposing a kep.

C

So KP means a kubernetes announcements proposal, so it is a standard way so that the uh it's a standard wave in which you can uh you can properly format what your announcement is proposing to change or proposing to introduce. So uh as an announcement while proposing an enhancement we, the developer, should follow a particular guideline, so I've linked the uh the the link in the chat in the chat just a second.

C

So yeah so is it? Is it visible the slides right now because the internet is flaking here right now,.

I

Yeah yeah, okay,.

C

So uh the the uh the KP is just a bunch of uh questions and answers and a bunch of layout of the what what the KP, what the announcement is trying to propose like what is the design that is going through, uh that you want to propose, or what are the uh factors that may uh hinder the growth of this announcement uh like? uh Is it causing some exhaustion of some resources? That might not be good for the kubernetes point uh kubernetes project or things like that?

C

So it's a very curated format where you want to be the announcement proposal owner should be as detailed as possible to to have a better understand to get with to provide a better understanding to the reviewers and approvers in order to uh smooth the process of tracking and enhancement.

C

So we talked about what and keep water cap is. But what is a prr, so PR are basically is a team, so PR means a production, Readiness review team. So what the production Readiness review term does. Is it ensures that whatever announcement that the developer or the user is trying to propose, uh does not hinder the growth of the cube, does not hinder the growth or does not cause lack in the release?

C

So we which might be cause which might some of the many drawbacks, could be uh that something is causing blockage in the into the release cut or something that is degrading. The uh resources of what and what the announcement kubernetes announcement generally looks like so prr team generally looks at the points of view of from the point of view of is the announcement observable.

C

Is it scalable to maximum extent, and things like that, so in order to get an in order to track successfully track an enhancement to the announcements freeze which I will be talking about later, one announcement should always have a prr, the production, Readiness review done, and it should be approved and after it's approved, then, and then only we as an announcements team could be able to track that announcement.

C

So so what is an announcement team generally deal with? So what we do is we we track all the announcement that have obtained in by these three leads label, so uh the seek leads, usually uh opt in the announcement that they want to. uh They want to include in the current cycle by the uh by applying the current Milestone like for this. uh This Milestone it would be 1.2 at a milestone and they would have. The announcement issue would have a lead operating level on it.

C

When the when the announcement, uh when an announcement has a lead octane level, the GitHub project board that we have uh automatically uh automatically adds that issue onto our project board and the announcements team then tracks all the issues that have opted in through the announcements board and uh reach out to all the announcements that have obtained in uh and uh ensures that the announcement uh satisfy the criteria that needs to be uh accomplished by the code. Freeze.

C

And before the code fees it should, uh it should comply with the announcements freeze. So what is an enhancements? Freeze I'll talk about what, before an announcements freeze, what doesn't uh the tasks look like and what does a task for an announcement team after the announcement series?

C

Look like so before the demands before an announcement series, uh as I said before, it should have a late updating label and the current Milestone applied to the GitHub issue, and as in when we Outreach to that particular announcement, we we, as an announcements team, ensure that the KP, the the kubernetes announcements proposal, should have the up-to-date uh milestones and Status applied to it. So it might include the status to be currently implementable uh and having the latest Milestone. That is for the current release.

C

It would be 1.28 and it should have the latest stage information. Also. So if this, if the announcement is tracking Alpha, then it should have Alpha stage under the kep dot tml field and after the KP dot EML is configured and updated it, which uh the announcement stream tag. What uh is the kep following the latest redmi template that we have for the release process?

C

Right now, so the redmi file would have a bunch of statement, a bunch of fields like the test plan, section, the graduation criteria, the design plan of the kep and a production Readiness extensive, extensive questions of the production, Readiness review. So all these announcements should all these announcements should have a detailed kep and it should satisfy the current KP template.

C

And the kp.ml file is updated. We'll then go ahead and check if the prr review the production Readiness review was done or not. If it is pending, we will mark this announcement at risk and if it is done, then, and then only after all the uh before all the criterias are satisfied and if the prr is done then, and then only we will track the announcements which would be good to go for the enhancements freeze. So what is an enhancement series?

C

Announcements freeze is generally a milestone before which, before which, generally all the announcements that want to obtain to the release, should uh should be uh added by the sequence so and after uh after the announcements freeze is done. We we usually won't. uh We usually won't uh add more announcements to the GitHub project board or we won't add more announcements to our the announcement stacking process, but a way to uh a way.

C

To still add your announcement to the announcements, announcements tracking for the current release would be uh proposing an exception request which I would be talking about later, but uh once the announcement space is done. So there might be two cases that the and we have the announcements.

C

Steve have opted announcements team have outreached to all the announcement issues and they have uh as the announcements owners to fill in the information that is needed to currently uh to be in compliance with the current criteria of the enhancement fees and now, if the announcement has successfully completed all the criterias, it has all the information up to date. That is needed for the announcements freeze and it has also a prr review done.

C

Then we will track that announcement for the announcement freeze, and that means that the announce that that particular announcement is good to go further for further Milestones, like code fees or test fees or further processes. But if the announcement does not successfully fulfill those criterias, it has remaining prr or it does it didn't meet all the requirements. Then we would Mark that announcement as removed from Milestone and what that means is. We will basically remove that announcements from our announcements, team tracking process after the announcements freeze.

C

If your announcement does not made this success, it does not mean the criteria and it is opted out. Then what the announcement owner needs to do is he needs to he or she or they need to obtain there and try to obtain their Announcement by proposing an exception request. So the exception request basically means that uh it's an exception request, which, which talks about why your announcement didn't meet the code fees. Why? What would happen if your announcement didn't made didn't uh get into this release?

C

What are the drawbacks, and why should we approve your announcement request after successfully after having a detailed discussion in the slack.

J

C

Announcement exception request would be either approved or rejected for the announcements please and after the announcements freeze around around four weeks or so, there is a gap under which, after which a code freeze will happen. So what happens between announcement fees and code fees, all the keyword, all the announcements that have passed the announcement freeze will now work towards getting their code request code, pull request and test request and documentations PRS get ready and for the code freeze, Milestone. They would have to get their code related changes inside the kubernetes kubernetes repository.

C

Their pull request should be either in the merge either should uh not either it should be in the merge registered and what I mean by that is. It should have the lgtm label and approved level from the Sig leads and the uh and the approvals that are needed for that kubernetes kubernetes code pulled request, so the court fees is a milestone which tracks all the cube, all the announcements that have successfully passed, the announcements freeze uh and it checks.

C

If the code code related changes, uh pulled, requests are successfully merged or are they at least into the merge, ready, State? And if they are they they satisfy those criteria. Then it will be successfully tracked for the code fees and, if the and, if they don't satisfy the criteria, that means that if they don't have their code related changes, pull request uh must uh much or they don't have the LG TM or approval on those PR's. Then it would, uh it would be, it would be removed from that code.

C

Freeze, Milestone and we as an announcements team, will remove that particular announcement from the tracking process for further uh further attraction of other other teams in the release team. So if an announcements does if an announcement doesn't meet the code fees criteria and it is removed from the milestone, so what we would need what the announcement owner needs to do is he would need they would he she or they would need to um file an exception request again. So what does an exception request? Look like so.

B

C

Would again talk about? Why was the ex? Why was the pr not much? What are the drawbacks of uh uh what are the drawbacks?

C

If this, this announcement didn't get into the particular release and uh and some other factors, uh there's a detailed template of what an exception request, look like I'll share that in a bit and after uh after the exception request is received under the slack, the announcement stream and the release team will uh have a discussion on whether to approve or reject this uh that announcement and if it is approved, it is then further uh moved forward to the further tracking process, and if it is not, it is then completely removed from the current 1.2 at milestone.

C

So this is what uh generally uh Cube announcement stream work. Look like and yeah, that's the general layout, but it would be a quite some. It would take quite some time to get into more details of what each thing will look like so yeah, that's a just a general layout I, don't have any images on my slideshows I, just I just saw it. I should include more images.

J

Yeah, thank you to everyone. Does anyone have any questions for enhancement and and how the team works.

J

If not uh Nancy, if you want to share your screen and walk us through I'll see you guys signal works.

A

J

K

Can you guys see my screen now? Yep, okay, great so hello, everyone I, am a CIA signal. Shadow uh for this release and I am going to give you a quick intro of the CI signal team. So the CIA signal team, as the name suggest uh it's a team that continuously monitors RCI for every Comet that goes into the kubernetes uh code base and we are responsible for ensuring the quality of the release by continuously monitoring the e2e test.

K

So the as you would think, the CIA signal team plays a crucial role in suggesting if the code that is in the kubernetes master currently is ready for going into every minus minor release, as well as the final release that we do and we try to foster a culture of continuous integration test and failed Fast Fix fast mindset so that everyone is uh every one of the owning sick groups is able to take early responsibility of what could be blocking the release.

K

uh So primarily, we try to look at two dashboards, so the first one is the master. Informing I'll quickly go through it. So this is what a master informing dashboard looks like and usually we try to look at what are the failing tests and what are the flaky tests?

K

So failing tests are the one that we give the uh utmost attention to and we try to open up issues as soon as possible uh when it comes to the flaky tests on the master and forming dashboard, we usually try to observe a pattern for the feeling test.

K

So, for example, if you look at the force test over here, it has field one time, but there does not seem to be a pattern to it uh if it would, if it would fail for a few more times in a row or uh if there is, if there is any other recurring pattern to it, we try to open up the issue as soon as possible, um so this is for the master informing dashboard. Only this uh we try to review it daily.

K

uh So in our team, uh every person is given a week that they monitor these dashboards and they try to open up issues uh as soon as possible, so for master and forming. They would open up an issue if they see a pattern in any of the tests, but for there is another dashboard which is called as a master blocking, which is the most important dashboard to block or release uh as it tries to run every three hours and it it tries to uh take a guard at the communities infrastructure issues as well.

K

So if there is any test that is failing on the master blocking dashboard, that would be given uh critical attention. uh Try to open up the or open up the issues as soon as possible, and also try to engage these sick six that are responsible for that test, uh either through this meeting that we are having right now or through the slack channels.

K

So that's what we try to do and yeah. So these. These are the two dashboards that we monitor but I'll uh post a link uh to the difference between a detailed difference between these two dashboards. If anyone is interested uh posted, a link in the chat and next, uh so this is the first responsibility that we maintain uh as a CR signal team and the second responsibility is early, go and no go signals for every minor release that we go through. So, for example, we are scheduled to do a 1.28.4 alpha 4 release tomorrow.

K

So it is our responsibility to monitor uh the informing and blocking dashboards, uh try to find out all the issues that could be blocking the release and inform the release branch manager as soon as possible uh if there is any uh blocker for the release tomorrow.

K

So that's what we do, um and so I was talking about creating issues for the feeling tests. So I would like to show you an example of how we create the issues. So this is an example of a master blocking job failure and we try to give as detailed information as possible to try to include all the tests that I'm feeling with a screenshot.

K

We also provide a test grid link and since we are the first person to look at and try out the issue, we try to give a good enough reason for failure, try to look at the tests and try to look at the logs uh and just put out anything that we seem as odd and will help the person who is looking at the issue to fix it.

K

um So this is what a good issue would look like, and then we also uh tag any relevant six. So, for example, you could see we have tracked six storage over here, so that a person from six storage could look at a triage it and open up a PR to fix it. So that's why I'll say it's a critical or responsibility uh so that we can catch any of these issues as early as possible and also uh provide triage like a not if not very good uh good enough for striage of the issue.

K

And next we also, as we are going through this process. We also try to understand and fix the test, uh try to also understand if and what uh test is doing and if it is useful uh for promotion or demotion in the next release. So.

K

This is the link that I'm posting in the chat is a detailed CS signal handbook, and this is what our CI signal lead would go through uh during our onboarding process, and uh it has a very detailed information on how and what to look at and how to um how to find out what uh priority should be linked. What six should be tagged in a particular issue?

K

uh We also have a section for um section for section for finding out uh what are the frequent tests that that have been feeling through the previous release so that we know uh what we should give our attention to uh once uh once an issue is fixed. We also try to close that issue. We try to uh find out if there are any other blockers uh that come through that issue, so that we can keep a track of whatever could be blocking the release so yeah. This is a very detailed handbook.

K

It would also give you information about how to open issues what uh what pointers to put in the issues so that it is as useful as possible. We also have a decision tree uh to find out uh about the pattern that I was talking about uh how to find out. If you should open up an issue for a test that is failing so yeah. This is a retail hand that we maintain uh recently.

K

We also did a cleanup on this handbook so that we can uh keep only the information that is uh vital to this team uh yeah.

K

So, lastly, I would uh just uh say that the CIA, singular team is responsible for maintaining the sustain focus on test Health throughout the release cycle, and we try to uh our lead, tries to uh inform and uh on board the shadows as much as possible on the test infrastructure tools and uh also looking at the test, so that we can provide a good enough triage of the issues that we file and yeah be useful to the release cycle as much as possible.

K

Okay, that's it from me any questions.

K

Thank you thanks. Everyone thanks Grace back to you, yeah.

J

Thank you so much medicine for the introduction.

J

I wrote books um next week, we have the bug triage, um they will do their intro as well should be interesting.

A

Yeah, thank you, Anthony. That was super helpful for everybody. Let me re-share the screen, even though I think we're almost.

D

A

Meeting um anything else, Grace to add or no.

J

Yeah, no, that is it! Thank you. Everyone for sharing or giving us a glimpse into your sub team.

A

Yeah um I don't see any updates from six scale ability but I'm assuming there's no issues there. Everything else is on track. Are we we're not walking the project boards still? Are we yeah.

J

A

All right um does anybody else have anything that was intended to the agenda they'd like to discuss. If not, we can end the meeting.

A

D

Right, let me stop.

A