Kubernetes SIG Release, 4 Feb 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes v1.18.0-alpha.3 live release 20200204

Description

Kubernetes v1.18.0-alpha.3 live release Feb 4, 2020

A

Hello, everyone: this is February 4th. It's a special edition of the release engineering sub-project meetings. This will be a live release, cut of the 118 zero alpha 3 release led by Sasha and fellows so without further ado. I will, let you all take it away.

B

Yay our job today is to cut kubernetes 1.18 alpha 3, and if we have a look into the branch management handbook, then it's pretty well described what we have to do today. So we can go to our alpha releases and usually we would have to do to use TCP manager and yeah find the ID stage master a built version, but we can also use nice coho command to actually get the latest built version available for our master branch, and then we would have to create a github issue.

B

This would look like this, so we can go to the seek release. Then we go to issues. This is the one for fr3, but we would like to create a new issue and then we go to cut a release, and now we have an issue template which we can use. So usually this template would look like this, so we have to create a screenshot of unhealthy release branch via test creature. Then we have to notify the release management channel and yeah. Then we have to actually do our only step so usually for our releases.

B

There are only 4 steps if to do mock stage, mock release stage and release. So these two I it depends. It depends on on each other and the stage and the release. Jobs are actually two yeah the chops which actually do anything and push the artifacts to the remote resources.

B

So if we look at preview released- and we can see that which has been done in the end of January- which did call us so what the table looks like this, if it's already filled out, so we have to run GCB manage or stage from master, because we don't have a release branch yet, then we specify our billet version and then we link the two cloudbuilt link here.

B

We also have to fill in start date, iteration and if it succeeds not, hopefully it will succeed today and it's pretty interesting because we changed the release, notes generation right now and if I look- and we had some failures today, but if I look into our currently running range, which should contain the fix I can link it to you in the chat.

A

So just note that the the Google Cloud GCB runs, the viewing of those vlogs are restricted to the least managers. Okay, people who are not on the release manager team will not be able to see that okay.

B

But right now it looks, looks pretty good, so usually the golang based version of the change folk generation script and actually does generate it. The change log for the latest alpha release and commits it into the branch and then I know go, will take care of pushing that changes to the remote resources as well. So if you now look at fr3, then we can see that there's nothing has has been done yet, but what we can do now is just.

B

Start a it's here for the mock stage. Then we can also find out yeah. We can also do it again, but usually nothing changed, and this would be our build our bit so.

A

So what might be helpful? What I usually do is grab copy the template into like a markdown file yeah and just have that on the side, while I'm, while I'm releasing I, mean.

B

We could we could do it like some before the table.

C

Where I used the headed in a github as well I, never caught them out there yeah.

A

Yeah the thing for me like if, if something goes wrong with github, if the internet goes out or whatever, while you're typing in all of this stuff, it's kind of a shame to lose all of this. Oh yeah, then soon as.

B

I, usually would know change it like this.

A

And grab the start date grab the start time directly from GCV sat.

D

A

Yeah after you do the run paste that it's that, in from the logs.

C

B

I mean, if you want to do it now, usually we also have to create the announcement, which is this notifying announcement into the release management channel I can just put it in life life so on like this and.

B

Yeah I am NOT on Fedora or Debian Ubuntu based Linux here so I have to create a container image or docker image to actually run TCP manager, which has some dependency requirements which need Ubuntu or Fedora.

B

So we created and container image for that, it's not in master yet, but for example, you can build it like this and the release repository that we wrote just specify local image, build yeah I did this sketch cached and then you can run local image run and then you should be in the container image which gives us the possibility to, for example, lists Howard g-cloud projects.

B

So this gives me pretty good tooling, and now we could usually run that command.

B

Okay, yeah I have to check out the master branch. It's okay.

B

B

This is our stop day. It's no.

C

B

This is our start date right, okay, so we put started in you know as.

E

B

As link and the link is.

B

Something like this and if no just copy the table back into the github issue,.

B

Okay, so this is how it looks like right now and usually this takes around 45 minutes. So we won't have the chance to finish it right in this meeting, but we can take that and also link the.

B

Message here and.

B

Then we can also create the table for test, create Eskridge, shot. The screenshot, unhealthy release, branch, thingy and I. Think this also runs on my so test. Grid shot is a little script, written by mostly by honest I. Think so does it have.

B

It has no I, don't know, and here's a little help that I think we don't have to specify anything in particular to run it on the latest alpha right, so we don't have to consider any I think.

A

I think just specified master ascribe master.

C

A

Like this and informing.

C

B

This used gsutil tool to upload the screenshots to the bucket right, so this would work on every machine which has the credential setup. We are G cloud or just G cloud, toolings yeah, so we haven't a 500 here. It's interesting. What 500 means that so did something wrong right.

B

Okay and then all this comments can be maybe.

C

You can try one page before sorry.

C

Can you try to access this link before we post.

B

C

Choose anyone yeah sure.

B

A

Mean it yeah it'll show you anything. It have preview anyway, when you do this.

B

When we do it, as as a new comment like this and then like that,.

B

Yeah so master pro King and master informing, and then we see the failing tests and the screenshots and everything should be available. Yeah looks good.

B

Yeah, so the first question would be I think for other releases. We don't have to do anything than just tracking that once all right, but master blocking, seems to be. When do when do it? Do we have take action on that.

A

We should always be taking action on that. If the time to be concerned is if a master blocking test pops up on this test grade shop, informing is kind of like okay, we should look into this eventually blocking is like. We should probably stop what we're doing and see if there's already an active issue looking for it.

B

A

So we're do maybe, if you'd like to share your screen and show the logs from the not not the one that you just ran, but the previous one where you can get French yeah yeah there you go so.

B

Just want to make it a little bit larger and then we go up I, think, okay. This should be large enough.

B

So yeah we have multiple steps and a process sort of first thing: where does it really start? This is just fetching go modules and we have, and so on the dependencies is more or less installing the dependencies of compiler released to us. Yeah.

A

Exactly can you scroll up to that fatal? Sorry, there's a two fatal, so I saw this.

D

A

Yeah hold on yep scroll up a little bit: okay,.

A

Was it down it.

F

A

Was it dad sorry yeah after.

F

The table so oh look.

B

This one's David, yeah.

A

So that's that's. A failure on a tag, I believe.

A

This one that's so we recently, as in this morning, were working on refactoring. Some of the compiled release, notes tool, so I'm, just I was just curious about that. I think.

D

That Island I have.

B

Good news, because I saw already saw its father.

A

Relief, that's coming up. Yeah.

B

If we have beautiful, multiple workflow steps, which are somehow described an N ago and yeah SQL now so the cumin is the release. Engineering toolkit is now available in path for inner core, which is pretty cool, which yeah basically means that we can my create two Krell or really use it in n ago, and we have something like a Krell version command. So it's checking the requirements and then we also can see which commit for crabby choose and the kid version is still empty. Yes,.

A

So that's really it so I think those two are related because that fatal, that fatal is a as a tag, error message for you, so I think it's failing to get the tag and then that's kind of bubbling up into the command. But it's not a big deal right now.

A

B

Be related to ticket version running in that container, something.

A

B

Okay, then we have the usual steps running, so this is not new, it's also not new and and we the release, notes generation somewhere, but this is cross compilation of the of the batteries inside the kubernetes repository.

A

So yeah this is, if you're familiar with the repo with kubernetes kubernetes for the people on the call. This is running a make, release, write or make release in a container, and that's the same stuff that you would see if you're doing that locally and.

B

What we, what we get here, are these tall balls, which are the release toss which are afterwards uploaded to the gzp bucket and then linked in the release notes. So.

B

Yeah other generate release, notes a little bit more down here.

B

So, interestingly, it seems to work and yeah. We are actually running the release, notes tool which for me or the base of the release, notes to which are the release, notes. Squat is using right now manually and we run it automatically with some wrapping code arounds, and we can see that we have three hundred and four commits to process, and we will also see that there are some okay. This is rate limiting.

B

This is no big deal right now and we see also that we can pass the commit their release, notes from some commits, and we also have to pee ask you I already checked those Piast. It's just fine they're, mostly those PRS or have removed release, notes blocks. So it's no big problem at all and then it's generating the change log file.

B

Writing as an HTML HTML will be used and the announcement of the release, and then we add the change to the repository and then there is a commit to the master branch and that repository and that's it.

A

Is there a I see that there are different levels, I'm thinking that maybe we would want to I mean? Do we want it to be this verbose running in and out that I? Don't.

B

A

We probably only want to surface the errors.

B

I'm not sure where it's coming from.

D

So the the log levels I mean for the dog yeah yeah. Probably we can change it as well and.

A

Yeah push those push that level down to debug, maybe yeah.

B

Yeah and it's attended that work pretty cool, so we can expect working build for the master branch as well. Okay sounds good.

B

And I think this is all we control right now. I think we can do in Q&A session, especially for people who never saw something like this or never saw this. So it's pretty straightforward to offer release cut.

B

Do we have any questions so far.

G

Okay, I got one: is there any, you know real I guess is there way this can fail in a really terrific lis terribly way or is there? A lot of you know fail safes in this process where, if it does error errors, gracefully I.

B

Mean you could basically show the stuff we did this morning, yeah.

A

Sure yeah yeah no backing for those little details. Oh.

B

Just one second I have to search for. We have some discipline.

A

Well, if you go back into the GTV logs, yeah.

B

Yeah, so this is a failure of this morning where what was it sure.

B

Yeah, sharing Greece notes failed, so the thing is, we have no I mean yeah, we can. We can test it for future Puranas that mock stage right, but if someone else contributes to one of those to us and we don't recognize it or don't test it accordingly, then is high likely that it will break something and then we will see them generating the release. So.

A

I mentioned during the release engineering call on Monday that I've I've been working on a test, for it's really just two runs the mock stage with no arguments and there's still some tweaking today, but essentially that would run starting up a new test grade board real sake, release rel Inge informing.

A

So that would run on that board, at least for now, and that test is going to be CIA release a nah go stage. Do you want to if you want to try to pull that up? If you got a test grade.

A

A

Just drop the link in the chat. You can see that it's pretty failure right now, but that has to do with wiring it into prowl, not so much actual failure of the stage job when that works, I'll send out an announcement about it.

A

I'm not even sure, if you'll be able to view logs, it'll I think it's stuck in pod initializing, so the logs never hit by glass. Okay,.

B

Yeah- and we also have this plan with the repository memory and also do, for example, we had that recently a feature to the change lock generation that we can recall the API so that we can also have some deeper integration tests on that side as well. So if everything works today- and we would see some release- notes four of our three sorted by a kind and not six anymore,.

B

Further questions do.

A

You want to kick off the release based on the one from your branch, because it just completed.

B

A

C

You need from the really k, slash release Braintree.

C

A

So basically, what I did was target a mock stage before the call against against Sascha's branch that had the fix in it. So this is equivalent to what he just ran on the call. So how does this.

B

B

Did comment and if you run that.

A

hmm That's a fun new one.

C

Yeah interesting.

A

C

Should cross the ocean.

D

A

I'm gonna try running it this wall, just just for fun,.

F

If you click the execution details, you can initial see the version.

F

A

Execution details yeah go back home.

F

There is a bit long, yeah.

A

Yeah so 113 6-1 edge, which is the right version and go into the failed one looks like it never submitted. It's just.

B

Yeah all right, it.

A

Submitted but it never started yeah so close.

B

Okay, so there's no, you crush crossword right, though.

A

So I did make one change to this, which was to add the compiled release, notes step in there and that shouldn't have broken anything on this life. But you never know with an I'll go.

A

Okay, I got the same failure.

A

So maybe pop up in the release, GCB release cloud build em. Oh.

B

It's dead, one.

A

You can see if there's something I missed here.

D

That'll, do it.

A

So, if you add, do you want to do do we want to live patch? This right.

E

A

Go go to the stage about um oh I.

A

Think really, all you need to do is add where you see towards the bottom.

A

Should have been listed there, maybe it's.

B

This one, maybe not.

A

Search for cross.

B

You should I edit, you.

A

Well, it should already be in the stage one so I think what's happening. Is that there's some pieces a or some piece of GCB manager? That's not interpreting, there's some piece of GSP manager in the release, step that doesn't submit that substitution and only considers that substitution for the staging step.

A

So let me poke at JCP manager to hold on.

A

A

B

Do we have any other questions so far.

B

A

Yeah found it if you, if you want to pop it up on the call, you can look at GCB manager,.

A

Yep and then within their two sections within the stage it functioned.

D

A

See the key press version so that only gets interpreted in the stage at section and then, if you look at the submitted section to you,.

A

Right and you see that substitutions passed on line the line, 296.

A

A

We're gonna do here.

A

Yeah I would say just add that piece too.

A

Yeah add that piece to the release it section right under the if D.

A

Alright and now.

E

A

That should be fine and then within the submitted.

A

We're going to drop the check for a command equal stage, or we could do.

A

Yeah I would say uncomment that uncommon.

B

A

Yeah and and and drop the the test or command equal stage end.

B

Okay, so we just remove are.

A

Actually hopes one don't do that.

A

Yeah, let's, let's add, dropped to drop the double ampersand substitutions, cube cross version copy that and move that into the usual suspects section.

B

This one complete-game yeah and you.

B

Okay: okay, okay,.

A

B

A

B

C

But this was a changing. We did it like in the morning yeah.

A

We just added to the morning: okay,.

B

A

So you're going in yeah you're gonna need to check out a new branch.

B

A

And we'll clean up the commit messages and all that good stuff later.

G

All right, okay,.

A

We're doing it live everyone.

A

Can do so we can you can skip it. We can skip the PR for now. Yeah go back to go back to your terminal and pull up tab up. One.

B

A

Right so we're gonna so now from here, you're gonna do just go to the front of that line. Yeah.

C

A

Think you dropped a three at the end.

A

Yeah go to the front of all in its release, underscore all caps in sorry, the environment variable release underscore tool underscore repo equals, whatever your Romero, so HTTP github.com, slash, Sasha, alright, / release, dot, git, alright and then space release underscore tool.

A

Branch on your score branch again equals whatever you name. The branch fix AGG management.

B

A

It hit that yeah okay, so that's not up to date with your local, so just make sure that's checked out in that container.

B

E

Think there is a typo in the version you drop the seven there before the three.

B

That's I mean is that correct I think it should be.

B

Alright, this is the the repository which will be used in the remote, so it has to be HTTPS, I think so. Yeah.

B

B

Verify that this is correct,.

F

B

Do I have to set that remote as well as I, can on.

A

The container yeah.

B

Okay, yeah so I hacked around alright. So now it's running here.

B

B

Just took a second.

B

Okay, so now we have to wait 45 minutes right.

A

So this one should take somewhere between 18 and 35 mm.

C

A

C

18 minutes 15 on.

A

A good day on a good day, it's it's 18 with it.

C

Today's a good day, okay,.

B

So what we do now is we test it, and if this works, we create a P R and then we have to fix that as well right before the next stage, yep.

B

No and the meanwhile, we could also have a look into this one. Thirty minutes right now. It should be. It should finish in.

B

Yeah in some minutes right: no, no! It's just splitting too tall, so 180.

A

Yeah looks like the last one was about 49 minutes. Yeah got a little bit to go.

B

Okay, so if you don't have any questions or anything we want, we want to wreck right now, then I would say we can also close this meeting. Do we have any questions.

B

Okay, I think then we can close this meeting and we can continue with our fixing session offline. All.

A

Right thanks everyone for popping on thank.

B