Ceph Ceph Code Walkthrough, 28 Sep 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Walkthrough: Ceph Release Process

Description

The Ceph release process. Presented by David Galloway

A

Thanks David for uh giving us a go away present with uh walkthrough of the release process for Seth um without further Ado I'll. Let you take it away.

B

um So I actually spent a lot of last week. um Writing uh this document um that anybody should be able to follow, um but I'll just I'll just quickly um Summarize, each each section, I guess and then I can I can I guess give a fake demo.

B

um Yeah, so there's there's just a couple things to be aware of um the signing machine uh used to be a virtual machine that lived in the OCTA lab behind um Red Hats firewall, um but um there was a a plea to move it to the sepia lab so that non-red Hatters could also participate in the release process if, if necessary, so um it's a it's a virtual machine. It's highly available. It uh lives on the Rev cluster that has been in place, for you know four or five plus years. Something like that.

B

So um the gpg key was moved to um a like a hardware uh secured token a couple years ago. um So in order to use the gpg key, you just need to um know the pins. Those pins live in a text file on magna001 in the octolab.

B

um The pin should only be needed if the VM is restarted. However, so there there's a gpg agent there and I think the timeout is set to like a year or maybe four years or something whatever whatever the max is um so um yeah. So the the key itself should only need to be unlocked if the if the signing VM is rebooted for whatever reason.

B

The purpose of the signing, VM, obviously um or maybe not so obviously, is um during the release process, all of the packages that get built get pushed to a chakra node and then the packages are pulled to the signing machine signed by the release key and then pushed to download sf.com um yeah.

B

So that's that um there's a another note here about new major releases, so this would be like an alphabetical release, so this will need to be done for Reef, um but there's like a separate package that doesn't get built as part of a regular stuff, build and I think what this package does is installs a repo file, I think I. Think I think that's what the purpose of this stuff release package is.

B

But um basically, once there is a reef branch, chakra will need to be made aware that Reef exists, um sort of like this and then shocker will just need to be redeployed, and then this job will need to be run. um You can see yeah, it gets run like roughly once a year, which makes sense so.

B

um Yeah, so those are the two prerequisites before a ceph build can happen. um I, don't I, don't fully understand what um what like Yuri's process is, um but the the current release process now is. You know, once once, Yuri has tested everything and found a stopping point um that we're you know we're we're ready to cut a new release. uh He he pushes that stopping point to um the corresponding Dash release, Branch um and then notifies me that uh the branch is ready to be built.

B

um There are a couple deviations from that, uh for example, if we're doing a hotfix release, what we would want to do instead is is check out the most recent tag. um So, since 1723 is the last Quincy release, we would check that out. Cherry pick any um hotfix commits on top of it and then push that Branch to Quincy Dash release um and it's sort of the same thing for a security release. The only difference is, it would get pushed to the Seth private repo instead of cef.get.

B

I, don't think um yeah I'll I'll leave that, as is um yeah, so once once, he already notifies the build lead that it's time um that the branch is ready.

B

um One would just go to this Jenkins job, um we'll use Quincy as an example, um so you would put Quincy there tag should be checked and then the next version is going to be 17 to 4.. So that would go there and then the only other thing that needs to be changed is the list of distros. Here um you, you could just hit build at this point and the ones that we don't normally build packages for would fail, um but it it's cleaner in, in my opinion, um to specify it so um yeah.

B

You can see here for Quincy we build for Ubuntu focal, which is 20, yep, 20.04, Centos, 8, Centos, 9 and then Debian Bullseye, so just copy that in there and then hit build um there isn't currently currently like a notification or anything when the build finishes, um I just I, just kind of keep an eye on it, but um once it's done it will it will look like this.

B

um Nothing will be blinking anymore. It should be green. um All that good stuff, if it's not done yet it will look something like this, where you've got the blue progress bar and the spinny Circle.

B

um Yeah, so those builds will take anywhere from two to three hours, depending on how busy the other Jenkins Builders are um yeah. So what I? What I normally do while those packages are building, is do the release notes.

B

um There are three different uh types of release: notes, I, guess, there's the ones that go um in sef.get I've got the most recent version of octopus here, as as an example, um so there's ones that go and seth.get these uh eventually end up um on docs.sef.com and then there's the one for the website.

B

um These end up as a as a blog post. So if you go to stuff.io news blog, this is filing filing. This pull request here resulted in this blog post here so and then the third one is just writing. Like the the email announcement, um the instructions to write, the release notes have historically always been on on this wiki page on the tracker um it it could move. I guess but um I, just I sort of clarified it here, um but yeah. All the commands are step by step.

B

Exactly um what's a run, um I guess I can show what the output would look like just because it's not there.

B

uh It's important to always check out main because the release notes script itself um only gets updated in in Main and doesn't get backboarded because it doesn't really need to, but um we'll activate a virtual lens and install dependencies.

B

And then you would run this command here.

B

I'm just going to modify it a bit, um so the most recent version.

B

Of Quincy was 1723, and then we want to know the difference between the tip of Quincy or one C Dash. It would be Quincy Dash release, but um just so you can see what the output looks like. So what this does is goes through using your GitHub API token and checks all of the pull requests that were merged between um 17-3 and the tip of Quincy, and then it will spit out a nice markdown version of of the changelog, and once that's done, you.

C

B

Read a pull request.

B

The output will get um put here like this, so this is eventually that command will return and and look like this typically, the component leads will add their own commits to this release. Notes, pull request with um any notable changes that are necessary um and then uh finally, the the email um so yeah I normally just start a draft.

B

From 17 2 4., let's see released and then.

B

um A couple things need to be replaced in here. Obviously, the version.

B

um Name here and then I've got this little note here. If there's like a major change that should be mentioned there, it would it would go there and then you can just see there's you know the other couple places that need to be fixed like here.

B

um The notable changes from this pull request should go here and then the version here needs to get changed and then the Sha one needs to get pulled um and then and then put there so.

B

I normally just leave this as a draft, and then you know you know, wait wait for the release to finish so um so that's that once the ceph build it's F itself is done. um You will end up with.

B

Something that looks like this so I know this is the sha-1.

B

For that release and now I can use the release, notes, script, output.

B

So the next step is going to be signing the build.

B

The first step is to pull it um and that's going to pull it from chakra.sef.com and the output will look like regular rsync output. This is going to be a little different since it's already pulled, but.

B

Yeah so that'll that'll take if I had to guess, I think it's about a half hour um and then once that is finished,.

B

Time to sign it, um I think Alfredo wrote this tool called Murphy I. Think yeah.

A

B

um Sorry, my dog is not happy um yeah, so the the next step is going to be signing the actual Debian packages. um This will use the gpg key. That's on that Nitro key. This is what the output looks like not super interesting and then the next step is signing the RPMs.

B

Theoretically, this could all be automated I'm, just not super comfy, with the idea of having a Jenkins agent running on this virtual machine.

B

um So that's that's why I haven't, but once that is all done, that that also takes quite a bit of time signing, um but once it's done, you just run the sync push and I'm, not I'm, not going to run sync push right now, just because um this octopus repos in a a weird State since I, haven't finished it, um but so that will end up pushing the packages to so. Here's an example of the latest Quincy the Debian packages get pushed here.

B

The um RPM packages get pushed uh here and then at the end of that sync push command or the the script. Sorry, this Sim link here gets updated, so people that have a repo file that still use packages um don't need to like change the path or anything. So this this is just a Sim link that points back to the most recent version.

B

um Once that's done the it's time to build the containers, um we have x86 containers and arm 64 containers. This job doesn't take any parameters you just hit build now.

B

What the job does is checks download.sef.com, if there's a new version, and if there is it will build a container based on it um and then those containers end up.

C

B

Key to IO, so you can see here: 15 217 was the most recent Upstream release, so that is the the last um container that got pushed.

B

um And then yeah, the last step is just announcing. um I normally make sure that the cef.io pull request is merged before sending the email just because the email links to um stuff.io um here.

B

um And then yeah the the release notes um PR can be merged sort of whenever. um But hopefully you know around around the same time, but um usually some review goes on even after the build is done in my experience but um yeah. That's that's the process.

B

A

Yeah David, so if something is goes wrong, what what types of things do you you see happen and generally, what what's involved with fixing them like Jenkins issues or anything like that.

B

um I I don't frequently see things go wrong um if I go if I go. Look at these I believe these failed.

B

Yeah, so this is like Centos 9, UM and this failed, because I I tried to build Centos 9 packages, but um it was at a point. The Quincy Branch was at a point where I tried to build 1723 for Quincy, but there were things merged into Quincy after 1723 that are are necessary to build Centos 9 packages. So the next version of Quincy should have a central S9 release, um but yeah and typically I, don't I, don't see things go wrong. um It's just it's.

B

A super lengthy, like many manual steps processed like you can see. Most of these are: are green um I'm just trying to jog my memory and see oh right yeah. So when I, when I change the process to use a different um use like the dash release, Branch, all of these failed builds were from from then so.

B

That's why there's a bunch of red and black down there, but um um yeah I'm, trying to think of like what could go wrong, but there there really isn't a lot.

B

D

Usually it's just something got missed.

D

Sorry. What would you do if, like one of the districts failed like say, Debian didn't build for some reason.

B

um Yeah I would go check and see what what it looked like. um So, if I go here, you get this pretty Matrix of of everything um and let's say the bionic build um I would just go check the console output and see what went wrong.

B

That sort of thing, I, guess: I, don't I, don't know if that's considered tribal knowledge or just experience or what, but um you know, based on whatever happened in the build, would determine who I would reach out to like. If, if the build failed super early like somewhere around here, for example, I would know, that's probably like something infrastructure related and something that the build lead or I would would need to fix.

B

um If it was a compilation, error I would try to sort of figure out which component it was breaking during um so like. If it was rgw I would reach out to the rgw lead, for example, um but yeah you sort of just can tell after a while, where in the build um like do I. How do I say this?

B

You can sort of tell eventually like what's a compilation, error and what's not um and typically, if it's not a compilation, error, it's it's um my role's responsibility to fix it.

D

And to recover from that, can you make me trigger just to build on that single distro, or do you need to kind of restart the process again? Yeah.

B

Yeah good question: yes um yeah, so you would go back to the Seth job.

B

And then you would just uncheck tag is the difference.

B

If you keep tag checked, that's going to change the shell one of the version commit um so as as long as nothing has been pushed to download.saf.com.

B

um uh It doesn't really matter um the the version commit, gets removed and re-added, um but yeah. If you just wanted to do one, you would just keep tag unchecked and then replace um the distros with whichever one broke and then, if you only need one architecture, you can get rid of arm2. For example,.

D

Okay, thanks that makes sense.

B

To be honest, I'm not sure these three do anything these three or this one do anything they're sort of relics from um when Alfredo wrote the job uh they're not harming anything and and being in there, though, and their defaults are off anyways. So.

A

What's involved with uh adding another picture uh David is it that's I doubt it's as simple as just adding another like myths or something right you just that would I think I mean we would have to change the cmake file. I assume, but.

B

Adding another architecture: um yeah yeah, I I, don't uh I, don't know. To be honest, um so you know when, when we, when we cut a new alphabetical stuff release, what I typically do is just go into this F Dash build repo, which is where all of the Jenkins jobs um file definitions are and just grep for Quincy and then check each and every one of those files and then.

C

B

Know then Reef will get added above or below or next to wherever, wherever it needs to. um So that's probably what I would do if we were adding a new architecture is um rep for arm or aarch 64. in the ceph build and cefgit stuff.get repos.

B

Yeah I wasn't I, wasn't involved very much in adding arm so I. Don't I, don't recall.

A

Yeah, that's all right. I wouldn't be surprised to eventually we'll have to do that, but um at least right now, I'm not aware of any plans to add architectures yeah.

B

Yeah I, don't I, don't think it would be very fun, but.

D

uh One thing that's come up lately is the idea of maybe doing a like the relevant release. Again like we used to a few years back. I was tagging things as like 17 or like 18 or 19 that one that's something. uh Would that be the same process, just changing the tag ID.

B

um Yeah and then I I would just leave tag unchecked um and then yeah. The version can be whatever you want.

B

I I think as long as uh tag is left unchecked I'm just trying to look at the what the Jenkins job does again. We used to have to like manually, create the tags and and push them, um but that's all that's all automated now, with the uh um releasing from a dash release branch.

C

B

B

I, don't think that even gets called tag is right, yeah, so um so the the on on the back end of this, um what happens when this job gets started? Is uh there is a uh CEF Dash release if that releases repo and it's basically a mirror of the ceph repo? You can see here it's private, but what happens is the version commit gets created and when I say version commit.

B

This is what I'm talking about this is all this is all that that happens really in the version commit, as this gets bumped and then DCH automatically adds adds this, but so what happens is at the very beginning of the job the version commit, gets created and then pushed to the self releases repo and then, as long as the build is successful, that this version commit will get opened as a pull request against the Quincy Branch automatically.

B

But um you can see here as long as tag is not checked, then that that part doesn't happen, so the version commit would get pushed to the self-releases branch. So if you wanted to do like a an RC release, for example, um the version commit would get pushed here, um but the next time you ran it, it would overwrite the tag. If you wanted to reuse the same number, if you didn't, then it would just it would just be another.

B

Another version commit.

A

All right, thank you, David any other questions, foreign.

E

Yes, I have a quick question, please um about new dependencies that we need to include, but they're not available in any distribution. Yet so usually uh we'll check if they're available in apple and if not we'll, have them build and available in a copper repo and then it will be available in the containerize the deployments.

E

But is there any way to make them available in in the non-containerized deployments?.

C

B

I I feel like this is like an age-old conversation that has has continuously gone in circles. um I I, remember specifically Alfredo saying something along the lines of we don't want to get into the business of building and signing uh packages that are dependencies that, like we didn't write.

B

I think so what we've done historically is like when Sentosa or relate whichever one was first um came out. You know we obviously hit a bunch of like Python 3 dependencies that that didn't exist and um Ken dryer and Justin um I'm gonna, I'm, gonna, murder, your last name, I'm, sorry karatsis, um build the packages and then pushed them to the Fedora build system, and then we would just sort of go around and thumbs up those packages which automatically I think puts them into Apple.

B

So that is what we would do for um an RPM build I, don't recall a scenario where we there was a new dependency that we needed for an Ubuntu based build and it wasn't available. I. Think it's more common with with RPMs.

B

Does that answer your question I realize I just rambled a lot.

B

So I think I think I think if you're missing independency your your best bet is to try to get it into Apple.

B

And I: don't I, don't have the documentation handy on how to do that, but um I know we've done it.

E

And if we cannot get it to Apple, we basically need to have it uh in a copyright repository until it is available, either in apple or in main distribution.

B

Yeah, so are you, are you talking about runtime or build time dependencies.

E

E

S I think, is in the future. We need them sure.

B

Yeah, so with with build time it's easier, um so we have historically always kept I'm going to share my screen again. um We have historically always kept uh what's like a lab extras.

B

Repo- um and this is something that I have maintained.

B

Embarrass me like this here we go um I think I have something weird in DNS mask that doesn't that doesn't like this host name um but yeah. So historically, we've we've had like a lab extras repo.

B

um This is used for dependencies that are not an apple or any of like the regular repos, um but these are only build time um and then like uh like QA Suite packages, um so these all were built by kifu and then I pushed them here.

B

The process to do that is um documented here.

B

B

We go yeah so once once the packages get built, um it's basically just a matter of rsyncing them to the appropriate directory and then.

A

um I'm gonna, stop. You. Are you working on a different screen because we don't see what you're doing? Oh.

E

I I can see that I can see our screen. I can see it too.

C

A

I mean right now, I'm seeing a Jenkins page. It sounds like you're working on something else.

B

uh Yeah I've I've got the I've got the wiki pulled up um I'll just I'll I'll just share the link instead um yeah, so there's there's a lab extras repo.

B

um So these these were manually built and then um pushed here. There may be newer versions of these in apple, and if that is the case, then those get installed instead and these these aren't used, but uh though I'm sure I'm sure you guys saw um these uh installing ceph test on the test. Nodes has been failing the past couple days, and that was because they removed the XML Starlet package um from the base. Os repo super not helpfully um it is going to be in Rel, 8.7 and I I.

B

Don't remember what sent to West equivalent but they're in the process of getting it re-added back to a different repo, but anyway, in the meantime, I I, just um they they did build XML, Starlet, um I, think today. So what I did is downloaded it and pushed it to lab extras so that we can continue testing um just as another example of a type of package that would go here um but yeah as far as like run time dependencies, if they're not available in another repo um like we, we can't.

B

We can't encourage people to enable copper repos. You know, like I, think I think that goes too far.

B

My in my opinion,.

E

Yeah, so that will make it available only for containerized deployments or we can use it when we build.

D

E

A

Okay, thank you. David for the walkthrough it'll, be we're all sad to see you go, but good luck in your new Ventures. Thank.

B

You I appreciate that.

A

Thanks for your time, everybody have a good.

B

One thanks bye. All thanks.

E

B

E