Jenkins Infrastructure Project, 25 Aug 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2020 08 25 Jenkins Infra Meeting

Description

Jenkins infrastructure meeting to review

A

Thanks everyone welcome to the jenkins infrastructure meeting it's the 25th of august. Let's take a look at the.

A

Agenda so um propose a topic to talk schedules. Briefly, then, tim talk to mirrors upgrade update center improvement. I'll give a brief status report on jira upgrade.

A

We have a sort of a hot topic on docker terms of service that needs further discussion status report on oracle cloud release status reports, and I think that was it any other topics you want to add.

B

um If there's time, perhaps the docker images and the use of version specific update sites.

A

Good daniel, I miss that you're here, okay, docker docker, so where how about we put that before oracle cloud and after docker terms of services, so docker image, use of version, specific update sites.

B

Yes, um if you access updates of jenkins.io, you get the plugins that are compatible with your versions of jenkins and the mechanisms the docker images currently are using is one I would like to retire um and it basically just needs a pull request merged. As far as I can tell.

A

Okay, got it all right, so topic is added, we'll let you let you talk to it. That's great! Thank you. Any other topics. We need to add to the agenda.

A

Okay, then, let's talk mirrors, upgrade tim and daniel.

C

Yep um so basically, we swapped um the mirroring service from mirrorbrain to mirrorbits.

C

um So then, so the main reason that olivia hadn't done this previously was he wasn't able to keep the plug-in releases in sync, because if you try and sync the whole folder, it takes about 20 to 30 minutes, um but daniel built some functionality into an update center, um which emits only plugins that have been up released in the last three hours. um So we're able to do a delta um sync, which is normally between 1 and 10 plugins, and it takes something like 10 seconds rather than 20 or 30 minutes.

C

um And so with that we were able to change it over. um There are there's been a few hiccups and there's um so we have issues worth um there was a http https issue which daniel fixed in the htaccess files. I think um well. The biggest issue initially was that nothing was getting mirrored. um Everything was going directly through to storage. So you see some.

C

The bandwidth just went through the roof on friday, um and we eventually tracked that down to there was a reject on what should get mirrored and hpi wasn't in there, and that was in the chart and fixed that eventually um and after we did that, then we noticed that mirroring was slow for the initial release and the archives download site as performance is just so bad, so that archives was our fullback which had everything mirrored on it.

C

But it's just performance on it's absolutely rubbish.

C

So we've um and it was kind of okay, because it was only used for old releases. So basically, archives is used as a fullback when something's not married, and um I think we mirror something like the last year of releases.

C

um So there are some some older gene conversions aren't mirrored, and I guess the conclusion was people who aren't updating often can deal with slow speeds. um The problem became, though, that um when plugins are initially released, mirror mirror bits takes a while to kick in, and so they were getting really poor speeds.

C

So um I've ended up, at least for the short term, putting in a um changing the fullback service to be azure, which has impacts on our bandwidth, um but it's only for well for plugins, it's only for the first just over an hour. It takes it, takes between an hour to hour and a half for mirabets to notice that it's on the mirrors, um I'm just gonna try and look at improving that if I can, um but it also does mean that we are serving traffic for the old gene conversions as well.

C

I think that's where most of the bandwidth is going at the moment.

C

um It's it's not huge, so I think we just need to monitor it. So we've made a couple of fixes.

C

We noticed that latest plugin, so there's a url, something like slash, download, slash, plugins, slash, plugin latest, which was getting redirected to to io to actually and the sim link update's, not getting updated um and so daniel fixed that in the update sender, to redirect to a specific version instead and it was- and then we noticed the next day that was the same issue for wars um and so daniel fixed that as well thanks for helping up that center here daniel um and so that should stop all of that traffic.

C

um So we're keeping an eye on some. I think yesterday, afternoonish um was the last fix yesterday evening. um European time I think was the last fix that went in um and I I was having a look at the logs this morning and there were only two plugins showing up in the logs and they were both plugins who were released within the last hour and the rest of it was all jenkins war files.

A

Now are the jenkins war downloads ultimately served from mirrors as well or from.

D

The chickens wars older than a year- oh, oh, okay, older than so outdated jenkins.

A

Wars I see thank you.

A

C

Yeah yeah, not necessarily older than a year, it's whatever. I think it's a year, but I'm not sure um the other thing was so just look, so it seems like mirrorbits is two stage um on on how it gets rolled out. So mirrorbits does a refresh of the local repository, and at that point it will hash the file. If it doesn't know about it um and then it also periodically scans the mirrors doing a full r sync of against the whole mirror, um which is when it notices whether mirrors have it or not.

C

um So basically, the issue is, it seems, it only does full scans and it doesn't appear to be a way to do a partial scan. So it probably needs a improvement in mirror bits, but then that could get our um time to mirror two seconds instead of an hour and a half.

C

And also part of this, um I think gunther I think his name is he runs. The eu mirror he's increased his sync time from an hour to 15 minutes. I think so, basically, for his mirror. um Plugins are mirrored pretty much straight away um and we also we're also pushing releases to the osu mirror as well. um So so plugins are available instantly on the osu mirror uh by oregon state university.

C

um But the issue is that mirobits just doesn't notice and doesn't start redirecting there for a while.

C

So, ideally, we wouldn't have to serve these plug-ins, but it's also, I mean looking at today's traffic. It's not it's not a huge amount of bandwidth, but if something like the aws sdk gets released, that could be a fair bit of bandwidth.

A

Right, okay, so, and the the place where you're going to gather the traffic patterns from the logs is that something you can share with us? Is it.

D

Available to somebody like me without yeah, that's.

C

D

I think I shared the.

C

Pleasure, the law of queries.

C

Yeah, um so um at eight o'clock am my time zone. um If you say, if you look in irc for you, you can check and loki um the urls there.

C

um You may even need to be added to some group. So just have a look. um If you're, not in the group, I can add you um actually that's one thing is: um did uh you mark or oleg get access to um uh key cloak? I know that was something we talked about before olivia went on, leave.

A

I think so, but it's a good question. I don't have the answer immediately. Let me put it as a reminder. I would check because.

D

I'm gonna keep.

A

Quote because we.

D

C

Otherwise, yeah yeah next week there might not be anyone around who has access to accounts right and if there's anything else, that I know I'm the backup on secrets um for olivia, um but I think we're probably okay. Well, so I'm the global bear cup there as other backups for some specific services.

C

um But I don't think that will be an issue for a week. But if there's anything else that you do need, while I'm off have a think.

A

Great, oh yeah and I'll I'll try to do some safety checks thanks. Okay,.

C

Yeah, yes, make sure you can log in and if you need it, if you need any hints on how to do anything, I can show you.

D

Great yes, thank you very much. I was.

A

Going to veto your vacation just so we were clear kim but.

A

Enjoy your time off, that's great anything else on the mirrors.

C

um So yeah there's two things, so this fixes two major issues with this that have been around for years. um It's been for a 160 and then for 361. I think um so. Emperor 160 is that update site reports plugins being available before they're, actually available and um users can't download the plugin and especially in the jenkins ui.

C

They can't even I can't even install the previous plugin, because there's no way to install a specific version, so you just get a 404 back and users that use latest or pull in a parent with a fixed version, but not the child get stuck with the dependency 404.

C

um So that fixes that issue- and I think it's 361- um maybe it may have the numbers wrong, but um 361 is about plug-in updates being done over https.

C

So now now plug-in updates are end-to-end over.

C

A

Excellent. Thank you. I cannot express the delight how how grateful that I am for that. That's wonderful, thank you to you and daniel. Thank you. Thank you.

B

Now, you're, no longer afraid of releasing git plug-in and annoying hundreds of people. For the next hour and a half.

A

Which today, I'm scheduled to really skip plug-in and annoy people for a long time, you're right, the google summer of code, git, plug-in delivery is coming and so there's a potential we're going to annoy a bunch of people with some really cool stuff. So, yes,.

C

This actually um so one other fix that we get out of this. Is we get um an updated geo ip database, the mirror brain gop data device was broken and it hadn't been updated in a long time, and it was very inaccurate, so poor users in the eu always got sent to the server ion mirror at least my geop data and many others. It's going to the server ion mirror, which was fallen over all the time, and that was what got me to fix it in the end.

C

um So gunpow runs a really good mirror um and, um and so a lot of eu traffic is now going towards his mirror um and he said, and he posts a screenshot in irc, showing his traffic climbing up a lot.

A

And and he's okay with that, I assume right he's not saying oh wow, I didn't expect to actually be a mirror at this level.

C

No, he seemed happy with that he's seeing he was seeing a lot more traffic now. So all the poor users heading server ion all the time are not it's still. It's still a mirror, but it's.

D

It's correctly now a mirror, that's that's! It's in.

A

The uae is that right, so it's it's a great year for africa or for portions of india, but it's not such a great mirror for someone in norway, or particularly in the uk, I mean you're geographically a long ways from the uae. So.

C

Yeah- and so I did some- I did um some timings just to show the improvement as well. So before we made any changes, I did a build with um my work um instance, which has 131 plugins, and it took just over two minutes to build.

C

um I did it after our changes um and we're still running against the eu mirror, and it was like 51 seconds to build, so it was over twice as fast.

A

Oh, that's great! That's wonderful!.

C

Just probably about the mirror, I was heading, but.

A

That's, that's. That's fine right! You! You saw real performance, that's great.

C

And I think that's everything unless daniel has anything to add.

E

A

Okay, next topic, then update center improvements uh stan.

B

What the lead sorry yeah- I mean they. These two are linked, so we've already talked about the updates and their improvements. uh There was the selected list of plugins that were released in the last few hours. That now makes the selective sync possible and we changed a bunch of redirect tools and urls around so now it advertises https, urls and no longer links or redirects to urls that don't really work on mirrors because they are not getting synced.

B

So this is really really tight. Together, a quick note on the transition, I saw some confusion, uh I think from jesse with a test result from friday evening there was a window of about maybe 10 20 30 minutes where it looked like downloads were failing due to check some errors.

B

That was part of the transition where we messed up and quickly fixed it. But some tests were failing on friday evening, utc with check some mismatches. So that's perhaps useful to know for anyone who's seen those.

C

Tests are also failing, because um they, because we changed from http to https, there was a testing core that tested that.

B

A

Right delighted to have failed, that was a oh. Yes, that's exactly the right kind of failure. We're we're better. Now we'll fix the test. Yeah. Okay, that's.

A

Fair great anything else to handle.

B

A

Okay, so that item on key cloak I've got it as an action item. Let me just put it I'll I'll note it later. As an action item jira upgrade plan, I apologize. I've made no progress since our last meeting. The notes are in the plan and I will be working that uh I've gotta schedule the next session with the linux foundation to review that here are the actions I've taken, we're ready for the first test and they will then plan the work.

A

They'll set up a jira instance using data from our backup and we'll be able to test that some of the crucial things there are ssl related. How do we delegate to them the permission to create an ssl name for something we use for tests?

A

Okay, daniel? Oh, no! Next, one change in docker terms of service, so docker has restated that pull limits will be real. As far as I understand it is that everyone else's comprehension as well, that if you don't have a have a pro or a team account, you will be receiving pull limits.

C

Yeah, but it's it's it's not it's.

D

It's at the user end, it's not on our end, oh so this is per. This is the the person.

A

Doing the poll- it's not for the repository.

C

Yeah yeah, so they put out a clarification like today or yesterday.

B

We are users of docker hub, I mean to build the images. We need to pull the upstream image all of our jenkins instances, pull from our own jenkins images or potentially jenkins infra organization.

B

There's a lot more going on than just us publishing images and not caring who uses them sure.

A

Yeah! Okay! So so that probably lobbies that we need to to look for funding to get a pro or a team.

C

It's account a pain, it's not about the account it's about having to put it everywhere,.

A

Right right, exactly it's it's and then we have to. We have to use. We have to dock or login everywhere right. We have to somehow inform them that, yes, we're using it.

B

Okay, how do we, how do our agents work when we use um azure acs? What's its name.

F

Aci azure container instances right. My understanding is that so azure is doing a poll. I don't know how they delegate are good, so basically the microsoft account would be doing the polling in that case is my understanding.

C

B

So um I get the impression that for our own pulls we could just not care and see how badly things break, because nothing is really relying on that. That much.

F

Well, the plugins, like the docker workflow, plug-in a lot of its tests. Do polls um some other plugins as well in their tests. They uh create docker images and and poll images, and things like that.

F

So the other question is like: if uh an image has you know, 10 steps in the docker file is that 10 polls is that counted as 10 polls when it pulls in that information or or what is? How is the count being done? Do we know.

B

I think from swan poll.

F

B

Unless you have it cached even right, so that's the thing.

E

Is this just a nightmare.

B

On the other hand, even even even this isn't catastrophic right, I mean if, if you plug in ci, just breaks, it's not like our site is going down as a result.

F

Yeah, I think I think it's going to be hard to understand all of the issues we're going to run into before we actually run into them.

C

And they're just going to keep changing it. Sorry they were going to keep screaming at them like.

A

Okay, so I propose we leave the topic on the agenda and annoy each other every week with it, so that we talk and and share what we currently know. I don't see any immediate action that we should take right now. Anyone have specific actions. I should be flat listing.

F

Here, I think one thing we may want to check is with azure and possibly aws. We don't have docker on aws right now, but it's something we've thought about with ecs, but maybe if, if they have any information, how it will affect their different technologies, whether it be aci or ecs, I don't know if we have any contacts within those organizations.

F

C

Yeah well, the recommendation is probably going to be to run a local container registry and import what you need.

B

Yeah, so about that we do have artifactory and it has docker support. I've never used it, but that may be something where we could offload some stuff too.

F

C

F

Do we have any limits there? Oh sorry, go ahead.

C

So, even if it's just for internal use, you can just sync them have have a script. Yep have a script that just has everything we care about and sync it. But it's a bit of a pain.

F

And do we have any limits? I know we're on a like a sponsored account or something like that. um Do we have any limit on the size that we can store there.

B

I don't know if we are beyond the terabyte of data so far, and we have not heard anything from jfrog that this would be a problem.

B

um It might come up in the future, but I don't think there are real official limits. At least I haven't seen anything. The problem that we have is that our instance also predates any official um open source sponsorship program they had. It was basically kk doing a deal with uh yarov and or the the founders of jfrog like eight years ago, or something.

C

Yeah they struggle to finally exist.

A

Okay, so is this one where we need to we need? Well, so is there an action item there? We should talk to artifactory talk to.

C

Jfrog, I doubt that no way exists without a lot of pushing okay, all right.

C

um But yeah that could definitely be an option, would be using artifactory, at least for internal use.

C

I don't think there's anything to do right now. I I also don't know if it needs to talk about every week either. Oh.

A

Okay, good all right, so I I'm just not yet comfortable that I see what's going to happen, but that's cool, we'll we'll leave the topic that should go next to it. When we need to okay, docker image, use of version. Specific update site is daniel.

B

Right so, um a few weeks ago, well months by now, um I announced uh plans and then implemented them to have finer grained version, tiers and update sites.

B

So previously, if you had looked at updates jenkins, I o there was one tier per lts baseline for the last five baselines or so and now um what happens is the updates, in the generator looks at the core baselines declared by all releases of all plugins and grabs those released in the last year and makes them into tears, and so I call them the dynamic tiers, because their existence depends on the actually distributed plugins.

B

So right now in the docker image, the old pattern, with the naming scheme stable dash, lts baseline, um is hard coded, and I request that this be changed so that.

B

The docker image queries the basically the root url for the update center json to get redirected to the appropriate, compatible tier and then uh continues from there rather than just assuming there will be this magic directory name and the way I propose we implement this in uh the update center is that the existing tiers, fixed tiers will still be generated, but any new ones would not.

B

um So I proposed to be started with jenkins2249.x.

B

The upcoming baseline, that's due to be scheduled to be released in two weeks, and at that time there will be not five but just four fixed tiers and over time for each future, lts baseline that exists, the oldest one will be removed until there are no fixed tiers left, um and that's that's my proposal how this can be implemented.

B

There is a pull request that I linked uh for the docker image that I think accomplishes everything we need here, um but it is not yet merged um and I think it's open for like the last three or four months.

B

So I think now, with the next lts baseline upcoming is a good time to talk about how to proceed here.

A

Okay, so any downsides, I don't see any anything we're sacrificing by by switching this technique.

B

Old containers will no longer be able to install plugins with the install plugin script.

B

On the other hand, nobody support well old containers that would be obsolete anyway, through due to the um the fixed tears being dropped right, that's the with with the way I propose this works is, um I think, 2.190 is the last the oldest supported fixed tier for the next three months, and that image will continue to work, but 2176 will no longer work, but it would not work anyway.

B

I don't think there are any downsides.

B

It might involve an additional redirect rather than guessing on the side of the docker container, but it's basically free.

D

Sounds fine to me, yeah. That seems reasonable to me.

A

And then, after the three-month period the 2.190 is working, it would fall off the edge and stop working.

B

Is that right, but that would happen anyway. So the way it used it currently works is we support the latest five lts baseline. So right now that's 235 222 204 190 176., once uh 249 is supported, we drop 176 once uh 2 63 gets released or whatever we drop support for 190 and so on.

B

So it's always the latest five and my proposal is: we change the docker containers to no longer use these and then stop creating the new, fixed tiers and retire the old ones on the existing schedule and the year from now there will be none left.

B

I don't see a downside, but I also don't see the pull request that um alex proposed merged. So uh maybe I'm missing something.

A

Okay and that's this, this 964 pull request; okay, great! That may just be that I need to be more diligent in doing coding news.

A

Any any other discussion there on docker image use alex any objection from you on the proposal.

F

No, it sounds fine to me.

A

And sounds great to me.

B

A

Okay, good anything else on specific update sites, daniel nope. That was it. Thank you all right, uh oracle cloud conversation, nothing to report, oof, sorry, oracle cloud conversation, nothing to report. I have the action item still to reach out to them, see if they're willing to provide infrastructure for us and at what level I'll continue that release status. Reports uh we'd, release 2.254 today um it went without any bump or brews 2.249.1 release candidate is coming needs testing and I've proposed an online meetup to highlight features tim.

A

The idea was to do it like september 10 or 11 and include tables to divs. As part of that, would you be willing to be part of that, or is that.

C

um I'm not really working on table steps right now. um Okay, I will probably leave it out for now. You.

A

Would okay so then that may justify that it's not not a big enough session to do an online meet up then, because, if I remember rightly, other ui improvements are not so dramatic as table to bids is.

C

Yeah, there's probably not a huge amount in this one. I think.

B

Mark, having looked at what else changed I mean it doesn't be just ui improvements right, there can be other stuff. I just don't know off the top of my head.

A

And I will I still need to do that. Windows support changes is substantial, but but that one needs more investigation before I propose a meet up with it. Yeah.

C

There's theming, possibly as.

F

I was going to say the theming stuff that should be the first lts with theming sport right yeah. Oh, oh right! Okay! So this this is the first.

A

Lts to support dark mode, for instance, and solarize, I don't recognize solarize. What's that in dark mode.

B

uh Solarized is a fairly well-known theme for a ton of different text. Editors um and I created the theme plugin for jenkins.

B

It's a bit weird because it only supports like four different colors plus highlight colors, and it's a bit difficult to make the jenkin to make the jenkins uh ui to restrict it. That much. But I think it looks fairly. Okay.

A

B

A

Might might justify being part of an online meetup then, because dark mode is certainly very very much interest interesting to people so theming, okay, good thanks all right, so I'll refine that any other topics we need to address. If not, I'm going to call us at end thanks, everybody.

D

A