Jenkins Infrastructure Project, 21 Apr 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: INFRA Weekly meeting 2020 04 21

Description

Jenkins Infrastructure Project Meeting - 2020-04-21

Notes - http://bit.ly/2T0oZ9v

A

Yeah now it's son, so welcome for this new infrastructure meeting.

A

The last week has been quite busy, so the first topic that we thought I would like to cover is automating Jenkins releases. So thanks everybody for the help on this last week. So basically we did agree to achievements. It was a two years long project to switch from KK basement to the Jenkins infrastructure in order to release objective version so yeah it was at least it was a bit painful, but I mean I. Guess it just mean that we have to do it more often, so we had a first 3ds last Thursday.

A

We did a second one yesterday, which was which was way better so right now we are still focusing on weekly releases. We now have to start preparing the stable than the security one I think for the next release.

A

We have to better work under communication, especially regarding the GJ of the key that changed for the debian and rip trees and fun to Reddit and NCOs repository, but yeah I, don't know if you have anything that you want to bring on this topic right now, although I don't think it's important to to cover all the different issues, because we have we, we had a little bit, it's a lot, something that we did not plan, but um yeah I'm.

B

Still looking for the Red Hat public key, so just if you can send it to me separately, no need to do it now. Olivier I just wanted to be sure that.

C

You and I on is a oh.

B

You did good okay, so I can grab it from there. Then thanks Tim, oh.

A

There's the green I mean, is it not? Is it not available on the website.

A

C

We thinking about mandatory claim.

B

That key has been updated, then the the old key is gone replaced by the new key yeah.

C

You can install the old versions because the old keys gone yeah.

B

That worries me okay, so if we broke an LPS by then.

A

No yeah so that I think that's a good point. Basically, what happened here is that we changed. We change the key for the new releases and if we want to use the old keys, they are still available, but in the different repositories like they'd, be unstable, reddit, stable, whatever and I, think it's important to have a specific backup, so that I mean you raise a good point here. Yeah we.

B

Need we need it? Yeah.

A

We need to be aware, we need to be able to provide oh.

C

Yeah, so you still be allocated mark if you get up in the LTS folder, but we should copy it to another copy it into the main folder, with a different name. Yeah.

A

Maybe maybe we can use that say that the GPG key name dart in the file name, so we know which key is I mean or what it's used so Jung.

C

Has joined us who join us Oh.

A

Regarding regarding the stable into the security version, the main thing that need to be changed is obviously where we are pushing the artifact so for the stable for a stable. We just have to fetch check-ins from a specific branch. What I'm still not natural yet is? Do we just use the same Jenkins instance by creed? Is that yeah the changes that I own? Do we use a different job for stable for security for weekly, though those need to be those need to be defined at the moment?

A

Another option would also to her to define a specific branch for the different release line, so we do not provide that option by parameter. We just have to think before switching through stable at this security really this line, but for now we still rely on critical for releasing those versions. That's.

C

Gonna be painful because the key is changing. Yeah.

A

So especially for the especially for the stable, especially for stable and I'm, not sure mark, do you have an idea when will be the next table, Reese yeah Wednesday on.

B

Wednesday, so we was indifferent to to 22.2. Do tomorrow, I thought that we were all have to double check, Olivier but see I, don't understand why that would be an issue, because it's a way it's well.

D

B

Old key is still in the stable directory structure and because the old key is still in the stable directory structure, kohsuke is builds for stable will still behave exactly as they should. Yes,.

A

That's that's yeah,.

B

A

It won't be, it won't be an issue for this week. I'll just ask once once we decide to switch to the new resource for a stable release, then we will have to maybe either modify the jenkins file. I mean we have to bring some specific configuration form for the stable, okay, I think I think that we I'm not sure what would be the best way to communicate.

A

I, don't know if you have to write a blog post, but I mean we have to be better at communicating the fact that you are changing the key for the stable release, yeah so certainly.

B

We'll need to go into the when we do the stable release, change, we'll put it into the upgrade guide for that version of the stable release. We should blog it agreed wholeheartedly. We should probably tweet about it every other channel, and then we should also put on our armor for the the enflamed users who submit bug reports having read none of them. Okay,.

C

I mean the big thing will just be to have to keep it up on the package touch against our website as well. So there, a lot of users were confused, that it wasn't that under that, because the website publishing wasn't working right, yeah, yeah, toilet plus the website would be the main avenues plus cover off other ones.

C

A

Yes, I think you have anything that you have to discuss about the automating Jenkins releases. Do we well.

B

So you had mentioned LTS first and then security I was prone to go the other direction, but I guess one is not without the other right. If we do a security release, it has to be both LTS and weekly, and so that's good and.

A

I think I I mean I, think the reason why I'm first mention it just is because it seems easier to me the zone, but yeah. You have to tell you right. We need to first be able to trigger secretary DS and then switch to do a test. The reason why the requirements asked by Danielle for the secretary releases, so you want to be able to trigger release and put a trapeze on a specific event repository and that a specific moment repository change for each specific security issue.

A

So that's only the security team and the people who reported the security issue can work with us. So this is from requirements by Danielle and so yeah. We need to be able to provide a dynamic way to specify which moment refers to really want to I mean we're we're going to push on the release, and that's why yeah I mean.

B

A

Friends, just like the process is amazing, but.

C

We can't really release a weekly security fix until it's they until that's done, I assume, because we got really choked. We change the key back.

A

So otherwise, the next to be that I, just want to briefly mention, is that you should know if you I didn't, have the time to to reason a lot of time and see. I, don't think you that are you but I guess we still have the issues with Amazon instances being disconnected I, don't know if you have any ideas how to solve this now I.

E

Think when you talked about it's possibly because they are so disconnected from the master okay, that I think we had a discussion about that on IRC, it ones link I was talking about moving a master to AWS at some point. Somebody talking yes in here, you just find I, suspect you've.

B

Just got your family in the background. Oh sorry,.

E

Yeah we do. There are seven eight of us, nine of us here so exact. You just got a family, it's okay,.

B

Yes, it's just it's just the background noise out as.

D

A family behind him.

B

D

Weird because I can't hear him at all Oh.

E

Keep going Alex, so we at some point: I, don't know if it's our plan to move completely away from Azure or or what but I don't know. If that would solve a problem living the master over AWS or not.

A

That could that could that could solve it and I think it'd be either either either either we deploy your communities cluster on Amazon and then we can easily really place the other changes that I use. Oh Team Jacob already work on a short, so we could deploy CIA Oh jinkies of the other communities, so that would be.

E

I know other people are seeing similar issues. I hang out in the git er channel 4, the ec2 plugin and other people are commenting that they have the same issues. I haven't seen or seen any resolution from anyone on comms issues. So it seems like it's a fairly common problem.

E

So I don't know if the ec2 plugin people are working under not that.

B

That's consistent with what Oleg Menasha had suggested as he'd had similar unreliable experiences with the easy to plug in independent of the network network distance between the azure cloud and AWS cloud. I.

E

Am looking at the ECS stuff, I, don't know, I, don't think that could replace all of our agents just because there are some that require in my doctor and doctor, and it would be doctor and doctor, which is has its own pitfalls and so like that, but it may be at least give us the normal plug in builds a little bit more stable and things like that. So that's something I've been looking into. We.

C

Could possibly do darker and darker on a dedicated, CI cluster? Anyone anyway.

E

C

E

Some pitfalls with doctor and docker, and you can't be that with Windows very easily. You cannot, you might be able to on AWS a little better, but on a drug they have issues running doctor-doctor on Windows, so that doesn't necessarily give us. Windows builds.

C

D

Sorry about that, it's done now.

A

I mean the Sun was weird for us as well I'm, just wondering if we are not really want to have that specific issues with sc2. Maybe what would be I mean? Maybe we could try to identify how we can improve the pinion? Who would be the best person to contact for this.

E

I can ask in the getter channel and see I mean we can contact the maintainer to are listed in either the repository update their area for that plug-in or directly on github and see if we can work with them. I could do that. I.

C

Think right on is one of them. One of the Jenkins containers I.

C

Tried to check out a github but yeah hops down. Oh.

B

Thank you. It's the end of the world, as we know it did you just say: github is down mm-hmm.

A

Two weeks ago, as well, okay, so to me to me to me the best word I mean one of the best world would be to first identify. If we can please a dragon, and if it's too painful to do that, because it's.

B

A

Also sign that course, and into the community right now. We don't want to use this to begin, because it's not really a reliable, which is not been if we can.

F

C

It I guess we can, if we can collect some logs and put them on a ticket, I may see to plug in and see if we can get some advice from them or our areas.

D

Version to fix it using the ec2 plugin for dynamic agents for genius.

E

D

Uncie house I've taken some house since I mean instances since this is in kubernetes. Could we just maybe think about using the kubernetes plug-in to launch dynamic agents within the cluster, so.

A

See I've got important, so somebody DC either Jenkins that I used not in communities right now. It's running and environment and a national photo machine. Tim Tim prepare our shots to deploy either drinking. That are you on community. So this is something that we could work on.

A

If we do this, so basically, what I would like to do is to deploy that community's concern on the Amazon account, so we could have a second cluster just dedicated scheduled included. Are you just.

D

Just in case, everybody knows that the the G kids control point doesn't have to reside in inside kubernetes yeah.

C

I know I, don't know what we we want a different cluster for it anyway, because this is CI. Ok, because public KX currently is how much the seeker information on there and we wouldn't really want the CI cluster. Okay, let's go near I assume.

C

And what we've got a cluster we may as well just edging fire on it, so it can be maintained properly. Currently, it's just manually, updated by mark, and then you mostly.

A

Okay, I think we cover all the topic regarding the radio on Amazon. The next one was is regarding our care objective, or so several months ago we had to move that specific machine, so that machine was running on the right space and since the Rackspace is sponsoring us again, I don't think this. This is a big priority, so what I'm just well I will just do here.

A

I will just leave that machine there and I will just deploy an addition on behalf on communities, because so we just deploy archives that I sure don't think inside I. Oh, that's it and we just closed that tickets.

A

Another work that we started working on was having agent for as throw ended at 390 and BBC 64 I think the main blocker is still too to configure this engine and to adduce to share the junky that are your rights, correct.

B

Yes, correct there they're running just fine for me in my test test environment, we just have to do the work to get them configured in the automation, so that they'll be maintained correctly in the CIO Jenkins that IO as agents now I understood from from Alex that we also have arm 64 capability now on CI Jenkins that IO Alex is did I understand correctly. All.

E

Right yeah, they either are ec2 agents, so they would have the same issues that we're having with another easy to agents. But yes, we do have arm 64 agents.

B

Though great so, if I want to, if I want to broaden the get client plug-in testing and the get plug-in testing platform wise, that's a that's a good excuse for me to target Thanks thanks very much, and it's.

E

I think it's labeled with on 64. That's the label and I think that's! That's only label on there right now, right.

A

Next topic is regarding fastly, so the now we now have a fastly accounts. It's continue. Working for plugins at Jenkins. Are you. We made the configuration for Jenkins, so first of all, plugins at Jenkins that I own it was really really simple to put in place. We also enable ipv6, it was quite simple. The only thing we had to do was just to specify a specific cname that support ipv4 and ipv6. So yeah. That's the first website that we support on ipv6 regarding Jenkins that I own. We already made the configuration, so it's almost ready.

A

The only thing is. If we also want to have ipv6 for Jenkins radio, we have, we cannot use up XM, and so we have to switch to www.jamesrollins.com.

A

Configuration directly on the hand, shots I'm, not sure if they are also redirect directly in the website like inside taking suppressor. Jenkins is something that I have to verify so yeah unless, unless you have good reason to not switch to the body to beat anybody to the subdomain did something that I would like to do in the coming days.

A

For now, we will just use the firstly icon for those two websites, so I would like to identify how much we are raising in the budgets that we can use upper months before before I mean before putting more more mobile website. There.

F

What's the budget that we've got named, we have.

F

B

The serious temptation to put a piece of mirrors or of updates is not viable and $1000. I.

A

Mean I did something that I would like to do. Maybe just a piece I would just like to see how much we are spending in one month. Some plugins I think even changes that I own another. Another option would also be to play the extra credits, but if we want to do that, we have to reduce our bid on Azure. So it's like we have.

A

We have a, but we have a budget that we can I mean we have a bit that we can spend under the CF parts and so right now we still have the target to reduce or assure bill, and so depending on how good we do, how do and how good we are there yeah. Maybe we can spend selenium fastly.

A

We are still also in the process to simplify the building of the infrastructure project, so we are started discussing waste and APIs from the CF to see how we can directly expense, send invoice to the CDF. So basically, what's happening Faraj or accounts right now, that's okay with paying the bill and then has to be reimbursed for fastly I burn my credit card to the account. So we still have like multiple accounts, where we have our own information and we would like to to have everything linked to to to the CF.

A

So it's you have work in progress, but now I just had to put that on old photo automated release. Another yes.

B

I apologize I've got to drop off, I've got to be in this next call. Thank you very much.

A

Then I would just skip with the last last point, so regarding the sponsoring page did something that need to be done. So a lot of things happen in the sponsoring area over the last few months, and so did something that I have to top it on changes at I/o. I would just probably put small icons instead of just the name. Something I would like to work on this week and I propose to stop the meeting here because yeah we are already relayed before the next meetup.

A

So, except if I mean do you have anything that you want to I'm.

E

C

We kind of like it if you've got a minute or two sometimes just effects: incrementals yeah.

A

C

Jenya once a new user yeah.

A

I saw I saw, but I was too busy with the automated release last week and I didn't want to switch the context attorney.

B

A

Yeah I keep that in my heart. Thanks for your time. Thank you to have Indies.