Jenkins Infrastructure Project, 31 Mar 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: INFRA Weekly Meeting 2020 03 31

Description

Jenkins Infrastructure Project Meeting - 2020-03-31
Notes - http://bit.ly/2T0oZ9v

A

Let's start this meeting hi her body, so welcome to this new Jenkins infra meeting. Basically, the main topic will be to discuss about automated Jenkins releases, how we can automate this. We were blocked because we were missing a consignment certificate which has been provided by the CDF last week, so we are now looking at. What's what are the next steps? Basically, so we started a new document. That's contained all the notes related to this.

A

The document is not to be to everyone, but it's more like if you're interested to participate just add a message and you will be added to the document. The main, the main reason why I don't want to be too big for this, because I want to be able to discuss about security or private stuff, whatever I'm related to this project, but after once the once the automated releases that determine can come back public so but yeah. If you want also to participate, feel free to ask the permission so I can just share.

A

We just share these documents for now,.

A

Can you see my screen release automation discussion, so the idea here is just to list all the missing steps that we need before going live. So the first thing I just did a few reminders about the different repository that we have here. So the first one is that our packaging, which contain the docker image used to run all the scripts and the packaging scripts. That's a quite big image.

A

The second repository is Jenkins packaging, which isn't the the repertory that contain all the script to build Debian, RedHat Sue's and in Windows package we have Jenkins in for a release that contained the the bash script used to release that contained Jenkins file used to trigger job released to trigger packaging. The button, please definition that we use to run everything on on kubernetes, that's the biggest part here, then we have Jenkins in Flast, slash, chart secrets those. This is a private triple tree that contain a kripp, encrypted secrets use in the release environments.

A

All the secrets are not defined there, which try to split the secret in multiple location. Some are in that repository other are stood on the other key bolts and finally, we have the change in the free, slash, charts that contains the definition to deploy the reason, vironment and all services related to the release environments. So if you have some time, if you are interested by a specific part of these projects, feel free to just look at this one of those repository very far review.

A

Those be sure that everything is fine and if you have any suggestions, feel free to document those suggestions here. So we can discuss the idea. Is we started this thing? A few duze think that you have to do those of minor bugs that you have to fix the main. The main.

A

The main thing that you have to do now is that we were using certificates like subsides certificates, temporary GPG, key and undo and stuff like this, and now we have to switch to on the official one so updated the the certificate, the code signing certificate to use the real one, I generate an official GPT team, and so now we are running the the release process and the packaging process.

A

You know to be as close as possible as what it will look like once we are going into addiction so now, depending and yet puts on the different people will review this project.

A

We will be able to go either at the end of the week or two weeks whatever, and we still have to say that I mean we still have a lot of uncertainties before I go into prediction. Do you have any questions so far from here nope, so I can continue. So as I was saying, the ID here now is more to identify. What's the first release looks like, and so there is a file.

A

So we have right now, so there really is our define in the directory called profile. The G and inside you'll have three different releases right now, right now, I'm focusing an experimental, but once the experimental unit is validated we can move forward. The the file is the same for the two other releases, so the most important settings that we have here is the git repository that we are using to build a release or right now I'm using my fork, um but we can do some examples.

A

The GPG key that we are using the Mullen repository where we are pushing the artifacts and finally, the release line. So the release 9 is the name of the release. So right now we only have a stable and with every eases so in this case I'm just creating a release line that are called experimental. So we can officially push, for example, on Picasso junkie. So tell you to validate that everything is working fine.

A

So once once we are I'm ready to move to go into production, we just have to be sure that the user- and we have the right settings defined here and the user used for releasing have the right permission. But you are right now we are still really doing some tests. The release of environment looks like so. The service is only available from the VPN, so right now it only have it has two jobs, one to trigger release a second one to trigger a packaging.

A

So the the releasing job is now working or is this so it's not working and it's using the official signing certificate. It's also using a real GPT key, so we can test artifacts. So if you want to have a look so basically this instance, this is only available from the VPN, but then it's public to everybody who isn't with them. So right now we decided to go as open as possible.

A

If, for some reason we realize that we that the that instance can be at risk, we can be more open, you can I mean in a more secure place, but for now we wanted to. Yes, we want to be sure that people can look at the bid outputs. So if you want to trigger- and you release it's as simple as this you just this- you just run a build and there is a parameter and the parameter will ask you which release you want to run.

A

Is it experimental right now only enable experimental to be sure that we do not conflict with the stable or the weekly, but once we already, we can just just add those parameters for the stable and the weekly and regarding the packaging regarding the packaging's job right now, I'm having some issues with the GP g g, so basically I was using the default GPT key provided with the scripts and now that I defined a different API key I'm having some issues and I'm doing.

A

If you seem to be sure that we are using the right, chicky chicky, but the job looked like this right now, so it's loading. So basically, what are the difference using steps here? So, as you can see, the builds are broken right now, so this issue is related to the gbgt. So we are using the wrong gbgt.

A

The next step would be able that we will be sure that will be the next I will have to be sure that we have the mission to the right location so something that I started working while we did not have the concerning certificate was to redesign by Carol Jenkins. That is so. We could splits the different services into different location and I. Sure I did not have the time to finish that work. So basically what I'm going to do it just change the publishing script, so we just could be all the file to our package.

A

Thank you that I own and we can move forward, so I wasn't able to do all the work that I wanted to do before going live. So now we are just looking at different shortcuts and seeing what's what really need to be done now and what can be delayed for next steps.

A

Is there any question regarding the automated release for now.

B

Like we can't hear you sorry no time so we have some action items on our side to proceed. Suppose me and Mark will be spending some time to help yeah I, guess that any external contributions would be great, especially with a dating beats once a day available. Providing feedback because see all the flow is open source. So any audit and comments would be much.

A

Appreciated so yeah, as so while right now, they are really now I know in um in the moment where we try to do audit and test that everything is working, fine and so yeah. If you have any experience with communities or Java whatever.

A

That's probably one of the good time to jump in another thing, that I really want to identify and be sure that everything is aware is that I want to be that at least two piece of two person who knows and really understand the different parts of this project so either from the infrastructure parts either from the environment, the services, the process or whatever, and we also have need to be sure that several people can change passwords or understand how the credentials are working.

A

So if you have any questions concerned, whatever that's really the right time to ask, and so we will be able to address this yeah. This is this is currently my priority. I would like to finish this work as soon as possible, so this.

B

A

Focus for communities.

A

So if there is no other questions, I propose to move on the two different to other topics that I put to the agenda. So there is one regarding Rackspace, so you may so. The message on the mailing lists, so Rackspace was a sponsor of the Jenkins projects for the last 10 years, something like that and in December in November they announced that they stopped open source projects and they came back to us last week proposing a new, a.

A

New sponsoring project idea would be to to not be sponsored by Rackspace, but be sponsored by spin up, which is a different interface of the Rackspace, but we have to. We still have to see what would be the condition, but basically they would provide some computes for the second heirs. This would allow us to not work on our contingency at or asleep Sebastian it's. This is where every need is more than welcome at the moment, and we don't have to do anything else than that.

A

So this we agreed in regarding CI regarded urgently you you are still in the process too, so we are now using Amazon for each appearance for Linux machines and pull Windows machines, so we're using the easy to plug in. We are having a really weird bug at the moment.

A

For some reason, the machine stopped working after why, and so we just have to be lunch. The agents the machine is working, fine know this base issue, I mean the machine is really I mean we totally totally correct. So it's just that there are disconnected after work, so we have to you know, start to work on this to understand what's happening. In our case, it may be related to some latency issues between Amazon and usher, but yeah. That's a pinky word issue and with the automated project, it's kind of difficult.

A

So if you have an experience, I'm using Amazon and Azure together, you're also more than welcome to help, but we are facing some bandwidth issues right now on the different projects. So.

A

That's that's pretty old, so for the past week any question no.

C

So I'm, assuming I'm gonna, continue trying to investigate those ec2 agents being unreliable after I get my my initial checks done on the current prototype builds or the release. Automation, outputs, I'm gonna try to write some tests to assert that those signing setups are correct. Watch the test fail with the current build outputs.

C

So so those are those are on my plate and I'll keep watching those ec2 instances and restarting them during my hours, winter or reconnecting them they're, not even restarting it Olivier and Mike I think I understand that those ec2 instances are sometimes actually recycled that they are destroyed and then recreated as as machines that they periodically go away and come a new machine comes online. So.

A

So basically, the plug-in is configured to request. The machine is configured to request a new instance when it's needed and what's happening here. Is that so the machine is correctly provision. It's correctly attached to the master it's correctly used and after while the machine is disconnected so the machine is still there. It's still running an Amazon and it's not even not deleted.

A

Sometimes the machine are just running for hours and then reuse and some I'm not sure I'm, not sure, if I a configure time out to delete the machine after while, but last time, I checked, it was not working, but.

C

A

Not anymore fresh in my head, so no.

C

Problem, thank you. I can I can read the configuration and look at it myself. So that's not unless you think you.

A

Should you should you should have access to the to them as an account? No I do so.

B

One thing to keep in mind today that this will have no single-shot agents and currently this quality PF. We see you cannot configure them because the easy-to-implement cloudy carries out hogs and there is basically no way to do that. So.

C

You say we have no single-shot agents, that's what I think I heard you say for easy. No, okay, great.

B

Agents might be recycled. Okay, even.

A

B

Set one executors and shatter it the rate of completion.

C

And that's and that that is different than what we have with the a CIA agents there right, because if I understand correctly, the a CIA agents that are running on Azure are single shot. They are single-use. It's.

A

The sei or containers does it so a CIO just a container were so right now the account was configured to use as your virtual machine, which is the equivalent ec2 instances on Amazon, and we are using a CI, an azure and then it would be to either move on communities or maybe use right.

A

This is an alternative of this yeah yeah just yeah. Thank.

C

You thanks for the clarity all right, yeah, the the and I was relying on the ec2 instances being reduced that I. My mental model is that they are static. Agents for at least a lifetime of the virtual machine. I just see. Sometimes the count of machines connected is less than at other times and I had assumed. That was there wasn't a.

A

Bug in the show, going and.

C

A

Issue to plug in these three instances, when you don't need them anymore, got it. Thank you. The only thing. The only thing that I noticed is that if the, if the agent is attached to the master, it's used and then II turned agent become in a disconnected mode and then the agent is not cleaned up clean up after a while or adjust it all just nothing I, it has nothing to do and then the is to plug ins correctly, delete the engines. So that's why, sometime you see five six, ten questions.

A

Another thing that I also did was to reduce the number of other instances that we can deploy the main reason to the system mainly to better control. Our costs on the other accounts.

C

Thank you. Thank you very much. Joe.

A

Otherwise, if you don't have any other questions, I propose to stop the meeting here. We don't have to hold this meeting for thirty minutes if we don't have to yeah.

B

One question is about security audit, so Monica had to drop, thank you, but he might need some guidelines and some access to do that. Okay,.

A

Good summarize.

B

What exactly needs to be explored and process or the provision in this data? Okay, it would.

A

Be great eye contact imminent discuss within how we can help in what kind of access you needs. Okay, nice thanks. Everybody thanks for your time have a good day.