Jenkins Infrastructure Project, 25 Feb 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: INFRA Weekly Meeting 2020 02 25

Description

Jenkins Infrastructure Project Meeting - 2020-02-25

Notes - http://bit.ly/2T0oZ9v

A

So hi everybody I'm, starting the recording and thanks for being here for this new Jenkinson for a meeting, a quick follow-up on the last week's discussion. So first regarding the infrastructure as sponsoring, we are still in the process to get sponsored by Amazon. So right now before working and see other Genki, that I owe I just try to be I'm just waiting for them, but this one is going to be officially enabled. So we can start talking about that.

A

It's the simple fastly, so that as discussion that I had with fastly is, we have to sign any contract and they should send me a contract, but right now, I still not know at the bandwidth that we could have with fastly or what would be the terms of the contract. So it's still ongoing and still in discussion regarding the address but other accounts. We managed to be around six thousand three hundred K for the months, and so the billing is next week.

A

So we should be around seven thousand for this month, which is way better than what we had last with months, which is what which was around seventeen thousand for the account. So we managed to reduce because, but we still have to do better right now.

A

Regarding the discussion that we had last week about the junkie that I own, we mentioned some work that that teams at Leeds, with Packer image that he was to create very small virtual machine image, for better coordination so to to to speed up the the provisioning of those champions agents.

A

We spent some time to work on those and we are almost ready to put it to put that in place. The last thing that need to be done is just to create a new to create new credentials, speak before that and based on figures. The other changes that I, oh so, there's something that should come in the coming days and otherwise we did not spend more time with the G cask configuration. Firstly, I, don't think is that I am the main reason to that is. We are still waiting for Amazon's, but spen.

A

Sorry sponsoring to be enabled so right now we don't spend a lot of time on see. I, don't think it's a trio. Well, we also mentioned last week that we had to renew to JIRA license. This is done now, so we should be able to upgrade it again, um something that you have to work on in the coming weeks. um That's all do you have any questions regarding this nope, so I guess we can start talking about the agenda items, so let first are the first time is the update, Center and github integration. Oh.

B

Yeah, it was planned for the previous meeting, which we discuss to them. I mean the changes are now integrated. Okay,.

A

C

Need to be done for this. Those changes were the ones that allow us to do using topics to to manage the labels on on plugins that Jenkins. That I, oh, is that right, yeah! That's.

B

Right so I'm gonna show why it's in the agenda for this meeting, because it was about for meeting two weeks ago. We have some issues where this Prestage consistency. So just second.

B

But yeah overall it works. Our main problem is eventual consistency, because once you set the label, it takes up to several days to have the plug-in site. Updated she's, definitely not sustainable. What we need to find the reason and improve that okay, but he it's a sample topic. Okay,.

A

Just to be sure, the next item in the agenda meeting are still old ones. I, don't know. If Alex you want to talk about our duplicating windows, 2012 and the Windows Azure agents. Yes,.

D

We currently have Windows 2012 PM set up for a sure and as wondering what the consensus is on deprecating, the O's. We have Windows 2019 and it's it's a smaller VM image and faster. So I would like to deprecate the windows 2012 instances and moved only using the windows 2019 to.

A

Me I thought that you were I mean I thought for me that already did creating those old Windows machines, so I think you can just remove the configuration from Seattle to the day. Oh okay,.

D

I will take care of that.

A

D

Mark I think it's been the only one so far that has been doing some testing he's brought up a couple things, but I'm gonna, look into so I. Don't think we're ready to move forward to general availability, but so I'll continue looking at those issues, we do.

C

Ya know now the the issue that I had flagged. There may not be a general purpose issue: I, don't know how many other plugins mandate that they must have command line, get installed on the Windows machine, certainly begin plugins have to have it. They can't run their tests without it, but others I know that we use Jake it throughout the CI Jenkins io infrastructure. So we've avoided installing command line, get on a number of machines. As a result of that choice to you, Jake it yeah.

D

I think it's a I know of other plugins that do use git command line, so I definitely think it needs to be resolved, so I will look into it for sure.

A

Okay, the next point is regarding the Jenkins improv for the curb it's um Jim. Do you want to give an update here.

E

Yeah, you might, as you guys might have seen in the IRC in the marking. You I sent out the term sheet for the s/390 and I think mark put that down on the government's meeting this week. I think Oh Wednesday, so I guess tomorrow. So that's good I'm still waiting on in terms of sheet term sheet for power resources, but once we have those you guys should have full access to pull those into your infrastructure in terms of the PR.

E

That's still open, I haven't done anything with that I guess we kind of really need a wait for these terms of use sheets to get you guys signed off and get you guys access to start testing the whole PR that I put out okay.

A

Just for the context, so Jim James said that we need to signed, and so basically in the past, what we did is was always on a personal basis. Were someone sign that document and one of the reason to move to the CAF was to have a legal entity above us so personally responsible for that anymore? Obviously, the step I think is not really yet to sign that documents. So the thing was the to discuss tomorrow during the governance meeting should basically should should I sign the contract or should I should we should.

A

We asked them on to the CAF um but yeah in this. In this case, in this document I'm not, we were to sign it by myself. So yeah we are just waiting tomorrow.

A

B

You kiss message to the developers many increased, because if you expect anything to be voted honor the governance meeting, there should be at least discussion before. Okay, like an edge over this key say it might be enough to just discuss it with the Jenkins bought, even resulting the public I, just I'm sure what would be exactly the objective there. But.

C

Yeah yeah Olivier, if you're, okay with it I, can do that or you can do it, whichever you wish.

A

Yeah, if you can send that, if you can saw the description under made that many bees that would be really awesome all.

C

E

That's really it for me for the Jenkins instructure dr. builds. We you guys. We talked about I, think briefly. We have access to power on either on Amazon or Microsoft's cloud service. All right, not power, sorry arm arm. Sorry.

A

To be honest, that you didn't shake, that's normal yeah I think we should have like. We should have our resources but I. They forgot to check this on.

D

Its own as your the they only have arms. Well, they have other ones but they're part of this IOT edge product, which I've tried to figure out. If you can run like general workloads on and there's no clear documentation on that, so I'm still trying to figure out I may contact. Someone gave the support and find out if you can run general like docker type workloads on on there. Okay.

E

So you, the the main thing, though, is they have arm 64 support right and that that's usually the common one going forwards right. The raspberry, PI's and other arm platforms right and I haven't really heard that many people still using armed 32 or I've. Yet what the actual name of it is our.

D

Arm v7, 32 or arm 32 b7, that's what it I'm 32 v7.

B

To the current situation,.

B

E

Anything's really I think I think the darker like official pipeline for like official images, I, think they only have access to arm 64 to I, don't think they have access to arm. 32 I can double-check that, but it might be only 64. So even if we get 64 it'll be good improvement. I.

D

Think we need to figure out if we even need to support arm 32 I, don't know how many people would be using it. Okay, that's that's one thing. We should look at how much effort do we really want to spend on an arm 32.

C

We're we're not officially as far as I understood supporting any other 32-bit architecture. Right we've, we've deprecated, wind, 32-bit windows. Haven't we? No? We.

B

Didn't no, we did not okay, so how the communication basically says nothing about business operating systems we support and with regards to Windows and basically says nothing about versions of Windows. We support so I still have an action item to create a job or what is the communication page to the commented but yeah right now? If you can Java somewhere, you can expect the Jenkins that's there, which is obviously not exactly what would happen on them by these platforms.

B

D

For and also this is more for, like docker agents not for running the master on that platform. Necessarily so I think this. The support is a little bit different. Based on that as well now going forward, we did talk about at one point in the platform sig meeting that we would not be supporting 1:32 with the new installer. The new installer is only a 64-bit, so that will come up at some point for Windows at least.

C

And I think that's quite reasonable. You you just described it. That would be the master running. The Windows installer that seems very reasonable. To me, Jenkins Jenkins, running in a 32-bit world is for agents, may be interesting, I'm thinking it's less and less interesting for masters. Yeah.

B

The support in principle, because one still can download file, get it running sure.

D

It just won't be at about another box, so.

B

Personally, I'm, not that concerned about windows, 32 state I, do know that some people still use it for armed and for the new platforms for which we had official support. Even if you want to increment it.

E

That's all I had to say for that good to know that we at least are moving forward with that.

A

Okay, it sounds like the net thing that you have to in these tickets. Is there any other topic that we want to discuss, so I can bring mine, Marc, you're, muted,.

C

Yes, I wanted to talk about C Ida Jenkins, that I are monitoring but I'll, happily defer to. After whatever topic you have mine is not a long topic. I just wanted to summarize what I've learned and make a proposal so.

A

Yeah feel free to feel free to to talk about the mentoring and then usual minds of each.

C

Okay, so I'm gonna go ahead and share my screen. If that's okay, because I it'll help me frame the conversation, what you should, let's see which green are? Am I sharing now? What do you see on your screens? Everybody Google.

A

C

Okay, so you see the right page, then excellent duper, so I've spent a few days. Looking at that monitoring and trying to understand our monitoring and what I realize is. We've already got just about everything we need conceptually to do a good job of monitoring. My proposal is: let's take the systems and the concepts we've already got and extend them. So data dog is a world-class monitoring platform. We get it for free and it works great. Let's just keep using it yeah. Judging.

A

Piece of work and I'm a German, my only question is regarding because I saw that data Dutch made several updates recently I. Think someone under the text started working in the data to begin. We in the past it unleash reef new matrix, so it was not that useful and so I started investigating primitives. So really that's the changes that are you I'm using primitives.

A

So if you want to have a look with the kind of data that you have there, I think the main difference is that the data type plug-in is just exporting specific matrix where the parameters it's using the matrix printing or something like that. There is one one field, specific ligands that export a lot more matrix. If I can find after.

C

A meeting thank you, I hadn't, I, hadn't thought about Prometheus I. Think it's valid to concern. Consider my thought. Right now was I think we'll get faster results and better results quickly. If we stay with the solutions we've got rather than making a shift of solution, I had actually started my own monitoring system on my local network and was using it, but realized data dog is worlds in a way better than anything. I might consider recreating. It does amazing things in terms of dashboarding and measurements.

C

It's I'm quite impressed actually coming from a business prior to the current employer, I was working for a company selling. Those kind of solutions and data dog is an awesome competitor in that space. So.

A

C

A

Don't get me wrong: the data dogie, sorry awesome and I think I sent you an invitation to the dog account, so you should be able to maybe dashboard or look at the monitoring. Maybe do some other things. We can organize a session specifically for that I told my dad.

A

The reason why I deployed the primitives plug-in was because I was missing some specific metrics, which was provided by by the primitives pinion, but the main difference is that those metrics are exported to a graph, another spot that you have to maintain yourself and obviously it's time consuming. If you want to be sure that the service is always up and running so right now, it's more for certain purposes, but yeah I would be curious to see how the data dock plugins evolve over time.

C

Great okay, yeah, so one place where the data dog plug-in actually could offer. Something. Is this second item the canary jobs? So this was a concept that Tyler, Croix and and old APA had started and using what what are listed as infrastructure acceptance, tests, they're set right now of four or five tests that check very specific things should work into end and I. Think that concept is a good one. We should teach those tests to notify data dogs so that they can.

C

They can then raise an alarm if they start failing, so that we have a relatively few in the hundreds and hundreds of jobs. Those relatively few will raise, alarms to us so that we know something is seriously wrong with the infrastructure. Okay, that's one I'll I'll, negotiate our work with you Olivier to try to get it. The other is that in choosing which checks we should make. We've already got thirteen jeering tickets based on past outages and I. Think that's a good beginning, so my thought is: I'm gonna create an epic.

C

If we don't already have one that tracks these monitoring improvements and then the last and actually probably the most important one is be sure that we've got more people looking and that I think is probably the single most important thing. The other things will automate, but human beings can do an awful lot to help us as we get surprises and learn how to monitor better, and that was all that I had questions or comments. Yeah.

A

My just when comments I don't know if she saw that there is a Jenkins in for a slash into the repository where we automates checks. Basically, we used a reform to automate those sites, so yeah feel free to have a look at this and usually it's easier to. First, look at the dashboard create those check manually and then export those checks into our form. So we have a way to share those and.

A

And and I also showed that the begin so when I said so, there is one plug-in called matrix that exported you, you, you dad's house, and so this is the one that I the primitives plug-in is using. So basically the primitives or the primitives begin is just a way to explore those that at those metrics to to graph enough, for example,.

B

A bunch of such plugins- and you know one of the issues they trying to surface to bridge pou monitoring services and push monitoring services, because there are some periodic jobs which collect statistics and prepare that on the month, so that we can serve the data quickly. Trying service sure.

A

So I would just get it. I would just turn my screen as well. I think.

C

That means I have to stop sharing okay, go ahead.

A

Can you see my browser? Yes perfect? So basically, with the recent with a recent issue that we had with Michael Jenkins, that yo-yo gets enter the roster Jenkins I tell you I spent a little bit more time to investigate different solution that we could put in place to reduce the loads and package the changes that they use so right now we have one machine located on Amazon account and that machine is hosting so the three services that I mentioned and more the really the packaging, jobs and tasks and so on.

A

So that machine is, as a lot of things, is doing a lot of things and also is quite outdated because mirror brain is not maintained since now, so you have old version of Python will be an Sun, so I try to find ways to split the different services running on that machine on different container services, but obviously the main challenge that I have right now is: they are all kind of interconnected. So this the the screen that you see right now is one of the services that we could use to replace mirror brain.

A

So the idea here is that we have your bits contains all the files that we want to provide since the mirrors and also, basically, you can just browse the files on its own. For example, as you can see here, I just synchronize one of the mirrors, so there are two possibilities: eaters we have mirrors a variable. Was this big file? So let's say, for example, I can read that and just set it one oops this one is not working of this new take a different one. So.

A

If the file, if the file is so basically what my robe is, does its create a ash of the file and if the file with the correct ash is available on one of the mirrors, then it gives you an opportunity to download it directly from a mirrors close to your location and if the files is not located on a remote neurons, you just give you our neurons, so it does that works so right now, I configure my mirror bits with six zeros, as you can see here, so it's only working on HTTP right now, so we can really enforce this, for example, and it's really easy to deploy.

A

The main thing is: if we plan to to use more bits, you have to modify the script that we use when we release a new champions version in order to push the new artifacts directly on the orbits and obviously then we have to update a different yours, but that's one of the work that I did, and you know this also provides few views of view. So this one is just provides all the mirrors that you can use.

A

They have another one that you just asked that so, if you just put some stats at the end of their URL, you can see how many time I don't know that this specific packages, and otherwise you have a list of the different mirrors.

A

So this is one of the certain lists. There is an open PR on the repository, Jenkins and press search charts, so right now, I'm not sure yet the way we we push the data on that service, so either it's a based where we fetch the data from remotes mirrors, or we push it at that directly on your bits directly, when we release that main thing that I have to figure out right now.

A

The second service that I work on is a way to deploy more viewers so just a hand shots basically what it does it just run on a regular basis are seen commands to the notes from a remote errors, and so we can just have.

A

We can just have more than yours. The main reason why I started working on this is because, right now we have archives and changes yeah, that orange I'm running on the Rackspace accounts, and then we need to move that service somewhere else. So initially I thought it'd be just a simple Muir's and then I realized. That archive has a lot more data than just Muir's, so I still I have to I, probably have to deploy and provision an azure disk and move all the data on that I should be so.

A

I will probably move our character change in ceará org to usher in Rackspace, one of the other services that is also running on the current release, mirrors mirrors to change that I/o is boom.

A

So so the service, so this is a different service that I also started working. So basically, in this case, we just have ways to.

A

So, in this case, we just generate website called reddit today on can't use.

A

Those packages directly into the operating system but I still need more time to test how it's working, if it's working but yeah. So basically, what I wanted to share is that there is some activities about the way we do blow by Kato, junkie, that you're in your in your brains and mirrors the Jenkins at I/o. So if you have any inputs or thoughts or suggestions, I think it's a right time.

C

So Olivier you mentioned repo and I thought. I saw what looked like an artifact er II image is artifactory instance. At all sorry,.

A

He changed. No, no, no sorry so I a man shouldn't repo, but I wanted to sit back here. Sir. Thank you so repo. So we are not changing the artifactory. This is just a map and repository that I deployed the release environment, so I can just test and it was easier to deploy it and artifactory with a setup that I have. But this one I mean we won't change. It's mainly Picasso Jenkins at I/o mirror also changes at I/o and.

A

In the update center, but regarding the update center I'm, not there yet the main reason why it's not easy to to work on those is because we have quite a lot of scripts that are either uploading or downloading packages. We are seeing or SSH with crunch up and supplied installed before before deploying and, for example, before switching from Europe brains. To me, worse, I just wanted to be sure that I don't bridge or lose that in the democratic process.

A

I just have another question, so if you don't have any question on these parts, I would be more interested by these scales more about Alex harm. So I had to look to the release environments. So I fix athletes. I fix a few things so from the release process. Part I'm, sorry!

A

So, from the release process, part I had some resources issues, so I just deployed bigger machines. So, as you can see, it goes from 3 hours to release some champions to 1 over 35 sore right now, it's working again as expected. So if someone want to test the output generated by this I would be really happy, but the more importantly is that I worked on the packaging parts, which is here, and so in this case I, don't remember Alex.

A

We can you remember, I, don't remember if windows artifacts are signed or nuts or maybe what would be the best way to learn files.

D

Like let me take a look because I think you can, if, when you download the MSI, you can right-click and look at the properties and I'll tell you whether it's signed or not.

B

Sorry, older folks, like aside on.

A

The new release process.

A

D

This is not kaykai's I really expressed this. This is the new process, so I just need to verify. I'll take a look. I have a VPN setup. Second, I can look at it. So.

A

Everything is located in the auto factories.

A

Sorry, everything is located in artifacts, so look and otherwise the main thing that I've been working so because we don't have the cosign if we don't have access to do code, signing certificates yet I'm, currently working on the finishing parts and one of the thing that I'm really really relying on is instead of using arson or SSH, to publish artifacts somewhere I'm using Azure blob storage.

A

So I can just mount the azure blob storage directly in the release pipeline, and so just just like move files from here to there and so on, and it's also the same edge of blob storage that I'm using in the other services. So I can mount the same, the same, the same files in multiple locations, and so it's it's really simplified way.

A

We are publishing artifacts because, as soon as, for example, as soon as the artifact is generated in the Argentine's instance, it's copy and the blob storage and then mirror bits can use the same storage and so bodies directly available in their orbits and also available in package in the package.

A

You, for example, so yeah. That's that's what I'm working on right now! Yeah otherwise I think I covered all the things. I wanted to show you. So if you don't have any more question, I propose to stop here.

A

There is one left in history so mark and I. We do a small session about configuring, SSH access and other changes appropriate rates. So if you are interested to participate and he's feel for me to say it's the IDS, we need to do some knowledge transfer so to take some time to ask question and to be able to answer those questions.

A

Thanks everybody for your time,.