Jenkins Infrastructure Project, 12 Jan 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2021 01 12 Jenkins Infra Meeting

Description

Jenkins infrastructure meeting with discussions including upcoming core release, archive.jenkins.io, Docker image builds, and Jenkins release candidate build/test

A

And so I'll take notes and rely on others to.

B

Chat, perfect uh just start the recording, so we are ready to start hi everybody um two days to the agenda. um We are the first announcement to say that, because we are doing a jenkins core release in the coming days, um please do not modify anything related to relays.ci um so hold on any pr that could affect that service.

B

um The second thing that I noticed today is archive. The jenkins I o is really unreliable at the moment. I still have to find some time to investigate what's wrong. um What I discovered today is it affected. The monitoring check that the check, if we can download packages um cara, uh made a pr about that. So the idea is to use, get the jenkins that I o instead of archive, searching that I o to monitor.

B

If we can download packages which obviously makes sense, um which is the service used by everybody um on another, so the next topic um we made some garrets and damian made some nice improvement regardless.

C

C

Mark did you end up reverting your change that made the pipeline not fail. If archives was down or did you leave, it commented out or disabled.

D

A

Point I have in archives.jenkins.org the change I made has been reverted, but it worked last week for the 2.274 release. So I'm not sure how to answer that tim. That's a good question.

D

There's probably a risk there that could cause the pipeline to fail.

A

Yes, risk that the 2.275 and 2.263.2 release pipeline won't be able to upload to ci to archives. However, now olivier, because that's a stage those are staged releases daniel's announced that they are security releases since they're staged. I assume they are not yet uploaded to archives.jenkins.io.

B

A

They're staged.

B

Yeah, which means that we must be sure that archive the jenkins leo is ready when we publish the packages because it may fail the publishing script. That's a really good point, or we.

C

Disable always re-add marks, disable or or true to it, because it doesn't really matter if they're right. Oh, yes,.

B

That's a really good. Oh.

A

That's um olivia: do you want to have that conversation with daniel beck to to see which is best for him? I think it would be best for now here to disable sorry, oh daniel hello, daniel. I miss that you're here.

E

Thank you. I didn't want to interrupt the talking, but now that you're mentioning me.

A

So do you have a preference there is, is since the release those releases are being staged, they aren't immediately dependent on archives.jenkins.io. Should we disable the attempt to publish to archives, because it's currently not trustworthy online.

E

Well, I'm I may have missed the beginning of this topic, um so what is archives used for exactly so? What we need is once we publish it, it needs to be made available to everyone. Yes,.

B

Archives archive the jenkins audio is used as a fallback, so in the script when we do the thing.sh one of the stage is to upload the packages to archive the jenkins that I o. So it's not directly used, but it may affect the thing that is such scripts and basically what's happening with that services.

B

For some reason from time to time, it's um really slow, and so we get time out issues. So people don't do not download artefacts from theirs, but as part of the release process, we upload artifacts on that machine.

B

So it's probably better to just remove to disable that that service for now.

E

So is that, what's what's currently behind the fallback host name.

B

So the fallback awesome name is um get the jenkins. That are you in fact, so when, when we do, when you do the release, we upload the art effects um to an azure file, storage and so fallback that jenkins io is the same. So.

C

Nothing's used right now, it's just a full wet fallback.

E

Okay, so uh we do not even need archives, jenkins, io, before all the mirrors are synced, because that is occasionally a problem. I think that was uh tim. What we worked on with plugins mid last year when the mirror distribution was delayed, we made sure that it's immediately available from somewhere- and this is not implemented with the archives, correct.

B

So uh for the plugin, uh maybe for the plugin here.

C

So it's using the it's using your file. Storage is full back right now.

B

So so, when you look, when you look at the the script sync.sh, you see that we upload packages into multiple locations. We upload them to azure file storage and we upload them to archive. um The thing is in the past. Archive has had a limit on the network bandwidth, it was really slow and the purpose of archive was really like as a full batch a service, so archive contain every artifacts that we published since the beginning of the jenkins project. While most of the mirrors only contain packages for the last year, something like that.

E

Okay, so it's not important for the distribution.

E

No, so if it can be safely done, my preferred approach would be to remove that from whatever scripts would do the uploads to ensure that they finish with everything else that is actually needed.

B

Yeah, it's safer in the current situation.

A

So I think yeah, so I think what that means is that's just reinforcing that we should reapply the change that I had done a week or two ago to temporarily disable upload tim had mentioned it earlier and I think that's that's just it. We need to apply again the um the skip archives.

A

uh Change from two weeks ago, when it was offline as well then great okay uh is that, do you want me to submit the poll request and then daniel you and olivier review it.

B

Yeah, if you can, if you can submit the pull request, that would be nice.

A

All right we'll do that later. Today, uh it's going to be several hours yet. Is that, okay, that it's delayed several hours while I'm in other meetings.

B

um Yeah, that's fine. um My only question is, I'm not sure to understand which fix is needed. I thought that you had to manually modify the scripts.

A

No, no, no! I I pushed I pushed a change into the repository it wasn't. It wasn't a manual change. It was it's really tracked as a pr.

C

Do you pull it on the machine, though right? Okay, okay, okay,.

B

I see which repository you mean: okay, okay, thanks any last um comments on this topic. No, um the next topic, which is the way we build the car images. So, as I was saying, garrett and damian made some progress here. So the way it's working now is the reason. So the reason why they started working on that is because we start moving more and more jobs on the jenkins running on kubernetes, because it's faster to to to provision jobs, um notes on kubernetes.

B

But the challenge is because we do not run the duck because we don't have access to the duck argument. um Yeah. We had to modify slightly the shard library used here, so I can um so. I won't.

B

I can show you what it looks like, so basically what it says right. What what we are doing right now is, if you want to provide, if you want to add a new um docker image on the jenkins here, infra organization, you just create a git requester with the name docker dash and the name whatever name you want to use inside. You create a jenkins file with the right write function and then it will automatically.

B

It would automatically create a pipeline job on infrared.ci and uh it will run some linting tests of with using adolence. So we can. We can look at the results of those addon tests and then it builds a docker image using both um the branches and the tags, um which means that oh yeah um is.

C

B

C

Is that just showing you consent for at the moment.

B

Or can it do jenkins as well? So let me let me show you um um mark. Can I share my? Can I show the screen, it is just doing jenkins. Yes,.

C

I think it has access to genesis, so we might be able to, because I've got a couple that ideally, we moved over to possibly that.

B

So I ideally one of the things that I see here is: we could move many of um docker image build from trusted.ci to infrared.ci, so in order to increase the visibility of those jobs and um it's just easier to manage right now,.

C

B

So if I show you what it looks like here,.

B

B

um New browser- I know it's already here so I can just show you.

B

B

It's here sorry.

B

Do you see my screen yeah, so we now have a new folder, which is the car beats, and if you look at it, we right now have a bunch of um docker images. So as long as you create a git, as I said, a gate requester with um with the name that match decker dash data log, it will automatically be added here, and so, if you look at one, we can either build image based on branch are on tag.

B

And so, if we build on branch it will push the latest and the mass branch, and if we we use the tags, um we can also trigger for a specific tag, and so, let's say, for instance, this one. So the idea was to use um release drafter um to to generate new version of docker images in this. In this case, so the version would not match the different version that you have inside um the docker image, and so we did so. Yes, we do some linking test build, but what is interesting here is.

B

We also have the view of adolence warning, and so you can see what are the errors um found in this docker image and if you look, if you click on the difference, so we see that um this error is located into on two lines. And if I look at the docker file line, seven, it suggests that we run application with no install recommends. um So we only install the packages that we need um and so yeah that was a nice nice improvement and then it automatically pushed the images on docker hub.

B

um So that's the current situation. um The manual procedure right now when we need to add a new docker image, is to create the hub repository in advance.

C

Is that actually.

B

C

I thought, as an owner of the orc, it automatically um created a repo when you pushed.

B

Undocumented, I thought so. Oh that would be interesting.

C

Least, under my personal account, I think it does that I'm pretty sure I can just.

B

C

Yeah, so that that that would be a nice improvement.

F

B

Creates automatically.

F

For you, uh even though it's a public one with the default configuration set so depending on what we expect, we are not totally sure so by default. If you expect it to be public with the default that it's okay, otherwise, you must create it before, as soon as the token is able to push okay.

B

So it's odd okay, that's that's interesting! Then you anyway, we don't have private images and we don't push. We don't have credentials inside decker images, yeah, so so yeah. um Now the question is more about all those docker files that we're building where we have the docker file next to an application um and the example that pop-up in my mind, is the account app where we have the java codes and the docker file.

B

So we may have to manually add the job here. What I've been thinking is instead of adding jobs here, based on the git repository name, maybe having a way to let's say ad, based on a git repository label or whatever, if you, if we have a specific.

C

Change, there's.

B

C

Yeah, there's a really nice pull request open on the github branch source for doing it by topic really efficiently too.

B

So maybe maybe that would help you know so we could have. I mean it's just a way to visualize all the builds, but in this case we can clearly see. I mean that would be nice to have every docker built from here.

B

F

um Just just to add something: there is an upcoming process that will come within the shared library that, if um an incoming an upcoming pull request that will allow to run some tests between the build and the deploy part when building a docker image.

F

This test will be only using a driver named, so it will only check for the presence you can only check for presence of files for file contents inside the content of the docker image. It won't use a docker engine for the security risk that it will cause inside kubernetes, but it will allow to document and work on tdd when you have a dockerfile image, because you say I expect to have this file with this properties with eventually this content inside my image.

F

So then you will have a kind of no no regression testing for docker images there.

F

If we want to go further, we must use a set of agents that will be able to spawn the image on vm, because if you try to run docker in docker, even though you have the root less available, you still need some capsis admin rights on the host underlying host.

F

So the best solution for today will still to might be, if you want to run some docker workload, to run a virtual machine dedicated only for that that will imply stashing and unstashing the tag image before pushing it to registry.

F

The goal here is to avoid pushing an image on any docker registry without having run some tests. Ideally, I would like to add then security scanning on the static image, but already having some tests. So you have the links that helps you to parse the dockerfile and find good and bad patterns. Then you build it with img, so no docker engine involved, and then we use a google container test for testing that will only deep dive inside the image and if the test pass, that will be run on each pull request.

F

And if you are on the main branch, then it will deploy automatically.

C

Security scanning shouldn't be too hard to add, would have thought, um doesn't need any doesn't need the vm for that yeah sounds really good.

B

Yeah, that's another, that's another improvement that would be nice.

B

um Any last suggestion question.

B

No, so the next. The next topic is about mirrors. um We start to need more mirrors um right now. um We had issues with severion over the past weeks, which has been unreliable and so now I would like to maybe write a blog post to promote and ask um contributors to provide new house. So basically, what I did is I improved the hem charts that we have right now to also install a nursing server in under my house.

B

The reason why we need air sync is because mirror bits: users seem to together metadata information for the files and to know if the mirror is there or not. So um so we just need to be able to reach from from your orbits the extending server, so everything is in place. um I just have one issue that I still have to identify why um it's the mirror is marked as down, even even if it's there um but yeah. That's that's the what's the missing parts um before before: um promoting that that components.

A

So olivier I've I've considered using some oracle infrastructure as a possible mirror just as an experiment. Would you be willing or interested? If I were to offer a? It won't be what I would call in an interesting locale, because if I remember they don't have a data center in india or in australia, but it could be phoenix arizona and the us um is that interesting or is it better that we look outside for for this initial experience? Is it.

B

Already mature.

A

Enough that you don't need me to test.

B

Drive it so so, basically, right now we have uh ham charts. So if you have a kubernetes cluster, that's definitely super simple to install, because the way the hem chart is configured is working is. It starts multiple container, one container that that provides you the content from a specific directory, just an apache daemon.

B

There is a second container, which is another sync that provides you the content from the same directory from I mean from the directory, also um as a nursing, and then you have a third container, which is a crunch job that regularly pulls information from from get the jenkins that I o so from from one one mirrors, one one trusted mirrors basically, and so the first time you deploy the charts, you need to wait the time that it takes to download the packages to have a local copy of one of the mirrors, and then we can use it.

B

If you don't, if you don't have a kubernetes cluster, if you want to maintain, I mean, if you just have a virtual machine, um you can reuse either the containers on your machine, but I mean at least you will have to do some module operation.

B

um So I don't have a ready to deploy thing for.

A

Well, but but that's great, because I would want to do it in kubernetes, so I missed in the notes the container with rsync a container that pulls from get get.jenkins.io on a schedule. What was the first container so.

B

You have ursing apache and um an enchrome uh cronjob, okay got it so, and so the idea is really. I mean it's really too to have it to work independently, um and so the rsync daemon is a read only server. So it's really just I mean you just provide information um and right in the current configuration.

B

It only allows connection from specific location, so the rsync is really designed to be used from get.jenkins that I o.

A

B

Which means that.

A

B

Don't know if you deployed it on your cluster, you won't be able to to test its uh frame from your machine. um We have, I mean.

B

And so we have dedicated docker images built on the jenkins project on inflated ci and publishing like a hub. So everything is probably there all right.

A

And did we did we have anything on the serverion mirror being unreliable? Do we need to notify them of that, or have we just given up that? We can't trust it and.

B

The I mean, I think I I have- I have to send them an email to report the issue. My I think the problem is they just have too much traffic. um They are not only providing data for the jenkins, they also offer a mirror for debian and other projects. So my guess, if we just, we just ask them to match, so I think it will be easier. The day we just provide more mirrors in there in the same area than them.

C

Yeah they just sent an email once complaining that we'd turn them off and I think forgot to turn them back on at some point. So they didn't notice that.

B

The the problem is, and that's the same, that's the same issue that we have on our infrastructure. You always have moment in the day where um you have a lot of traffic um the moment between europe and united states. um During the morning I mean the morning from for the united states. You always have small peaks at that time and it's exactly the same.

B

That happened on the update center, and so, if you only have one mirach and in the case in that case, I think you we only have two or three mirrors in the united states, um the problem. The problem is, and it's hard to detect uh the problem. Is that me, the mirror? Is there it's working and then for 15 minutes, 20 minutes, 30 minutes the mirror is down, and so nobody is able to, let's say, install plugins for instance, and so we just have to be sure that we have enough.

A

Yeah, so so the the mirror, their mirror is actually in the netherlands and one of the problems is, it is incorrectly identified as being on the east coast of the united states and and that that causes all sorts of other issues. So for me, leaving it offline for now is fine. It's healthier for our east coast users, they're not getting they're, not being not asking for data from the netherlands that could be available from new.

A

B

um Any last question regarding mirrors infrastructure, so the next topic we are running over time, so I'm trying to go quick. um So we have few prs that we affect infrared.ci and release.ci. So I would like to plan a maintenance window next week with garrett and damian. So we upgrade.

B

We merge those pr manually and we upgrade everything. So we are sure that um we are running the hamvit v3 hem chart for jenkins and uh a few other pr. So basically, what it means is. We need to send um uh um an announcement on status.jenkinsonio to announce that the service may be down um and also we have to announce the mailing lists. So that's one of the things that I have to work with carrots.

B

Regarding that um the two next topics: um we have one about acceptance tests, so I would like I would like to to to use the jenkins acceptance test to test um changing stacker images I found to get repository, but I'm not sure um which one I should use, but I propose to to to to talk about that um during the next jenkins information.

B

um It's not urgent for today, since we are already running our time, uh proposed to move it to the next meeting and the last topic which is highlight for the new year post, um unlike oleg, but oleg is not there.

B

I don't know. If mark you have some inputs um on that topic.

B

I just okay just share information, saying that you wrote a document. I guess.

A

So sorry here come so there is. There is a new year's blog post due to the continuous delivery foundation, the 13th so tomorrow- uh and this is the draft of that post.

A

uh If we have specific things in infrastructure that we would like to highlight in that 2020 summary, we should propose them to it today, all right, the I've I've put some things done, infrastructure in here already yeah. I like.

B

A

Like releasing jenkins core, exactly like releasing jenkins release, automation, like uh all sorts of other things that are infra related, are in here already, but if you please feel free to review it and if you think we missed something. This is a. This is the the day to propose a change to it, because it's due tomorrow, okay.

B

I'll try to look at it.

B

So any last thing you want to discuss. Otherwise I guess we can finish the present the talk here.

B

So I'm going to three one, two three, then, thanks for your time, everybody um see you in rc and um see you in one week.

B