Jenkins Platform SIG, 15 Jan 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2021 01 15 Platform SIG

Description

Jenkins Platform SIG meeting January 15, 2021 with topics including Windows Docker, Jenkins Enhancement Proposal for Docker platform support, agent image improvements, and ARM processor support

A

Welcome everyone: this is the jenkins platform special interest group 15th of january 2021., happy new year. uh Here are the actual, the items that I had on the uh on the agenda or here's the whoops. Here's the agenda uh windows, docker images, docker image, build success and the jet platform proposal.

A

What other topics do we need to put on so alex? Is there uh you've got a pr right now for docker image. Build rework! Is that something you'd like to discuss here.

B

Yes, that's on the docker agent. We could talk about it.

A

Okay, all right great um gareth were there any topics that you needed to to be sure that we discussed here. uh Actually, I've got one arm uh increase, improving our arm, support that I'd like to put on the list.

C

uh No, I mean no real topics for me. I think for the windows, docker images. I think that's pretty much done. As far as I'm aware, there are some uh questions from daniel about the use of the jenkins for eval orc. I don't I'm not sure, I'm not sure what his suggestion is actually but.

B

I think he, I think what his concern is. Is we consider jenkins for eval to be a um like? uh uh What is his word excellent.

A

B

Experimental so he's saying: why are we publishing something that's pulled from that into a an official image in the jenkins org, so I think a way we could well. We can talk about it when we get there, but.

A

Great yeah, so so that's a good topic on the windows, docker images, anything else.

A

Cara were there any any particular concerns for you that we needed to put onto the agenda today.

C

A

All right, then, let's, let's go ahead and run the meeting, so we like to review open action items first, uh I still have the open. I am so proud that I've finally created the google doc, but I haven't touched it in a month with the holidays, so I promise that I will get back to this and and go with our further discussions.

A

The problem with the challenge here is that debian buster is our debian, which one is it debian 9 is coming to end of life, is reaching end of life and is already off support in at least some descriptions, but our images are based on it.

A

So we've we've already got that challenge, and this jep is a good excuse to drive forward to make those changes any concerns or topics there.

A

Okay, next centos options for adopt open, jdk, alex.

B

Yeah, so I I created a pr, I think it's linked there. um It just needs review. uh I oh uh olivier has reviewed it, um but um I don't know if we want any more reviews and we had a question as to whether we needed to um communicate that to users or not um so that was kind of where we're at I don't know if we need to if it. If it's you know the same, uh if it functions the same, do we necessarily need to make a big deal out of it? I don't know.

A

Right well, and- and we fundamentally don't have a change log for for those I mean we're typically not tagging our repositories when we deliver a new docker image right, so there isn't really a place for us to document a change log.

B

A

Now, maybe that's something we should change and find a way to to document.

A

Okay, great and.

B

We do have a way of doing that in some of like the docker agent and other places. So maybe we can reuse what we're doing there, which is just using github releases, but we just have to figure out, because we have so many different images and so forth. How do we want to do that?.

A

A

Good observation: thanks: okay, anything else on the on the centos options.

B

A

Okay, plug-in installation manager is, is still growing in its adoption and its usage we're now using it in the documentation.

A

um I like the idea alex that you had suggested earlier of a shim to allow install plugins.sh to continue to exist as a script, but just have it called plugin installation manager.

B

I I started on something like that. I I can create a pr from that, so I was just running the same tests that we had previously uh for the install plugins, but with the shim to make sure that it functioned the same way according to what it was. So um I I can probably maybe have a pr for that next week.

A

Oh okay, that's that's very encouraging! Okay, so if it were, if we were able to do it with the shim, I think I agree with you that it may not need a post. It's just yes! This is the new implementation of of plug-in installation manager.

B

And maybe we can put out a um uh on the developer list or something once we have like, builds going to jenkins for eval or something or push some images to jenkins for eval and say hey? Can you guys test this in your workflows, just to make sure there's um no corner cases that we don't have covered in our tests.

A

uh Now, and is there a facility in the image that we could deliver some well good good, so some sort of a pre-release.

B

A

You get feedback, excellent, okay, all right anything else on the plug-in installation manager.

A

Okay, next topic, then, was um multi-arch ci or multi-arch, alex I'm not nearly up to date. On that one, as much as I should be anything you want to share there I know jim is, is off on other projects right now at ibm, so he's not available for immediate work right.

B

Yeah, I think um so I mean I kind of reviewed what he what he did and it seems fine, but we also need to figure out how it um how it needs to fit in with, like the j, any jenkins file changes and things like that, so that it actually gets built, uses the correct agents for the different architectures.

B

All the um uh you know, all the images getting tagged correctly, that sort of stuff, so um I don't know I might set up like a test instance or something and locally and and run through some of the flows. Have it pushed to my um just my repository area on docker hub just to see how things work.

A

Good, okay, so that I like that, that's that makes sense um that that how do we? How do we stage tests or how do we stage? Development is interesting also for the jenkins core releases, particularly security releases, where olivier is wanting to move more and more towards staged, builds as a as a standard process, and we certainly don't stage. Docker builds right now right, our docker builds. Are we build it and it goes. It's published immediately.

B

C

Okay, great okay, on that multi-arch stuff, is that just building for multiple architectures or actually using like docker multi-arch support.

B

So we have well, we have um a an s390x and a powerpc agent that have been um donated by, I think ibm right right, and so there is, um there are people who want. You, know s390x and powerpc and and arm agent images.

B

So it's building and releasing those images we've done it in the past with uh qemu, but it it provided some challenges in terms of performance and um actual functionality once the image was actually being used, um which is why we wanted to go to um using actual physical agents uh for those architectures, and then we.

C

B

Do have arm on the um aws uh arm, 64 agents on aws, so we've we have um physical agents that we can use for all those architectures. Now.

C

So that's that's still pushing it's pushing multiple images like an image for each architecture, correct.

B

C

B

Yeah just different architecture um for the for that image.

A

So so, but there is a concept: isn't there in in docker of a single image, that is multiple architecture or have I misunderstood that.

B

So it deals with manifests and things like that. I'm not.

A

B

Up to speed on that, but you have to push it like a manifest, um and I think that jim's changes do that they deal with the manifest and and so forth.

C

You push like multiple layers like a layer for each archetype, each architecture, oh that and that makes sense yeah. So when you reference it, you just reference whatever you know, I don't know like a jenkins lts and if you're on arm you would get the arm version. You don't have to explicitly say. I want the arm version.

A

Okay, so, and and alex your observation is, as far as you can tell jim's jim's pull request is proposing to use multi-arch that way, so it would be that really would be published, publishing an image for all architectures. That is a single image, but with multiple layers with multiple with layers that are platform specific.

B

So my understanding is that it's using manifest to do something similar. I don't know the underlying implementation, but like the manifest, is basically a a generic definition of the image which then has what architectures are supported for that image.

B

um And so, but that's my understanding, I'm not by any means uh any sort of expert on the multi-arc stuff. But that's my understanding of what I've seen.

A

Great all right, and that may be one where gareth your understanding of, might be benefited by. We, we get get some benefit by having you review that pull request. So that's.

C

Yeah I mean this is something that we're I've got a pr, that's uh on top of damien's pr actually for the changes for the sort of jenkins and for pipeline library to add in multi-arch support for that. So it's something that we're sort of actively testing out, seeing how we can do that.

A

Great okay, so so there are other places where you're looking at going multi-arch yeah, because I'm I'm interested from a from a jenkins infrastructure. I'm interested because amazon offers lower cost compute from arm now than they do from from amd 64..

A

So there's some some interest for me and lower, potentially lowering infra costs by considering putting some of our images into arm put some of our workload onto arm rather than being on on.

C

A

C

A bit of a spider in the works as well with the new mri stuff.

A

Another example, m1 is, is a good example of that for desktop for laptops.

C

Everyone's having to update.

A

Right: okay, anything else on the multi-arch docker question.

A

All right next topic, then docker images for ltsc, 2019, gareth.

C

Yes, I think um to the the windows, the main, the main images are all being published uh and built, and uh so are the agents all of those pls are merged. um There is yeah. There was an outstanding question from daniel around using uh so the adopt open, jdk images that aren't yet available.

C

We were publishing uh publishing them under the jenkinsfreeval.org, with the idea that we'd swapped them out when I think, as alex as your pr was merged,.

C

But I'm guessing that the when we're publishing the agent images on top of those we're actually pushing those to jenkins- and I think is that- is that the way the confusion is lying or is that what the end of where daniel's question is coming from?.

B

Yeah, I think so yeah.

C

Because the parent image effectively becomes yeah, you have a jenkins image depending on jenkins for a vowel image.

B

So we could, I mean we could push those other images, those base images to the jenkins org um that that shouldn't be a problem. We could do that um and then once they release their images, we could just switch that over. um That would that would I mean it. The only reason it's not I mean the only reason it's not official from them is because that pr hasn't been merged. So um I think they're it's hard to tell when they're going to merge it.

B

I've had ones that have gone for six months, some that have gotten some prs. That are, you know two days so um depending on when they merge that we can just push those base images when we build them into the jenkins org on docker hub for now, um and then maybe that would just alleviate daniel's concerns for now.

C

Do we we know, obviously people run um agents on windows um or they run jenkins on windows. How many people do we know of that? Use these windows server, core actual docker images.

B

Well, I think it's going to be going to become more useful with uh like kubernetes. um If people need to do, um linux builds and windows builds, they may have, um they may start using these images more with kubernetes.

B

I don't honestly see a bunch of people using these just directly, just because docker on windows is not as capable as docker on linux. So that's my take on it is. I think that the major use is is would be for the kubernetes people.

C

It's just that the size of the image is, is the images that produce it, they're pretty big about 2.8 gig um for the windows, images, which may be a concern.

B

So I did I have, I don't know if I submitted it, but I had another pr to do to enable slim images for the adopt open, jdk windows and I was able to get it down to like for the nano server at least down to like the 300 megabyte range, um which is a lot more useful. I think but yeah. I agree. The the 2.8 gig image size is pretty prohibitive, so.

B

But I I don't remember if I got that pr merged or not.

B

I'll look at that, but again that's nano server as well, um but for agents. Nano servers should work. Okay,.

A

Yeah I I thought we had bumped into challenges with nano server because it was missing crucial things, but but I I don't remember so: okay, we.

B

Do have nano server um images right now for the agents um not for the reserve.

C

B

For the server or for the controller I mean, um but we do have for the agents, um so um we have and get runs on it um and and some stuff like that, so at least those things work under the agent image. So I I like. I have no idea how many people are actually using the images.

B

That's that's kind of hard to know as to whether they're actually useful to people right.

A

Okay, gareth anything else on windows, docker.

C

I know that's, that's, that's it ready.

A

Okay, so next topic was the ci dot jenkins at io master branch, and the question was: could we set a goal to bring make it pass and we're not passing yet any comments or feedback there.

B

It's it's the publish, experimental step, that's failing! So the build is working correctly, but it's once it tries to publish those to the jenkins for eval. um It's failing because of we need to update the jenkins file um because of some of the changes that uh on a pr that jim made.

B

um I believe I have a pr to fix that, but I need to double check that the other option is we just disable that publish to jenkins for eval for now, because I don't know if people are actually using those images or not again, we don't it's hard to know whether they're using those who's using those and how many you know how many and stuff like that.

A

B

A

Are the two options? Okay,.

A

A

Right so next topic I'd propose to skip, because I don't ha I have done nothing to prepare myself for a conversation with it. Unless someone else feels like they really want us to talk about the docker platform. Jeff draft I'd like to skip it for now any objections.

A

It's fine with me. Okay I'll excuse. My shame for not for now we'll revisit, rework, revise and reopen discussion.

B

Is that is that document um open for commenting.

A

It is okay, it's it should be. Let me double check just to be sure it should be shared for. Oh, no, it's not open for commenting. Shame on me now it is okay. Anyone can comment now. Sorry about that. I don't know why I didn't do that. That's that's! That's embarrassing! In another level, okay.

A

Docker image build rework so alex.

B

Yeah, so this is so, I have a pr open um on the docker agent to rework so that even the linux images are built and published from within the rci infrastructure.

B

And the reason for this is there have been some prs in the past that want to add things like build date, metadata and things like that and using the auto, build features of docker hub. Do not allow you to do things like that so or override environment variables um for publishing or tagging, or things like that. So the the pr just um builds and publishes the images for both windows and linux within the ci.jenkins.io or trusted depending on on uh well, for it would only publish untrusted.

B

It would build on a cia.jenkins.org for both.

A

Okay, so it builds on ci.jenkins.io and publishes okay.

B

It basically takes what we're doing with windows right now, which is.

C

B

Doing the build and publish all within our infrastructure because they don't support auto build on the docker hub builds. So then it moves both into our infrastructure.

A

All right so then need to review there. Are there any compatibility risks hiding there compatibility threats that we need to be aware of or notify people of it should be just a direct port over or.

B

Yeah, it should be, um and one advantage it also gives us is jeff thompson has had um you know, issues having to go through like 20 docker files and update the remoting version.

B

um This also reworks it so that you can edit a single file, and it will pull that remoting version in um for the new build, so it it. It provides us some additional capabilities um that are gonna, be nice. Hopefully,.

A

ah Okay, now I've see I've, I've been thrilled with depend about in other areas, can depend about manage docker files as well or no for and don't.

B

Know results. Yes, it can.

C

B

Aspects of them.

C

Yeah certain aspects of them yeah.

A

Okay- but this does not this- this pr does not go that distance. It's it gives us a single place to manage them, and then we would submit prs the usual way for that single phrase. Yes, okay, excellent anything else on the image build: rework alex nope.

B

And this is just kind of the first one I after I do this one. I I and you know, get feedback and so forth. I would do similar things to the other images. um This is just gonna, be the easiest one.

A

And, and- and is this for the core, the the the controller image or.

B

No, this, this is for the docker agent, which is.

A

B

Base agent image for like the inbound agent, uh it's not the base image for the ssh agent, but it is the um the base for the inbound and just kind of the basic remoting agent.

A

Got it okay, so once that is implemented for this base, image then look to apply the same, apply the same techniques in the other images right got it. Okay,.

A

Anything else there alex.

B

Nope, that should be all right.

A

Okay, so my last, my topic here is improving arm support and- and this is more a question than um than an answer- is what steps do we need to do to take to publish arm images?

A

Is it just do the the multi-arch that we've already talked about.

B

So, are you saying for the controller or for the um for everything.

A

Or what so, for me first would be agents and then controller, because, with with I find the the amazon arm, eight instances interesting even for the controller they're, they like offer a 40 or 20 to 40 percent discount on terms of compared to intel architecture.

B

So we do, we do have um amis for arm 64 right now that we're actually using in our in that are connected to like ci.jenkins.io. um So that might be something you can use in the interim. um It's it's our it's the normal. um We use packer to build it and it's basically the same as our ubuntu based build agents, but it's arm 64 instead of the amd64 and we've been using those on c.j concedo.

B

So that's one option initially: if you just need the ami identifier, you should be able to use it um on aws if you have like internal usage for yourself, but we already do have those connected to um ci dodge inside oh and trusted. I believe.

A

Right and and so I've been I've been using those quite happily the the arm, the arm agent on ci.jenkins.io, I confirmed yesterday again it works great.

A

So then, in terms of image generation, the arm support on our docker image, as I guess is what I should say, because what we've really got, I'm confident we run on arm I'm using it. um I haven't, you run the controller yet, but the agents, I don't think we have an agent's docker image yet for arm. Is that correct, alex? That's.

B

Correct we don't have a, uh we, don't, publish an armed version, you could take what's there and build it on an arm system, um and it would it would. You should be able to build it. Just fine.

A

Got it okay, thank you, and then I assume the same applies to the controller. Is that as a first test, I could build my own image on arm and use that for experiments and to see hey? How does it behave.

B

Yeah- and we have that in our our flow- to build um those images right now, once jim's, yara and stuff get merged in, but you can still also do the same thing and and build the normal um like debian image, for instance um on an arm platform, and it would generate a docker image for arm that you could use for the controller. I've actually done that and it works just fine.

A

Okay, excellent well that address that answered my question and I've got got things going on there. That are, I think, arm is a particularly interesting platform. Great. Thank you.

A

Any other topics we should address before we close our meeting. Oh I take it back. I've got one more contributor summit, I'll, be sending a proposal shortly for a contributor summit associated with fosdem or roughly two weeks after fosdem, uh and what I'm going to suggest is that sigs spend time in the in the introduction segment summarizing their results and where their and their plans, and so we'll we'll likely get that responsibility and I'll probably try to be first voice but assure that you're. All aware of that proposal and I'll send a copy to you.

A

A

Else all right, let's call it a meeting. Thank you. I will post the recording after the after it's finished, processing thanks.