Jenkins Platform SIG, 13 Aug 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Platform SIG 2020 08 13

Description

Jenkins platform special interest group August 13, 2020 with topics focused on Alpine Docker image updates, AdoptOpenJDK for Debian and CentOS images, alternatives for image rebuild, and plans to drop support for Debian 9 (stretch) as it approaches the end of its security fix support.

A

And then off we go so welcome to the platform sig meeting the 13th of august we've got a topic on the agenda related to adopt open action items we'll review briefly, adopt open jdk for docker on alpine or for let's type docker on alpine. Only uh alpine doc adopt open jdk 8 for docker on alpine and then we'll create a separate item for adopt open jdk 8 for docker on other.

A

A

Okay, so there we go.

A

So there's been a and so we'll discuss that separately: okay and deprecate, the docker agent stretch images github app authentication on ci.jenkins.I. Oh, let's see am I sharing my screen.

A

It would help if I shared my screen instead of just talking to it. Sorry about that there we go okay and the plugin installation manager in docker images.

A

And I that was probably tim yeah and I think that one is just a status report, so that can be me or you alex right, because it's it's availa, it's intended to be available preview mode uh and we'll we'll talk to it anything else that needs to be on the agenda alex.

B

um I don't think so: okay, great.

A

All right so open action items, awkward and an embarrassing one, a jet for docker operating system support has been made intensely more needed by recent disclosures recent public disclosures.

A

Then we've got the docker, build, rework pr, that's going to be delayed and then an alpine image update pr that one I'd been evaluating, but I think we need to shift our focus for today on to adopt open jdk8 for docker on alpine.

A

So this one. The story is that a there's been a public disclosure of security, vulnerability.

A

So work is being done now to address that and there's really no fix based on open jdk, because open jdk has dropped alpine support.

A

So we must use adopt open, jdk8.

A

If we want um want to fix and that's what oleg has submitted as a proposal alex any concern there from you in terms of accepting a breaking change in order to make that transition.

A

And so the there are pull requests pending to update uh docker the docker image, the agent images uh and it's multiple agent images. If I recall correctly as well.

C

Yeah, we need to ship all the repositories, so in this case, docker ssh build agents is the biggest priority, taking the nature of disclosed issues, but other images also require update, so go ahead. Yeah inbound also with agent also all glp agents- think because they also include dependencies on alpine versions.

C

So I will focus on jenkins controller release and on agents first, and I would appreciate some reviews for ssh agents. I tested the link to the chat addicts because the pull request builder fails on windows due to strange issue and yeah.

C

Since it's on windows, I make both assumptions that we don't uh want to get blocked by that. But at the same time I would appreciate review if there is a kicks, we could apply.

B

Yeah, I think the issue is that um this one was never updated to pin the pester at a specific version, so I can I can put in a pr for this. It should be easy.

C

So if you'll have someone wait for that, it would be great. I mean.

B

C

I handle the controller and common ancient images sure thank you.

C

A

All right so then we we would benefit some additional testing, but the oleg, I think you said that the automation other than on the ssh agents- and it was only the ssh agents that had a windows agent test failure- is that in general, on all of them.

C

No, it was under one image, the point about engines. We still need to get the upstream image here to this first before get full cir running. That's why I focus on it first: okay, because yeah we need uh additional time and yeah release draft, isn't working as well likely because of some issues so yeah. I will switch it to github actions.

C

A

All right anything any other items on adopt open, jdk 8 for docker on alpine.

A

So we had discussed in previous meetings potentially doing adopt open jdk8 on buster and on santos monsentos um again. I think the in this case I believe we're not yet distributing those so buster because stretch.

A

Will be unsupported.

A

January of 2021- and I guess this leads us to a question of- should what should our debian plan be in general, given that the debian project has announced the intent to release debian bullseye in january of 2021, and they only support, they only provide security fixes for current for stable and previous stable, not for anything, older and so beginning january, 21 or sometime shortly thereafter.

A

They will stop security, fixing.

B

B

I thought they'd already done that. So that's that's good news, at least, but that, if we're so, if we're not pinning uh to a specific version of like adopt open jdk, for instance, um when they.

C

B

We would get the upgrade to whatever the new version is so.

A

Oh well, and I think that that the pattern oleg's shown us is he's pinning strongly to a specific version of adopt open, jdk's, uh docker image, and I prefer that as well. So does that? Does that mean that this discussion, the discussion of buster or centos, is actually no? I guess it's not irrelevant, because they adopt open jdk provides a buster image.

A

Is that is that correct alex? I don't remember there. I don't remember if they provide a separate buster tag.

A

Yeah, let me do a quick check, excuse my having to do the check while we're live here, but adopt open jdk and what we're looking for is the top.

B

One is the um the top one, and then this is the official. Oh, oh, and we need unofficial so well we're we have to use some of the unofficial images like the buster is part of their official.

A

Oh, oh, it is okay, all right. So if we look at let's, I think it is. Let's look first then, at adopt open jdk, and these are the docker official images and if we look here.

A

Okay, I don't see and the filter tags.

A

There we go. Thank you. Okay, so don't see it there how about alpine and we knew that album is not yet official. We know it's right. It may not.

B

Ever be so, the docker official images are the ones that are built by the um docker people themselves, right the adopt open, a jk um repository or whatever you want to call. It are the ones built by adoption jdk, so that one right there is built by adopt open jdk. So it's not like it's some third random party, or rather.

C

Personal, they do not have full test coverage being compared to official images so platforms meeting several months ago when we were discussing that uh well, it was brought up as a main concern block and jay of alpine images and, as alex said, there is no guarantee that they will ever be in ge.

B

I mean there's no other source for these images unless we build them ourselves and test them ourselves, which I don't think we want to do.

A

Yes, yeah so like I'm trying to trying to understand that one, because I thought this was the image that we, this is the image that we need to use in order to do alpine in order to so so. Therefore, I think we are accepting that we'll use this.

C

Yeah yeah there is even um to do in the pull request. You reviewed this pull request to do successfully references it as experimental ones right and well. We couldn't do anything better and, uh let's be honest, I'm not sure what is better partial experimental test coverage, but by adopting ngk a conglomerate of multiple engineers and contributors working on the baseline or in-house builds by docker, for which we have already experienced quite a number of problems.

A

Yeah, my preference is the just yeah. My preference is to prefer what what adopt open jdk, even though they call it unofficial. I think it better meets our need.

C

Yeah well, anyway, the release is going out, so we agreed I'm shipping them, but a person. I don't expect degradation and test coverage and, if you're concerned about test coverage, we should rather focus on our own test coverage, because we still have on the smoke test for the images right.

A

Okay, so back to the buster question it was. We think that buster is in this nope, don't see anything there so stretch.

B

Okay, so they don't have a stretch image for sure, okay, all right, so they don't specifically tag the uh debian. Oh maybe it's debian, but it's oh! Oh, should have thought of that right. It's probably not specific yeah.

A

So they don't call out that it's buster, but but that that then indicates that we would be if we choose to use, adopt open jdk as our base image, we would take whatever debian they're running whatever debian is in their image, because we would be based on their image.

B

And the other thing we could do is propose or add additional tags to the upstream. I've done that for windows to request a specific tag: port buster.

A

They don't do a stretch image at all and yeah and I I don't think, there's any compelling reason for us to ask them for additional upstream tags.

A

Let's just take what they've taken so so what we're really doing is is this. Is it's saying that it's on buster and that that's likely accurate, because you said they don't do stretch but based on the adopt image, adopt open, jdk, jdk8 image.

A

And it's bundled debian version right I mean it's bundling some debian version and it's whatever that is so in our example. Here it might be.

A

Let's see, let's take one.

A

Yeah so, for instance, if we took jre 8u 2665 b01 debian as an.

A

Example- and this would be the thing that would be debian basis,.

A

A

A

So really the reference to debian is just not relevant.

A

And now on to the question of centos, we are delivering a centos image for sento7.

A

A similar similar pattern, I assume that they tell us they, they choose a centos version and that's just implicit in the.

A

Yeah here we go so 265 and it's whatever centos version they're bundling at that time.

B

Okay, now we're we're.

A

B

Not pulling from there right now from their image for its sentiment. ah Okay, all right from send to us not from adopt open, jdk, centers.

A

ah Got it okay? So what you're saying is the current build process uses a centos base image and then installs adopt open jdk?

A

Well, it installs this page adk. It just does uh yum install java java develop uh okay, so it's installing a in installs all right, so it sells jdk with yum, with the package manager from the os.

A

And the same is true for our debian images right. The current build process uses a debian base image. No, I don't think so.

A

Oh okay, all right so.

B

Our current debian, it's from open, jdk, a dash jdk that stretch.

A

Okay, so that's that's a distinction between the two build processes, then, so the current debian build current centos build process uses.

A

Centos image, whereas the current debian build process uses dev, uses the open, jdk base image and doesn't therefore need to install a separate, a separate jdk using the package manager.

A

Okay, but the proposal there in terms of the transition to adopt open jdk, is it to consider using adopt open jdk as the base image for the debian build process as well.

B

I think we need to define that because um the one.

A

B

Of using the package manager is you don't know explicitly what java version you're getting um so yeah?

B

I guess it depends on whether we want to use a specific version of java on in our images or if we want to rely on the package manager, we could do the same thing for debian right.

B

We could use a from debian and then use that to install open jdk packages.

B

I don't know if that's better or worse.

A

Yeah, so what it, what it's it at least highlights right now we are: we've got a a disparity between centos, build process, that's based on core operatings on the operating system. Image and the debian process is based on the open, jdk image.

A

The alpine process prior to oleg's changes was based on the alpine image, and so the new alpine image will be based on on adopt open, jdk alpine image right.

B

Yeah, it's apparently he and he put it on a specific version of the jdk.

B

A

It you and it was 262 if I remember right.

A

Okay, all right.

A

And there the question for me: I think that the majority of our installations are using the debian, our debian based image, I'm hesitant there to say, hey, we'll, accept breaking changes there.

A

The number of potential people affected, but if I remember correctly, the even even on our debian image, we're not running a brand new. It's not running a latest version of uh jdk because the debian base doesn't offer us a or the open jdk base doesn't offer us a newer.

B

B

Alex I think, let's see if I look at that uh the debian. Oh, the current debian.

A

Yeah, that was what I was just looking to see. I thought that.

B

A

B

Open jdk, 8 jdk stretch.

A

Okay, all right, so it's it's using there we go yeah, okay,.

A

B

So you don't know what java version.

A

That is or anything right now right, but I think that's one. We should be able to find from docker hub right just by searching for that.

A

Oh okay, I need a open, jdk, slash.

B

You know it's one of the it's one of the um it's, not a organization. Oh, it is not no, it's part of the official docker. uh So we go to the official docker open, open, jdk.

A

Yeah, so it's part of what they call verified content.

A

And then here, if we filter for.

A

Search for the tag right there, we go yeah, so this one, this one is six months out a day. Therefore it can't be current because certainly we they we delivered their. The jdk projects have delivered up two updates in the last six months.

A

Yeah, let me do a quick check here. I think I've got a running installation here.

A

That I can do a quick check on system information that will tell us system information right there.

A

Okay- and this, if I remember correctly, is running the current image yeah, it's 242, so it's not that far out of it. The current release is 262 or 265. So it's it's six months behind. Basically,.

A

um I I that's a good question, I don't know, let's look so the question was: does the official image have anything for buster it it had so we need.

A

Eight, yes, they do and it's much more recent.

A

A

A

So we could switch current debian build to buster.

A

To extend the life, the support life of that of that base, image right.

B

um At least give us a way to pin to buster so well since adopt open. Jdk does not provide a buster specific tag.

A

B

Just just not yet I don't know if we want to have different base images depending on the os. I guess we already do so.

B

A

It seems like that might be less disruptive to users to switch switch. The base image from open jdk stretch to open jdk buster than to switch to constructing it ourselves or to switch to adopt as the base image.

A

Now I thought one of the reasons that we had considered shifting to adopt open jdk for the base image was worry that the debian project's maintenance of open jdk might be less diligent than the adopt open, jdk's maintenance of of the jdk.

A

I thought there had been an episode one episode some years ago, where the debian project wasn't able to keep up with all of the all of the open jdk changes.

C

You mean the event when uh medium 3.6.21 broke, because the issues in debian right.

A

Yeah, I think that was it right. I think it was. It was maven related, but but there haven't been issues since then, so I'm not sure that that's a compelling thing, but there was an episode there was that one episode.

A

B

It really is a jet.

A

B

A

Yeah yeah, that's exactly what I was thinking thanks, you remind very well. What we really need is a is a technical discussion and we have a forum to do technical discussions called the jenkins enhancement proposal process yeah. So yes, wholehearted agreement, and- and that's one of these- this is a an education point for me that our choice of which base image- okay got it. So, yes, that needs to be there.

A

I'm not trying to turn into the best market. No, no we've got truly, we've got it. We've got to make the decision and the decision needs discussion and the jeff is the place to do that.

A

All right are there other things we need to discuss here, other than that a jep's got to be created and we've got to discuss that. I don't think we have any any issues that are burning things down on these images. Currently we know that january 2021 21 is a uh or let's say it differently. Debian bullseye release.

A

Is a major event because it will drop security, support.

A

For our debian base operating system image, do they have um docker images for both idea uh yeah? I think they do. Certainly the the the open, jdk images do not have a bullseye image. As far as I know, let me just prove that, but I would be surprised if they did because bullseye is, is not released.

A

Oops, bullseye yeah, so the the official images from from docker do not have images based on debian. What will be debian.

A

A

Okay, so the crucial action there is action. We need to need a plan, need a plan and completed.

A

Work to switch before end of support, end of security fixes.

B

The other thing we need to think about is a rebuild plan for images, because we use like app to install packages and things like that.

C

B

Need to have a schedule for building images so that we can pick up updates to those packages for security issues as well. So do we rebuild the images every month? Do we pre-built in a reporter.

C

That's right, which imagery is doing drinks, controller or agents.

C

Yeah, I guess the way to go is just uh setup dependable.

C

uh There might be tricky parts. For example, how do we update remote inc and other things, but for base images? Dependable will do the job so.

A

Dependable will change the base image that we're running. I thought alex was also concerned about, um uh so that updates to the versions of items that are that are specif specified by version in the my concern.

B

Is let's say that they, um you know, there's a package that we're installing via app that has a security issue right. That's that's not going to be copied.

C

Maybe we should that security discontent or to our pipeline then.

A

Okay, well, that's a that's, also a very viable thing to consider so security scanning whoops. I.

B

Mean wouldn't that require us to build them on a schedule at least or is there? Are there apps that check things? I I'm not aware.

C

So yeah it depends what uh would be chicken dependencies automatically then, when it discovers dependencies it would be creating pull requests.

C

uh This request will be built by jenkins ci here and if there are related security updates, etc, you would be getting information in the security tap on github. Basically, any maintenance will have access to this information so default. Behavior is that update is public, but the security details are private, though, if you want you can disable a security full request at all and just get security controls.

A

Yeah and, and that would that would, for instance, offer us that would automate as an example that would automate the update from jdk8u 262 to jdk8u27 to conceptually, if that were the number they choked.

A

But oh, like back to the the question that alex observed earlier.

A

If the ssh package inside that debian image had a flaw that dependable won't detect that right that we would need something like security scanning or some other technique to depend.

C

On how security is kind of currently working now for docker, because the bot has two littles, so I'm not sure what would you get now and I'm not sure what would you get tomorrow and nevertheless yeah? If we really want to have a deep securities coming, then we should just at the tool like ensure or whatever in our pipeline.

C

I mean we do scanning just not on a continuous.

C

A

Okay and and that's that's the kind of thing that ensure or another tools like at snake or we'll, do a deep scan. Looking for issues inside the image.

C

Okay and I'll show: how deep is this coming, but the better in the current situation.

A

Okay, all right.

A

Any any other topics we need to note with regard to image rebuild plan.

A

It's yes, if so like, for instance, the the technique you're using with the alpine update is well suited to be used with the pandabar right, because it explicitly names that it enumerates the version number inside it and defendable will offer a new version. Number update.

C

It's not really a question of dependable because dependable, let's say it's additional feature: it's rather a matter of reproducible builds so well, I'm still using attack, uh but at least it provides some uncertainty. What you said the image, because otherwise you may easily end up with a situation that you cut an lcs release.

C

Then there is a java release or whatever release happening in parallel, so that you ship one of the just one version of java. Another version of use, another java- or you might be a shipping conversion which wasn't tested by your continuous integration environment and you discovered it on the one you cut the release because release pickup picked up another version from the same tag so in the current state, our official packaging is exposed to all these issues. Pinion conversion is what I would highly recommend for any use case of docker.

C

And yes, I do have strong opinions about that and dependable is not the main reason. I see. Okay, all.

A

Right so it's it's more about build repeatability, uh your preference towards explicitly enumerating the version numbers inside the component inside the the from line of the docker image.

A

Okay, all right any other items we need to discuss there.

A

A

All right alex anything we're we're nearing the end of our time. Anything you wanted to report on proposed deprecation of the stretch. Images. Is this the same thing due to yeah stretch, retirement or stretch end of life.

B

We actually already have um buster images for the agents, whereas we it looks like we don't for the controller, so it would just be in this case not publishing them anymore.

A

Is there a? Is there anything we can or should do to alert alert people that this thing is reaching end of life? I'm not sure how we would communicate to consumers of the.

A

A

Of the the docker agent stretch, images right, I mean they've captured an image. There's there's not really a way for us to tell them. Hey we're not going to deliver new versions of this.

C

There is a way for us to do that. We have entry point which includes a script and we can easily add a warning to this script so that every time you launch this entry point you'll get a warning and uh console that this image is deprecated.

C

Yeah, it won't cover 100 of use cases, but it will definitely cover the most common use cases for that good yeah. If you want uh to go further, you can do the same, for example uh on the remoting side, so that it also appears in the build blocks, but for build locks. You will really need to tweak remote ink uh well unless docker plug you know, kubernetes plugin recognizes messages.

A

Oh, oh, I see what you're saying what you're saying is: if we taught remoting to detect that warning and raise it to the user, I'm not sure. Could you describe further what you mean there? It's.

C

Even easier, for example, you're at java system property and in remoting, you can just read the system property and display it in the system log, so you don't need to invent any new mechanism.

A

Okay, so so that was, that would would need a add, a property to the agent image, and we would need to extend remoting to show that or.

C

Yeah, so if well, if you want to provide good user experience, then yes, you would need to change remoting. I see okay. uh Well, there might be other approaches, but yeah. It's not exactly a huge patch got it. Okay and this patch can be actually useful for many other use cases. So I would be definitely willing to support.

A

A

A

A

Anything else on deprecating the docker agent stretch.

A

A

Okay, next topic we had listed was github app authentication on ci.jenkins.io. Oh, like I think there, the report is it's in use, it's uh being used and working.

C

Yeah so yeah, definitely we could do a longer demo, but well, I don't think it makes much sense, because next week we will have a presentation by casual, including sergent and sao so just uh join online youtube, if you're interested to see details how it works on little plugins.

C

uh But yeah the agency was switched so now. We've got a higher epi rate limit and also pipeline libraries, etc. Now use github checks, api for reporting, as well as multi-branch pipeline.

C

So there are some issues we discovered, for example, duplicated reporting for javadoc and other things, so we likely need an update of github checks, api plugins, but overall it seems to be working.

A

Excellent okay.

A

Last item using plugin installation manager and docker images, so this is in preview preview from uh tim jacobs, pull request uh have: has the image been released with that.

C

Preview in it, no, we agreed to delete until uh there is a security release. I think in the current situation.

C

Well, I'm not sure whether we agreed or not, but I think we need to deal with it a bit more, so we firstly should adopt gdp changes and after that we shipped the preview, though I would be interested to have it in the next weekend and well. I believe that we will have it in the next weekend right.

A

Okay, well so yeah when, when you say weekly, oh, oh right, weekly, the docker image based on the weekly. Is that what you mean so so, uh which is 2.253, will be the next weekly.

A

Okay, great anything else we need to put on the list for today. Oleg do you want to give us a current status on the alpine release process? You've been running okay.

C

So well, quick update. We have a release of docker agent base image so I'll paste. The link to the chat but yeah right now, dr hub, should be building this stuff as well as trusted ci same for jenkins controller base image.

C

Also it's building, although you still need to see what it will be resolved for that, um but yeah the image is running so at least latest versions should be updated, um I'm not sure about text, but this is something we want to verify. You know quite fine, then, for sage, build agents, I'm still waiting for puller requests to complete because alex updated faster before that it was filling on windows.

C

So we need to pick up a fix. Then we spend the release or maybe do a blind release with valencia, but yeah right now. It's still running and yeah for inbound engine and for gnlp agents representatories. We still need to do the stuff so optimistically. By the end of the day, everything will be released, um but yeah. It's not something like. I will finish it in one hour.

C

A

That's thank you very much. Thank you for your heroism on it. Thanks thanks all right. I think that ends our topics, I'm going to go ahead and close the session anything else before we end.

A

Oh thanks mark thanks.

C

Everybody thanks all geez.