Jenkins Platform SIG, 7 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Platform SIG 2019 11 07

Description

Jenkins platform special interest group meeting. Topics include AdoptOpenJDK (hotspot), AdoptOpenJDK OpenJ9, Plugin installation manager, Windows installer, and alternatives to broaden hardware support from the Jenkins docker image

A

Okay, welcome everyone. This is the the Jenkins platform special interest group I'm, going to start sharing my screen and let's look at our agenda. First, we're going to review the review the agenda. The topics on the agenda be sure that we've got all the topics and then we'll work through the topics.

A

So you should see my screen with at the top the November 7th agenda and, whilst we'll spend some time reviewing open action items, then we'd like to talk about adopt open JDK as the base JDK for Jenkins image images instead of open JDK.

A

Adding adopt open, JDK j9 as a new image for Jenkins, then we'll talk status on the windows installer. We had a plan to talk about Octoberfest results, but Oleg is out today on vacation. So we'll defer this in two weeks and then Natasha added the plugin installation manager tool. As a topic for this agenda, are there any other agenda items that people would like to add before we start working through the agenda.

A

Okay, I'll take silence to mean mean that we can proceed then. So, let's look at open action. Items on the mark is embarrassed. We've got I still have to open the James Jenkins enhancement proposal for doctor operating system. Support and platform selection rules. The issue there is. We really need as a project a set of well-defined criteria.

A

We use to decide when we will add a new docker image in particular because they are sticky once we add them, they tend to remain forever and as an example, the alpine non support with Java 11 has been an interesting one with a sticky image.

A

Oleg has the action item to open a jet for Windows support policy.

A

Alex you've got the action item on the windows installer code, signing with only via their name. Anything you want to report there.

B

The ticket from the CDF is moving forward. They they created a new legal entity in order to have an entity, they can purchase the code signing certificates under. So that's where we're at they have established that LLC and now we're just waiting to see what the process will be for getting the code signing done.

A

Great yeah- that's, that is, that is wonderful progress there. That was the sticking point for us getting that code signing certificate and, as noted here, we also need that for the Jenkins release, automation project, that Olivier is leading. Thank you alright. So in turn next item adopt openjdk as the base jdk for jenkins images. So the the thinking here was that adopt has better platform. Support, has has active platform support for some interesting platforms that we might need, and the question was: what do we need to do next? So we need an integrate.

A

We will fundamentally, we need a pull request, which proposes this right and I have not seen one yet to propose change that change from a dot from openjdk to adopt open JDK. Is there anybody on the call who is intending to take the lead on creating that pull request? Are.

B

We talking across the board or are we were just starting with the master images, because they're also the doctor slave, the doctor, jnlp slave and the doctor SH slave there. But just we provide as well.

A

That's it that's a good question, so it would need pull requests. You said for master for the doctor slave an LP slave and isn't there an image that is the basis for those two. That's docker slave, oh, okay! So then it's ss8. What's what's the third dollar ssh slave? Thank you, doctor, ssh slave, all, right! So there's it's not singular, pull request! It's multiple, we'll! Even several pull requests. I.

C

Think mark for right now, due to adopts limited support of like base images right now. At least you know we talked about in the last call that I'm working with the adopt team to update their testing, so they can do a PR to the official docker library to add support for a lot more. You know, bases like Santa, ass, Debian stretch, foster and all that good stuff right now, I think they only have Ubuntu 1804 and I'm blanking on the other one, maybe ubi I think they have official ones. I.

C

Think starting off with a small PR would be a good idea, and you like a proof of concept, so maybe focusing on that master and seeing how we get you know some reception and see how it goes be. A good idea: okay, well.

A

And so my thought was that as I've looked at the Jenkins docker image, it seems to have a base operating system. Oh no, that's right! It does use a it uses, something provided by open, JDK, good, okay, so Jim. Your point is that since adopt doesn't have some of those other platforms, that would be a really significant change. Thanks, good point: okay, yeah.

C

And that's that so that was my like first kind of idea was just target one of the bases they do have I know they have and Jordan I've been using the Ubuntu 1804 base for adopt on us 490 and we haven't had any issues. Is they had been actually really really stable and wrecking really well unplug and support, surprisingly hasn't been an issue either. I know that's something we talked about last time like oh well, you know.

C

Do we know of all the plugins, you know work on s3, 90 or zombie, eight bugs, but we hit any and we're using the gans, full docker and a couple other major right. These I think major plugins, right, yeah, but I think I think targeting just the master or just a kind of proof of concept.

C

You know showing hey here would be the here's, how we would do the switch over to a dock. For me, a good idea, then I'm gonna continue putting a little more pressure on adopt team to support and get that PR underway to get the needed base images so that we can, you know, convert all our images over to adopt.

C

B

But by the way, I landed a PR recently with the adopt team for nano server images. I, don't know, I, don't know what they're there, how they're going to roll that into official rather than a non official? No, because the build environment stuff that they have is all linux-based yeah.

B

So I may do another PR to get actions involved because they have docker on Windows agents. Forget hub actions. Oh, it's awesome. I.

C

Didn't know about that I can reach back out to them. I was on a talk, I think the other week with them and I know they just got more Linux server capacity. So that's why they're more open to supporting many more base images, but I didn't hear anything about Windows I know they do windows I, think core, not nano so would be interesting to see they would be able to support. Now.

B

Yeah, it's it's really easy to build the Nano images if you can support Windows images but for my understand, there's someone who is actually like building them manually and pushing them to doctor hub, but.

B

C

An automated version would be better. Okay, it's something I have not touched in my experience with docker working with Windows images, it's always been Linux I'd love to pick your brain a little bit more on that, because whether you do a PR or I kind of prod and internally with a top team, we should be able to get that through.

A

Cool, so that's good Alex, it seems like the windows. Nano server images would be very interesting for the agent, for me, at least for the agent side for the doctor slave or the docker SSH slave yeah.

B

Definitely they are much smaller, like the normal images like to point something gig for a surfer core and then a disturber images like 500 bag. So it is a pretty significant savings in space and download time. We.

A

To be willing to provide, you can do it after the meeting provide a link to the PR so that I could embed it into the meeting. Now it's Alex yeah I'll. Do that. Thank you.

A

They are there other items so Alpine support status, Jim I think you had some information on this one that I didn't capture in the notes. Could you tell us what is adopt OpenJDK x' position with regard to alpine? I think you described at last meeting and I didn't capture it. Yeah.

C

Sue the digging through and III, maybe I'll get the links for you guys to digging through the last official P or the open, JDK open, again stocker library, which doctor library, for those who don't know, is the official way to add official images to docker hub.

C

They had Alpine supported and if you look at their unofficial images, there's a lot of Alpine images out there, but the official docker library staff was a little concerned with how they were implementing Alpine. Since Alpine uses muscle versus live GC, you know pretty much. Everyone uses lib GC on you know their base images except Alpine, and the problem was the dive team. Paulie did was disabled muscle and then pull down live GC and put it on top at Alpine, and then just you know make adopt.

C

You know work on top of that, so they were a little concerned about having that happen they were. They were looking more for official support of muscle with adopt on top so I, don't I talked to the devs for the adopt team and they look like they're gonna, try Alpine again in the official PR and I'm hoping it goes through because they I think they've done a lot more testing a lot more vetting.

C

But my concern is that the official docker library staff are still no say no on that which would be a big loss, at least for I like out pine. At least I know some of the community lakes at mine too, so they adopt team definitely wants to support it, but I don't know whether it would be blocked by the official docker library staff. So, okay.

B

Are there OpenJDK images for alpine I thought, I saw them yeah.

C

There are but the open OpenJDK, but how are they implementing? It is my question. Oh my.

A

It's a good question: I.

C

A

Recollection was that there isn't an open, JDK image for for an alpine openjdk image for Java 8, but if I recall correctly, not for Java 11 and that the open JDK project bluntly declared that they would not support muscle. That was my my that's the poor. So much for recollection I'll, double-check I can do some I think that that that open JDK refused to support al fine with Java, 11 and I.

A

Think it's a long thread on it, but it does support Java 8, it's just if it only supports Java, 8 and Java 8 is approaching an eventual end of life. It feels like Java 11. Non support makes Alpine at least to me less interesting, yeah.

C

Devlin does getting to Java 11 you'd, be really nice for everything, but al exactly bring up a good point if adopt got alpine through to the official image. Maybe they did it in some way that please, the docker staff they'll be interesting. The kind of daibul deeper into that and see how they actually got is supported, at least from my understanding was that the docker library staff didn't want to do it because they thought it was not I. Guess 100%.

C

Okay, with you know, switching out the C compiler on top, but I mean if open JDK you got through, then I don't see why I doct wouldn't be able to.

A

Write don't yet have pull requests on it, so I'm going to delete those two items, any other items with regard to adopt OpenJDK as the base JDK.

A

Okay, let's talk about j9, okay, so Jim. Could you give us just remind everybody? What is open j9 and you can keep it short and simple, but tell us what open j9 is and then we'll get into the details. So.

C

Open j9 is a kind of different implementation of hotspot. You know what hotspot being more of oracles implementation j9 has a different approach. It was originally I think made by IBM, but then since open source state. So the advantages at least from what I've seen is a huge, huge ramp up a huge speed boost and ramp up so open j9 has a much faster time, ramping up to against full capacity or full for peak performance.

C

If you want to say that, and then we had a win on the call last time and he's talking about how hot spot usually edges out, I'm sorry, open, xenon, usually edges out hot spot in performance, there's just a couple of edge cases where hot spot you know you might you might want to go with hot spot. For you know certain use cases I figured what he said: I'm blanking here, mark I, don't know if you know.

A

I I, don't remember.

C

A

You gave a great summary: that's wonderful! So thank you very much and we've got a pull request. Open we've actually got to pull requests open, I. Think the top one listed here 890 is more active right now and what we need right now, as far as I can tell is more review and feedback.

A

One of the things I thought of is I, think I'm gonna I'm at a good point personally, where I could take the pull request and apply it to my tooling and see what breaks my tooling involves: Windows agents, many different linux agents all hosted off of a docker image on on linux, and so with that tooling. That should give us a little bit more evaluation: hey. What are there any surprises and running open j9.

B

My PR for adopt for nano server images I did find that the open j9 crashed when it was when I was trying to run a simple like Java exe, be her version. I haven't reported it to anyone because I just have time yet, but that shouldn't impact for master, because we don't Windows master images, but just an FYI. Okay.

A

So and I'm expecting I was expecting all sorts of problems just related to surprises around garbage collection or around the startup sequencing, because Jenkins is huge right, there's an awful lot of Java code in Jenkins. So so for me that I assume we're going to see some interesting failures as we run it at larger scale.

C

Yeah mark I would love to hear about that. It's you guys do run into issues anxiety, I can definitely you know kind of axed. You know ask the internal, open, j9 team and see what they're doing I know they do a lot of stress tests. That's actually part of the test, we're actually pulling in for the adopt images we're actually running through our the open, j9 test suite, because you know we can use it for non open.

C

J9 things to just bunch of Java tests so be interesting to see the results over there, but from Jordans AI implementation of it. It was I mentioned this last time to mark, but it was kind of funny we we spent up hot spot and that's 390, and it took like a good amount of time for the Jenkins master to come up and get you know get to that web. The web UI, but when I did with open j9, it was instantaneous which was really nice to see an interesting to see.

C

I actually thought I did the open, j9 first and thought that was like the normal, gia Jenkins kind of startup time and then I did a hotspot. Did it break something you know? Is it not working on s/390 but I just took a little time with hotspot, which is interesting to see.

A

Excellent. Thank you thanks for thanks for doing the end. Is that that your notes on it running on open j9? Could you post those also to that poem request that 890 pull requests? Are you using that docker image as your test on its 390.

C

I, don't think I am well, I did and there's probably some performance. Tweaks I noticed they had a couple Java Flags. What Jordan I did was we just actually pulled down the the master image for Jenkins and then just swapped out the from up top. So we were probably running open. You know running on open j9, but we're probably not optimized for anything like that. So, oh.

A

C

It'd be even more interesting to see if we can find tuning you more I'm, not a Java wizard by any means. I'll leave that to the open, j19 great.

A

Well, yet because I know that 890 definitely has some. What looked to me like ahead of time, compile or maybe it's not ahead of time, but some caching operations where they invoke something in order to get it pre cached into the docker image. So I think that maybe even more interested in great thanks, Jim yeah.

C

Oh uh ping, that thread and see if we can move along I'll be sure to see on the PR.

A

Well, that's that's very encouraging, then anything else on adopt open, j9.

A

B

Had so Jim you have contact with people internally that work on the open, 319 yeah.

C

I'm over at IBM, Jordan and I, so I, the last call I was on with a adopt team. We pulled in I call people from the adopt internal team and a couple of people from the internal open, j19 yeah.

B

Is there a place to file issues online.

C

For open, j9 or adopt open.

B

J9I for the Veneto server crash that I saw I was just wondering if I could report that somewhere.

C

Having bugs I haven't looked around on the github I was being mostly on just a job. I can.

A

Take a look back.

C

To you, yeah, okay, cool thanks, Oh.

A

Jim I know should item for you too excuse me find a location to report bugs to report yeah open j9 bugs yeah, oh dude, great thanks any other topics on open jade, on open JDK or adopt open JDK open j9 as an additional image for.

A

All right next topic, then Windows installer, Alex.

B

So the windows the saw is currently being built on the release instance that le is set up so that flow is working. We just need the code sighting certificate at this point and then we can start releasing so that'll, be nice.

A

So really the the parts and pieces that you need, except code signing are all functional could good? Can anyone grab those bits and run tests against them if they'd, like I,.

B

Do have a build on CI know drinking said. Io I will send the link to the latest artifact that people can look at.

A

Thank you thanks very much.

A

Anything else there Alex that you needed a report. I know.

B

That's all right now: okay,.

A

The next topic I actually want to talk to hacked over fresh results. Briefly, just to let everybody know we had an amazing hack, Tober fest. The month of October is a sponsored month where digital ocean and github create this promotional event called Oktoberfest, encouraging people to contribute to open source projects we had by the midpoint of the month. We had over 50 new contributors and 50 new pull requests coming in really really great results. Oleg will schedule a retrospective exercise so that we can see what we learned from it.

A

What we need to do better and differently. One of the key things he noted was the amazing power of having newbie, friendly or first-time user friendly issues and in particular having them as github issues more than having them as issues in the Jenkins JIRA, because github has provided some easy facilitation to find friendly issues.

A

It's made me think gee, maybe I want to shift some of the plugins that I maintain to be tracked there to track their issues in github rather than in in the Jenkins JIRA. So there's very interesting results. I suspect he'll be with us in two weeks and we can talk about it then now has Natasha joined us Natasha. Are you with us? Oh yes, there she is I, see her so Natasha stopa has the plug-in installation manager tool. Natasha. You want to give us a status report. Yeah.

D

Sure so, after our conversation two weeks ago, I tried doing like the first official release, but it got like an error and so like it basically I couldn't deploy the I guess like the artifact. So we looked at some of like the permissions like the permissions files and I think there were some like, maybe naming discrepancies, so we fix those and then I tried doing it again yesterday and still I'm having some issues.

D

So I was just gonna see if you guys had any ideas and actually overnight, um like Tim, had replied back to me, but um I don't know I. um So one of the things that we had done over the summer was like. We had renamed one of the jar files after I guess like using maven and I I.

D

Think I'm, like a little bit confused about so I, think that had to do with like the order in which things got sent to artifactory or or something I can't remember exactly why we did the renaming, um but I know that like without it that had caused problems, but um yeah I, don't know so I, don't know if you guys have any ideas or that made any sense.

B

The maven release process is always like a black box to me. I, never know why it's going wrong. So, if we're still having problems, maybe we can try looping in someone like J click, Jessie, okay,.

D

B

Something because he's like an expert at everything like that, so, okay.

D

B

Might be a good one to ask, maybe ask on the dev mailing list okay and see if anybody has any input they're, usually.

A

Usually Daniel, Beck and all like are able to help with help us diagnose those kind of things but yeah. If we had to, we could certainly appeal to Jessie I'm hesitant to pull him in any any earlier than absolutely necessary. Yeah.

B

A

B

That death mailing list, then people you know when when they, if they have cycles, they can respond instead of going directly to them for now. Okay,.

D

A

Natasha, you said Tim and I assume that was Tim Jack, home yeah.

D

Okay: okay, great.

A

Yeah, so the dev mailing list feels like that was the place that I had to go to anyway, when I had really had problems doing release, and in my case the problems were related to to getting the right right bits and pieces into the file system locally and getting the right bits and pieces permission granted on the remote side. Okay,.

D

A

D

Okay, yeah I can do that.

A

Anything else report their Natasha, no.

D

I think that's it.

A

Alright, so I think the our action still stands, then, that the goal of Natasha is to get plugin installation manager window released and.

A

Thanks any other topics that we need to cover today before we conclude our meeting.

C

Mark I had a one thing: I wanted to bring up that. We come briefly mentioned. Last talk was and I thought I was interesting, or at least you I think I thought you thought was interesting was we were talking about reclaiming the official docker image from the docker library, the official Jenkins one and the interesting part I know you guys had some trouble in the past for them in terms of availability and, like you know, quickly, pushing out images when you guys need it.

C

The one thing I think you found interesting was their access, their build pipeline, which has access to pretty much every single arch they support, so whether it's arm power as 390 x86, which could be helpful for us. When we start you know, say we do do the switch to adopt and we get on something that has support for many different architectures.

C

Would it be worthwhile I'll he's trying to reclaim that and utilizing some of their resources and then one of the things we also talked about was kind of modeling. After how adopt does it or adopt has their unofficial repository, which they put out nightly? Builds they push out. You know pretty much anything to in everything to their night unofficial, docker hub repository, but then they push out long-term support major releases to their official one. So we're not swapping the docker library staff, but we're just pushing out our best images over there. I just thought.

C

That was an interesting point and something that doesn't need to happen now or anytime soon, just something to get the ball rolling and get people's thoughts on yes,.

A

So the hardware access thing for me is is interesting. That's that's does. Does the doctor official image you actually have access, for instance, to s/390? Yes,.

C

They do I once tried to ask how they have it and I just needed to trace it down. No, that's a security concern, anything that I'm sure we gave it to them once a long time, but yeah they have access to power in Esther, 90.

B

The one negative thing is: they do not have Windows auto built. Really, yes, yeah and there's a there's, a question in a forum from 2017 that they said it's in the roadmap, but there's been no update for a year and a half yeah. That would.

C

Put a damper on mmm some of the stuff we do, but that's interesting. How does how do official images actually go out then for Windows? If it's not automated Jeansonne main, we have to go. Do it well.

B

I'm pretty sure that, like Windows they'd have their own hub instance right, they don't publish images to docker hub. They had their own servers that you pull from and I for the like for the Maven images. I did recently I set up actions for that and but I'm not sure how other people do it. Yeah.

C

Because looking just looking at adopt right, they have Windows server core on the support up on their official image, so they got through somehow I think.

B

They pushed those manually but I'm, not sure interesting.

C

Yeah, that's something to definitely track down because I know the doctor library staff do a very thorough inspection. I would imagine they want to do a thorough inspection of the windows images, but they don't have something I'd be interesting to where they test it or how they test. It.

A

Excellent, thank you so so Alex did I capture that correctly that, as currently as far as you know, docker official images don't have an auto build, whereas these other architectures, like s/390 arm and power, do but you noted that github, a good hub actions delivers Windows images, can github actions deliver on arm power or s/390 I mean do we have do we do we consider some hybrid or that.

B

I'm not sure I haven't when you specify the OS. You know you have action, I, don't think. There's this place to specify architecture, but I'll have to look again. Okay, the.

C

The other suggestion that we use internally and then a lot of open source communities I've been working with, has been Travis for builds, at least where they support our mmm I, think they support arm. It might be in their beta same thing with power they lease for in beta and then I think just recently, s/390 got into their public beta, so they'll be interesting to see if we could possibly use the Travis public images or public resources to build some of our images. I don't know if they windows support being I, think they do that.

B

Windows they do but I believe it's only Windows, 2016 and so, and most people are because of the huge increase in support and Windows Server 2019 have moved over to 2019 interesting, ok, so it's still road block there. Well.

A

At least for me, I'm very interested in Windows Server 2019 because of the addition of Microsoft's supported, OpenSSH server. It makes managing Windows agents much much simpler for me, thanks to that, OpenSSH server, I've, they've they've, given me much better results thanks to that served a new server, that's added in Windows 10, 1803 and later and in Windows, Server 2019.

C

Alex is the github actions. Is that the new like to get ho just pushed out we're like they have that the new tab up there that the only thing I've seen is like? Let's, you like? Okay, if I do a PR, it can automatically build something. Is that what you talk about when you're talking about actions? Yes,.

B

So I and if you want to look at something I've done, if you look at si Sanchez's maven repo I, recently added a PR for building the windows, images on github actions, okay,.

C

Yeah take a look at next I haven't I heard about github actions, but having done deep, that's deep sorry.

A

So Alex would you be willing, in a future meeting to possibly present a tutorial for us on github actions and what you'd seen what what what your experience has been sure.

B

I, don't know that I'm an expert but as I banged, my head against the wall a few times no.

A

And, and for me the banging your head against the wall makes you an expert by virtue of the banging, so I think I put it on next next agenda. Alex review. If you'd have actions I think Oleg has also done. Some things would be a good good topic to have next time that we meet with you presenting and you and Oleg together, reviewing and discussing sure sounds good. Okay, any other topics we need to cover today.

C

Sorry to keep interested Marc is there I know, I know you mentioned and I think we talked about the last time to getting access to s/390 hardware and power.

C

Also I think we talked about this briefly last time is the Oregon State. Has a power in s/390 Jenkins build server, you can do, but we also can get at least I can get you access to just a raw server for at least s/390. You would have to go trace down some power people, but I've done in the past to get power access that'd be helpful for you guys, as we gonna venture off into the adopt, and you being able to run your test cases and stuff like that. It.

A

May I would hold that at least for two weeks, because I think there enough I think we've got enough action items that will keep many of us occupied the amount that we can for at least two weeks keep that to keep bringing it to us, because it's a good reminder that hey there are compute resources available that we may need to negotiate with the for me until we've been through some of these earlier action items. Yes, ok, s/390!

A

Isn't human isn't as interesting to me yet because I haven't even done the basics on open, j9 I have you know, I things that I think I need to do with with equipment that I already have available that I'm conveniently using and then I'm quite experienced with so when problems occur, I recognize them quickly as problems. The s/390 stuff I would see problems and they would be complete, surprise, I'm, sure, yeah.

C

Okay, I see, we mean.

A

Great thank you.

A

We should include.

A

A

All right: well, thanks everybody. Thank you very, very much for the session today. I propose that we go rats. I may have forgotten to turn on the recording. Well, we'll hope that I have the recording on thanks. Everybody thanks very much for your time. Today. Let's go ahead and end our session. Now I've got to stop. Sharing, hey I did record it thanks. Everybody.