Jenkins Jenkins Contributor Summit - Oct 2021, 18 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Jenkinsfile Runner status and its roadmap, Jenkins Contributor Summit, Oct 2021

Description

Let’s review the current Jenkinsfile Runner status. How is the project doing? What are the obstacles and missing features? What is the roadmap towards the 1.0 release?

Jenkinsfile Runner: https://github.com/jenkinsci/jenkinsfile-runner

Meeting notes: https://docs.google.com/document/d/1QLWXNG23ui-LvQXth3UREzOvLYgTMSZcv-El0H141a4/edit#heading=h.ywqy0v70eb25

A

Into the cloud okay, so I want to spend some time uh to discuss drinks for granted experiences and what would be the roadmap for the project, so I believe that levy has some experience with jinx followers. What about you, martin.

B

No, I don't, I don't have a experience. I tried it long time ago, not recently.

A

There were some updates recently, but in principle it remains the same. It's a tool that takes jenkins file and some configuration executes this jinx file and returns results. So basically it's quite straightforward.

A

There was some revolution in the recent.

A

Months and you can see that there is now the 1.0 beta 30 and I still don't have this 1.0 release but yeah I want to eventually land it.

A

So there are a few items which are currently on the roadmap, so the current state that it works pretty well, there were some packaging tools introduced that actually allowed the packaging to run easily as java application and more efficient way.

A

I still have some items in my backyard, for example leaving quartz, because built-in timing is initialization and other things which would make it even more performant. But currently performance is not. The biggest concern is fast enough. So for me, the biggest problem is what uh user experience we want to deliver before the 1.0 release this one. I would appreciate feedback, especially from you davis, since you use it, so I lined up a few topics which I would like to consid to finalize.

A

So, first of all, I want to have java 17 preview uh for 1.0, maybe even before so that ensure that there is no breaking changes needed. Also. We are closing up interactive, ui capabilities so that you can start up a junkies file runner in the ui mode on demand and browse what happens to them. Then I want to finalize custom word packager. So it's the original tool which was created for jenkins for runner, packaging.

A

I kinda abandoned it for a while, because I created new java based packaging, but this java break packaging is quite complicated, especially for non-java developers and also it doesn't use some tricks which could be used elsewhere.

A

So I want to release custom word packages 2.0, which actually supports new packaging formats for gengxis file, runner, so that you can define jxfile runner as yaml and I'm about simplifying this file so that I will drop group ids everywhere. So they will be on the artifact id. So plugin, ids and record will be much more simple for common cases.

A

Then yeah build time initialization, maybe not for 1.0, actually uh also, I need to finish packaging because currently packaging relies on docker hub and it became a bloody mess which actually just needs to be rewritten so that if the release agency is used or did have actions to be defined, I also want to have support for techtron out of the box and for github action, but have actions may come later because right now, I still cannot use work on that and another topic uh which I already started and finish this jbank integration.

A

So if you're not familiar, je bank is a cli tool which can uh execute java as script, um and it has a lot of enhancements. So probably we could execute jenkins file as crypt using this engine without standalone tools, so that everyone who uses jbank they could use jinx for the runner out of the box.

A

So this is what I have on shortcom roadmap, also abandoning java 8. It's a wide open topic, I'm so tempted to do that. At the same time, there is no particular technical justification to do it right now. The junkies fell around codebase is relatively simple.

A

But if I want to abduct quarkus, I will actually need to abandon java 8, because quarkx 2 0 supports on java 11.. So, and there is no point in using java carcus one one to have temporary support for javi.

A

So it's one of the reasons why I might drop it.

A

So this is what I have on my roadmap, but I do not have much feature-wise because yeah there is quite a long of at least quite a long list of items.

A

um Some items yeah thanks to levy, I actually still need to process all of them, uh some of them they actually addressed, but they would benefit from a better user experience.

A

So, for example, yeah primal library management, it could have be done much easier with jenkins hall runner and the same for other components. So I think that we could optimize it a lot. So that's why I would appreciate feedback so that we can see what needs to be prioritized.

A

Maybe no, what do you think.

A

What will give the top items for you.

C

Yeah, I think I think what the jenkins file runs on. Is it almost three years old now.

A

uh Yeah something like that, but it has long history because uh well the baby gaps in development related to some non-technical reasons.

C

A

Yeah, it's almost three years old.

C

Yeah, I think it was almost three years ago that I found it by accident. um I think the most challenging part is if you are not a if you don't have a java background um or if you're not in intimate with jenkins, it can be hard to set up from scratch.

C

um You know, jenkins is a very customized tool. Everyone's, like version of jenkins, is different because of the plugins and how they bring it up. You know whether they're configuring it with jenkins configuration as code or maybe they're, using custom, groovy scripts, and so in order to use jenkins file runner, you almost always have to customize it um to for you for your liking and as of right now, you know you're going to have to know some java stuff, like you're still talking about the packaging.

C

That's kind of a a java thing you know, setting up the jenkins war is kind of a java thing um and that's where I've always struggled um with getting it to run, so I've never been able to do it on my own. Instead, I find someone else, that's done it and how they did it, and then I kind of modify what they did to work for me and what I think would really um help, and I created an issue.

C

The other night inside of that repo is if there was an example for using the jenkins file runner that was based on the community docker image, because I feel, like a lot of people, use the community docker image.

C

um They already have documentation on how to install plug-ins and set up things like that, and then all the jfr repo would have to do is um probably give you know like a java, a docker file that started with the community image so from jenkins, jenkins, lts and then provide three or four lines to execute to install the jfr binary into that con. You know into that and then have that pre-configured to use the existing installation, if that makes sense.

A

Yeah, it makes total sense.

C

Yeah yeah, I think a lot of people are using that image for production, and so if they could take that production, docker image and put jfr into it very easily, they would have something they could use in ci cd. That's exactly what they're running in production.

A

Yeah, I think, that's totally reasonable and yeah. Basically, junkies file runners started like that. I deleted back to java development flow because I wanted to just optimize speed and size of images, and I've been quite successful with that, I was able to reduce the juxtapod runner size, maybe from 400 megabytes containers to 120 megabytes container by now as minimum, but you're right that now we should circle back and provide better experience.

A

Did you have a chance to take a look at the customer packager.

C

A

C

Extensively at all the documentation pages in that repo- um I I did take a look at it. I don't remember exactly what it says it was a year ago, um but that kind of stuff tend to tends to go over my head.

A

Okay yeah, so it started as a development tool. It has a lot of stuff that is not really related to jenkins fall runner.

A

So at some point I was thinking whether I just dropped it completely and built a new tool, and I ended up basically, instead of a two building proper packaging form which you can extend but yeah. I think that you will still need a separate cli because custom work. Packager is not very good uh for junkies for run resist.

A

uh It has a lot of steps and well that designed for classic war files, so that uh yeah, it seems a bit over complicated uh yeah. For example. Here's example for djing's fall runner uh pekka junk, and it looks like that right now, so a lot of optimizations could be done.

C

And so what's this thing, it's it's bundling the plug-ins into the jenkins war.

A

Actually, more because what it does it bundles uh plugins it bundles uh configurations called groovy hooks. It sets up system properties.

A

Everything is packaged, so yeah, it's redistributable um also it can build components on demand and it can operate libraries so, for example, for junkies war file, it can insert additional libraries like newer version of remote inc, or maybe just a client library like let's say, java agent and whatever, which is not available by default.

A

So it has a lot of edit features but you're, basically just taking everything and building this single image. The most important part is actually here, it's also pre-initialization, so you can define it here as well, either as files or even in the same file, but in the same file you don't get a syntax validation. So in my case I finally opted out to having a separate, initialization and scripts.

C

So is this like a way to define your jenkins as code, but, instead of being a docker file out of it you're getting the actual jenkins binary so to say.

A

Yeah, so it actually can produce docker file as well, but it also can produce jenkins war file why it was done uh historically custom word packages was created for testing purposes, so we have frameworks like jenkins test harness which take work, war file and test plugins against that, and I wanted uh to test features like pluggable storage.

A

So, for example, artifact manager is three plugin when I you run a unit test, for example, for jenkins core but uh add additional plugin and the red configuration so that you can test this pluggable storage behavior with the existing tests you eat.

A

So that's why this custom work package was initially created and it generates war file with all the settings and configurations inside it, which is quite cozy but yeah. It can also use a docket image for you, though, for docker image you can achieve the same without a custom. Word packager.

C

Yeah now that I kind of know more about what it does, it seemed to be extremely useful.

C

um My only feedback would be not only allowing the plugins as yaml, but maybe allowing a second configuration option where you point to some kind of existing, um I'm not even sure the name of the file, but be able to define your your your dependencies in a way that um dependable can modify so whether.

A

That it's supported so, for example, there is aging cell runner, it's simulation of c agent in sayo. You can see there is dependent board and how it's defined. So there is comex ml. uh This paw maximal.

A

You can see it's quite long.

A

It uses jenkins bill of materials which was briefly pitched by today, so you can see that some plugins don't have versions because they actually come from external definition.

A

um And what happens next here that yeah there is also a configuration, is code and then all the packaging.

A

Well now it doesn't even use custom word packager because it uses agency's parent chrome, but before that it was taken from xml with custom word packager. There was an option for that.

A

So sorry, if I confused you, uh let's yeah, I just took a time machine. So the version I had here it takes basically build settings and it includes a palm from which you get plugins and the word file.

A

So you can see that the format is not that obvious, because it was well basically uh created quickly just to get things integrated.

A

That's why I started to do zero to rework it, but I have now finished it, but you can use dependable.

C

Yeah, I think, that's a that's a definitely a big deal um getting to depend upon health and maintaining your jenkins. I mean the jenkins community. Image is great, um but yeah. I I feel like this would be better if, if you were going to have a complete pipeline to deploy your jenkins and especially if you're, both doing it for your jenkins production and your jfr, um you really get you know the advantage using the custom war packager.

A

Yeah so see edging seo runner build 4, so it produces war file and it produces a jinx file runner but yeah. Currently it always packages jinx file, runner into the docker image, though technically it's possible to create a binary, didn't support it in custom word packages, and maybe I will do it later, but let's see right now. I still need to finish all this packaging for newer versions of jenkinsfeld runners.

C

In the spirit of jcs, what what needs the most contributing.

A

Well, currently so custom work packager, maybe makes sense to rewrite it completely because yeah. Currently it's designed for multiple cases, maybe it's better to create just separate genius file, runner, packager and simplify the logic and, if you're, not up to that, just making some incremental changes would be super useful.

A

So, for example, just getting creative group ids, which is straightforward and yeah, maybe adding some just verification flows for newer versions of jinx valve runner and jenkins, because this tool chain was created two years ago and now it's a bit dated.

A

So upgrading everything also migrating to pick a cli. I believe there is a task for that.

A

If not, it should exist because yeah, I think that current cli is also not good.

A

So current cli is very simple, but maybe it makes sense to implement separate commands so that it's easier to manage, maybe even by moving some options from yaml to just click.

A

Okay, so if you're interested to take a look at that, just try uh yeah, it should be operational uh but yeah. Actually, I wanted to to work a lot of the things here.

A

Just have never got to it.

C

Yeah, I'm a bit uh green when it comes to java packaging. I don't know if I could rewrite this from scratch, but if.

A

Yeah, if you don't want to touch it, don't because yeah you can give it to somebody else. There are some other use cases we could do. For example, there is github action which needs to be finalized, so it was created by johannes, nikolai but yeah right now, it's rather stale.

A

So it needs a serious update, but it's generally a nice prototype and it would be nice to adapt it to the new link.

A

Also, there are image, packs and the image box. Well, I was able to just add one image because then, due to again non-technical reasons, I was unable to proceed with this repository, but if you're interested, you can create more image packs and they look like that. So that is, for example, they're just jenkins yaml, which is really simple. It's for configuration is code. There is paw maximal which basically defines the build. You can see that there is a bunch of plugins and again here many things could be optimized and then basically, there is docker file.

A

So creating uh a few extra images for other two chains would be definitely cool.

A

So if you're interested it's one of the options, you can start from.

A

So what I would like to see is grader with something for node js development, something for python.

A

So basically any standard flow you're interested in please feel free to submit an image.

C

Gotcha so an image pack you're, saying kind of like a jenkins or a jfr in particular, and it has just the kind of common plugins for that language are not only.

A

Plugins, because it also includes tools so, for example, for medium what you get you actually get a local installation of maven on your.

A

Image so jinx file runner can run on agents, so it can connect agents as well, uh but the in majority of these cases you can also run locally, so this image is designed for local run. So basically you run on the controller, which is the massive and the pattern for other jenkins use cases. But here it's okay, so here.

A

It embeds not only the plugins but also required tools and configurations for them. So, for example, here there is jenkins yaml each it initializes git, gdk and maven tool, so you get them out of the box.

A

Makes sense yeah yeah, so again a lot needs to be done so easily. I want to have uh junkies yaml integrated this system definition into a single file.

A

Yeah depend on what, of course, I want to have it, because, even with that, it's a nightmare to maintain and yeah. You also need a lot more test. Automation for that, because I still haven't finished. New version of jinx file run of test harness.

A

There is one which was created by evarist and francisco from claudius um it's available, but uh it's based on sh unit 2 and it's a bit heavy because it it needs a docker, etc.

A

So right now I wouldn't say that this test framework is what I would recommend for production use, but yeah it would definitely benefit if somebody takes tower and refreshes it.

C

The other kind of general question I had around the jenkins file runner is, I use it. I tend to execute. I haven't run about 15 to 20 jobs back to back to back, but um even even when I keep the container running um and I'm just calling you know the jfr command line, executable like the jenkins internally is restarting in between runs, and so it's it's doing things like you know the startup initialization and things like that. Is there a way to like put put jfr in like um like a damien mode.

A

Yes, it's possible, so actually one pull request, which makes it fancy spending uh so once uh so romance for border risk from his immediate option to keep zhang's fellow running without hacks like sleep and once it's already, you just get things for runners with web interface. Once you set up a open web ui and wait an exit option also, you can start up in the cli mode and if you start in the cli mode, then you can basically interact with instance like with classic cli.

A

So you can initialize multiple jobs trigger them, including in parallel and collect results.

A

I would just say that the experience the user experience for these cases is not perfect, because yeah jenkins.

A

Designed for parallel jobs usage, but in principle you can do that.

A

I can probably show it said it was designed.

C

For what usage.

A

uh For sequential usage, so when you're on jinx file, runner in console and it just executes one pipeline.

C

Yeah yeah yeah yeah, so I'm going to call it sequentially. um I just don't want to pay the startup tax like if I'm pulling down like configuring, the shared library over and over again um every time.

A

C

A

Well, it's not a huge deal actually time wise, but yes, for that current you can keep it running.

A

And yeah also, I started working on initialization with quarkus so that the initialization time would drop from a few seconds to about one. Second, it's also something on my roadmap.

A

So yeah I want to do many things around there. Let's see how many things I actually will be able to do, but here you know, jinx fall, runner is quite flexible and more and more options could be added.

A

So if you're missing something just submit a github issue like you already did and yeah, I promise that I will process the current backlog and yeah. If you just want to adjust uh execution. It's super straightforward, yeah. There are three levels of execution, but once you get familiarized with the code yeah you just get it trying.

A

So actually yeah. There is a bunch of directories there, but yeah. There are just three companies which mata up bootstrap and set up for common development. uh The rest are the utilities, developer, tools etc and later it will be probably regrouping it into a separate repository, maybe even in a separate github organization, but yeah likely just a separate repository.

A

So any questions comments.

A

So, okay, if you want to try it out, martin any feedback will be appreciated.

A

I think that they will be usually definitely in the coming weeks, because I want to release scale changes and I want to finally unblock docker images, so there will be some changes.

B

I I'm gonna have to try it again. It's been a very.

A

B

Time since I used the jenkins file runner, I don't know.

A

But it's up to you whether when you actually need it, then please try it out and yeah anything will be appreciated.

B

Yeah yeah. Yes, yes,.

A

Okay, so should we close down, then.

C

Yeah, okay, it's been a long day for us. Okay,.

A

I think that we don't host after party because people almost disappeared so yeah.

C

And the last two sessions was the after party.

A

Okay, so then, now after patia and yeah thanks everyone, I will follow up in the mailing list and I will try to publish all the videos tomorrow or on monday, the latest okay yeah. Thank.

C

You thank you bye.

B