Jenkins Google Summer of Code Office Hours, 30 Aug 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: GSoC 2020 - Custom Jenkins Distribution Build Service project demo

Description

Presentation by Sladyn Nunes, GSoC 2020 student in the project. The main idea behind the project is to build a customizable jenkins distribution service that could be used to build tailor-made jenkins distributions. The service would provide users with a simple interface to customize the configuration, they want to build the instance with eg: plugins,jenkins version, docker image etc. Furthermore it would include a section for sharing community created distros so that users can find and download already built jenkins war/configuration files to use out of the box.

Talk abstract: We would be going through how to add plugins to the configuration and the various package configuration details that need to be entered so that the service can easily generate a packager-config.yml, essentially the first step of generating a fully usable jenkins.war, we would then go through the process of downloading a war file as well as taking a look at how one would go about accessing the community configurations.

References:
* Project page: https://www.jenkins.io/projects/gsoc/2020/projects/custom-jenkins-distribution-build-service
* Main repository: https://github.com/jenkinsci/custom-distribution-service

A

Thank you for feedback thanks again for the presentation, so the last presentation uh for today sladen will present uh carton jenkins distribution, build service, yeah I'll share my screen. um Okay, can you hear me, and can you see my screen?

A

Yes, that is great.

A

Let's begin, I'm not sure, let's go okay, let's begin so yeah, um so my presentation is going to be about the custom distribution uh build service, so this project was basically um basically thought out as a replacement um for having to you know, build all of the jenkins um jenkins war files using using the custom using the wall um by downloading the var, and then you know beginning all of your work.

A

So let me just introduce my mentors first, my christian and martin and oleg was a technical advisor in the project, helping out on all technical issues.

A

So yeah, what's this project all about, so we try to address major four features. One of them was having an out of the box usable jenkins distribution, um a simple user interface so for your users to be able to configure the plugin the plugins right out of the box by making it very very simple. So, as you can see, it's it's supposed to be very uh user friendly to use community shared configurations. This was one of the most demanded features on the thread that was that only originally conceived the project uh there were.

A

There was a lot of demand for having configurations that are being widely used around the community, so be it aws or be it uh be. The kubernetes plug-in uh users wanted to have a community shared configurations page where they could share uh where they could. You re use um configurations that were developed. um You know by people around the world and the last one was easy access. So sometimes there have been complaints that uh it's difficult to access the plug-in the update center.

A

um This um service tries to circumvent that problem and tries to add easy access to be able to. You know um you need to add your custom. um Maybe a custom update center that you can. You know download uh plugins from you can view plugins from and so on. So that's major four features that we're targeting um yeah. What was the current problem? I'll not spend too much time on this, because I've already explained most of it. um For many users they download jenkins, first and after downloading, that they select plugins and configure them.

A

So this might take a lot of time and we try to reduce that time using the custom distribution build service that contains all you need on a web application online and you can easily do it without in the comfort of your browser.

A

Okay, what was the project motivation? um So we work hard, so you don't have to um so we do all of the heavy lifting and you just have to download the var and you can run it right out of the box with all of your plugins configured very, very easily okay, so I just run through the major features that were implemented. These are a collection over all of the three phases, so we started out with the bare bones: the package configured config generation.

A

uh You need the package config generation, it's a vml file and this yml file is then used by the custom wire packager to package it into a war file. So you needed that, so we went ahead and you did that in the first phase, then came the war generation sequentially uh after you had the package configuration. You could then generate the wall. The third one was to able to uh ability to configure custom packages.

A

um So after that we added the features to be able to not only generate custom packages but also be able to edit them live on the editor, and then you can make changes right out of the box. So if you say you downloaded a particular jenkins version, you're not happy with it, you can go ahead and change the junkies version into it down. It will regenerate the watt file for you.

A

uh The community configuration as requested one of the most uh requested features so having having uh community configurations where you can use blue ocean, or maybe aws kubernetes, and so on so forth. uh The next one was the ability to download var so update up under phase two. You only had the generation. uh We then supported. We then supported downloads. uh So you could you could click a button and uh download the var file right out of the user interface, the ability to search and download community configurations.

A

So, after adding support for community configurations, we added the support to then search and download them. You also have, if I'm not mistaken, you can add your own repository url. So if you're not comfortable with using the jenkins community configuration url, you are free to use your own url, uh the ability to share community configurations again, that's an extension of shared community configurations. You can yourself contribute to the immense repository. That's there.

A

There are a couple of repositories. There is one by even rick, which I will demo later on in the project.

A

He he maintains something called as jenkins formulas, and uh they are a collection of ready uh in usable jenkins uh distributions and self-hosting capability. This was one that we weren't quite sure about in the middle of the phase, uh because we we needed permissions to host it on jenkins. uh So we decided to add this feature anyway, so it is now self-hosted as as well I mean it has the ability to be self-hosted. So if you want, you can spin up by just two commands.

A

You know just you to have docker on your system and you can with two commands. You have the entire service up and running in minutes. So yeah. That's another uh amazing feature that we have okay, uh making the initial jenkins setup simple. So I'll just walk you through the steps that you use for generating. um I hope this is not okay yeah. This is not uh objecting, so you select your favorite plugins that you need.

A

uh You then customize the configuration and after you just relax, because after customizing the configuration the heavy lifting is then done by us, because you just then need to generate the package and if you want you can download the packet or you can proceed with generating the wire file, so the service will take care of generating the jenkins. for you. You just need to download and run okay. Where is this app hosted, so the app is currently hosted on customize.jenkins.io?

A

um I will be uh while I do my demo. I will be taking you out of the slides, um so you can have a look at it later on.

A

Wait. There's more so, you can also download yeah. As I said, the previous slide, as I was going to take you forward, the jenkins formulas is a repository that you would want to check out. uh It has a it has pre-built configurations for jenkins, like blue ocean and so on. You can. You can definitely use them out of the box and you can put those urls into the custom distribution, self-hosted version. If I told you want to run it self-hosted, oh time for some demos, uh okay, let's, let's go out um okay yeah!

A

So, as you can see, this is the plugin page. um We have all of the plugins that we pull in from the update center, the jenkins update center and over 1700 plugins. uh One major challenge of this particular implementation was: how was this was taking too long to load sometimes pulling in 1700 plugins. So we have caching support for that now. Fortunately, and uh yes, so you so, the steps are so I'm just going to be adding a few uh plugins uh you can.

A

You can just choose them if you want, you can even sort them out so mostly installed, uh trending and so on relevant. According to your liking, so if you want by trending, you can just sorry so yeah um I'll, just select a couple of them.

A

The next thing that you, after selecting your plugins you do, is you submit your plugin and on submission? You will get a model and on this mode you can enter all of your details. So you have the jenkins online meetup. You enter up a jenkins version.

A

You enter the artifact id, so if you whatever you want to name it suppose, if you're developing a one where it's a sort of package, you can rename it as per so. This is for demo purposes.

A

Okay, furthermore, if you wanted a docker build, you can include a docker tag. um Currently there are a few issues. There's a disclaimer. There are a few issues with the docker uh building inside, because this runs in a docker container, and then we have a docker in docker, that's a bit of a disclaimer, um so you can include the build tag, though, because we wanted that feature to be supported.

A

You click generate package, so yeah. As soon as you hit generate package configuration uh the service generates a package for you all of your plugins come right in as you selected um previously and after you select all of these things you can, then you have two options, so you can either download the wire file or you can download the package or configuration. So if I just download the package configuration as you can see on the left of my screen in my cursor, you will have your yml downloaded.

A

So so you'll ask me: what's the advantage of having a yml, uh so the jenkins custom water packager currently takes this file and generates uh generates a wall package for you. So if you are not, um if you do not want to use the service- and you want to use your custom packaging, you can use that as well, because this takes in because the custom wire package takes this as an input so yeah uh just before I demo the wire file.

A

um I would like to come to the community configurations because that's where um that's, where the minimal working configuration of you know of jenkins is hosted, and we have a value jenkins version that so yeah. So we have two versions uh this. These two ymls are currently hosted right here. So if you see yeah custom jenkins distribution service community configuration, this repository will down the line be hosted by jenkins uh on there on the repository page um on the github organization, I'm so sorry um where you will then find all of the configurations.

A

So whatever configurations you can see here, those are the same configurations you can see. So that's what's essentially happening as in when we expand these community configurations list. You can add, you will find more and more community configurations listed out here, so they will be named aws, azure and so on. So forth, blue ocean, and and so on, okay uh yeah, so after you so after you have them all displayed in this page, you can definitely choose to change the version during sorry, the url that you would like to point this community configuration to.

A

So, if you're not happy with custom distribution service, community configuration, that's not a problem, you can enter your own url um for your, where your repository has been hosted for community configurations yeah. So if you click to view details yeah, so you get to view details, you see one of the um one of the sample configurations that I had set up, so it generally loads the entire configuration and you can then uh click on download wi-fi. So I would like to show you so yeah when you click on wi-fi. What happens?

A

Is uh the custom wire package on in the background, runs the entire um generation of of the wi-fi, so I'll just be I'll, just be waiting for a few seconds around. It takes around 20 25 seconds to to bundle all of the plug-ins.

A

So while we, while we wait for that, um the wall packager built by oleg built and maintained by oleg, is one of the heavy lifters of the custom distribution service and it uses it relies on the custom wall packager to package and bundle. um One of the current one of the features that we plan to include here is the ability is for more um for more robust errors.

A

So currently, if something goes wrong, you might not get a very good error message on the on the screen and that's that's something that we've been trying to improve in the future. So yeah, as you can see, the wire file is being.

A

Built you think it should take a few more seconds yeah there. It is so yeah as soon as the bar file completes building it. It happens in a um it happens in a temporary folder. So as soon as you, you have the wall file, it returns it. So, as you can see, there are almost I've. I've been generating all files for a long time, so there are 13 of them, but yeah. This is the one file that you can just run with jenkins jar jenkins war.

A

That's that's the command, and then you can spin up the jenkins war file with all of the with all of the plugins pre-bundled. So whatever you have selected here, configuration is code, um ci, matrix, auth, so on. All of them will be just bundled in that one right on the box, so you don't have to leave your browser. um Yeah. That's, uh I think, is that is it for all of the all of the demonstration of all of the features. One thing I would like to show you is the repository.

A

uh The repository is self-hosted, so you can use just docker compose, build and docker compose up and you'll have the service running up in seconds. uh Apart from that, if you do want to add certain configurations, we have the environment files supported. uh That is a pull request, still in development, so it might it might, it might come, um might come soon, uh where use where you can change your you environment variables, inside of environment files. So yes, that's. uh That was one of the one of the features that that is coming uh last.

A

I wanted to talk about.

A

How can you participate so we have our github distribution service as uh the service posted on the repository. um We also have a gita chat where you can ping in or send in your queries, feedback, questions and so on, and we also have a project link, so the project link has been updated with all of the phases all of the development that was done throughout all of the phases.

A

We have also the demo links of the previous phases there uh what's next, so I wanted to talk about what's next, because the plugin has a lot of the the service. Has a lot of potential uh and it is, there are a lot of features to come. I think three months to be honest, was less to implement a plethora of features that we tried to put in, but yeah there are still that need to go in.

A

uh So if you don't mind, I can take you a bit back to the roadmap and, as you can see, there are a lot of um issues here, but uh where oh yeah yeah, let's roll now so one of the things yeah one of the things for the road map uh is it's a prototype for the jks creation. So jenkins configuration code is very popular among the community and we would like to support that as well.

A

So if you see this page right here, you just have you just have your bundles and stuff, but if you, if you want to include something like a jks section, um we would like to support that as well, where you just put in your jks, and you have your jk yml as a as a as a support. So you can definitely include that as well here and then you will have not only your bundles packages and plugins, but you'll also have your jks yml supported. So you can.

A

You can just start your jenkins work right right out of the box with a few clicks. That is one of those most uh features for the road map, one most important one. uh Apart from that, we wanted to support running the service on windows. That's a feature that is that that might be coming way way down the line. uh Another thing that we want to support is pull request creation, so currently you can only you can only view them.

A

So what if I create a new configuration here- and I want to create a new pull request or I want to share it with the community- that's a feature that we've been planning for the roadmap ahead, uh so that you can just create a pull request using maybe github bots like ci or jenkins.io, has a github application. Maybe we can leverage of that to create pull requests inside repositories?

A

That's one of the features for the roadmap and uh another one was um yeah. I don't. This is uh unable to that's, not a roadmap, I'm sorry just taking over that for a minute and uh explore synergy with image controller. So this is still under development and discussion. uh We were like. We would like to take a look at this later on.

A

uh This basically deals with the kubernetes operator and the sort of images that red hat has on their repository list, and maybe we can support creation of images uh right out of the box so that you can not only now generate a war file, but you can also generate an image so yeah. That's that's! That's for the roadmap um and yeah! That's that's about it for my presentation uh for my demo. Lastly, just before I finish, I would like to thank my mentors kristen and martin.

A

They have been super super helpful throughout the entire gsoc phase, if they've helped me in almost all of the all of the holes that I've dug myself into um and oleg as well on occasions to to help me with all of the technical stuff um yeah. So I would like to thank them and I really really appreciate it uh for giving the time out during this covet pandemic. It's not easy, uh but yeah we've, we've kind of managed it and I'm happy to show you that it's also been hosted on on on its own domain.

A

So I've completely forgot about this, so it's it's also hosted on customize.jenkins.io. uh However, there are bugs here. uh This is not.

B

Fully functional, as of now there are you you might receive some 404 headers, uh but yeah. It is also hosted on customized or jenkins or io. So, if you want to play around, uh we need to still.

C

Patch up a few things, but uh we do have an alpha version deployed um on it. So yeah, I think the project has been has been, has been really really really great, so yeah, that's it! I'm handing over to mike back to like sorry I'll, stop sharing.

C

C

C

Okay leg: okay, maybe your leg is um not oh he's back! No, I'm! Actually! Here I was just trying to unblock myself um yeah and so thanks a lot uh for the demo, I'm glad to see that it's working yeah there is a lot of stuff here and there to complete so yeah, probably the.

B

Scope was too big, but yeah started so many stories. Many of them seem to be finished now, but still it's a great proof of concept. Yeah custom jenkins for packages as a service.

B

Hopefully it can be expanded from that to support more and more user features and yeah. I'm looking forward to see what will be user feedback there. So thanks for doing that, I think it's a quite important step for the jnk system.

B

So I'd like to thank uh sladen for his work on on on creating this this web app, and one of the thing that I really like about it is I can I can I can. I can use it on my local host. So if I'm working on configurations that cannot be shared with the world.

D

I can run this entirely locally so that that was one of um that was one of the features I guess, which was, which was important, is the ability to run sort of behind a firewall.

D

So if you don't have to use the hosted one, if you have some private configurations that you need to be working on, just wanted to enhance that capability.

D

D

Right- and it's been great like it's been a great project so like this summer, has been very good and it's been really great to see the pro the tool be built and to see like it kind of develop and go through different phases and see the ui and knowing it's essentially like two different applications running so slated had to create like two different things and like doing a whole front end and a back end.

D

So it's really interesting to see the tool come together and it's really amazing to see it posted in kind of its outperform. So congrats you did a great job this summer.

D

So now that's all these the presentations uh for today um but yeah. We will still have four presentations tomorrow same time to bmutc and everyone is welcome to join and tomorrow we will also have presentation for the janky sex project. So if you're interested in junkies, please join as well.

D

For now, I'm planning to stop the recording. But again, if you're interested to have informal discussion, please stay on the call, so we'll have some time to basically discuss any topics related to google summer of code and to the presented projects thanks again to all the students, mentors orchid means and other contributors who make these presentations possible and thanks to everyone who watched the demos, we will appreciate if you try out the projects and provide us some feedback so that we can program going forward.

D

Thanks all thank.

D