Jenkins Google Summer of Code Office Hours, 21 Jul 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: [Jenkins Online Meetup]: GSoC Midterm Status Update & Demos - July 21, 2022

Description

2022 GSoC Contributors have been working hard to improve the following projects. Join them on this call as they present their progress, lessons learned, and demos:

- Plugin Health Scoring System by Dheeraj Singh Jodha
- Jenkinsfile Runner Action for GitHub Actions by Yiming Gong
- Automatic git cache maintenance on the controller by Hrushikesh Rao
- Pipeline Step Documentation Generator Improvements by Vihaan Thora
- Link to all projects: https://www.jenkins.io/projects/gsoc/

Timecodes
0:00 - Intro
3:37 - Jenkinsfile Runner Action for GitHub Actions
18:11 - Plugin Health Scoring System
36:39 - Automatic Git Cache Maintenance on the Controller
52:13 - Pipeline Step Documentation Generator Improvements

A

Okay, recording has started, am I: are you still able to see my screen.

B

Yes, well, you need to put it full screen. This is perfect.

A

Okay, so I can't see the time. Okay, it's actually we're on the hour. So let's get started. Yes,.

B

A

All right, hi, everybody welcome to the jenkins google summer of code 2022 jenkins online meetup. This is our coding phase. One midterm demo and um status update from g stock contributors.

A

About jenkins online meetup, so this is a virtual meetup group we're and it's intended for users and developers around the world. The aim of this group is to conduct regular online webinars um about all things jenkins, so we are always looking for jenkins stories, how to's best practices and, if you're interested in being a speaker, please reach out to us on the advocacy and outreach getter channel.

A

If you have questions throughout today's webinar, please put it in the chat window. Our speakers will get to it after their presentation or if it can be, our mentors will be online to help answer the question uh during the presentation as well.

A

You can also find us on gitter and jenkins discourse after this event, and I also like to highlight that the code of conduct applies here. So if there's any questions about it, it's basically being nice and being respectful to one another on the agenda, we will quickly go over what gsoc is and what jenkins in gsoc is and then we'll go straight right into the project, demos and presentation by our gsoc contributors.

A

So what is gum? Google summer of code, so google summer of code, is a global online program focused on bringing new contributors into the open source software development, space, gsoc contributors work with an open source organization, in this case it's jenkins and they spent about 12 plus weeks programming projects under the guidance of mentors.

A

This is our six years participating in um gsoc and we have four project. Ideas which are gsat contributors will now provide you with a um what it's all about, how they progress on it and um let's go dive right into it. So jean-marc.

B

Yes, okay, welcome everybody online or listening to that recording, so I'm very pleased with the org admin team to have the presentation of the various projects that are participating to this edition of google summer of code.

B

So next slide we're going to start immediately with the first presentation from yimming about jenkinsfile runner action for github actions aiming the floor is yours.

C

Okay, let me share my screen.

C

D

C

B

Just make it full screen there.

C

Okay, let's start so hello, everyone, my name is emin I'm, the jenkins resort computer contributor of jenkins file runner as github actions this summer, so I'm honored to give the midterm presentation today about this project. I like to give a brief introduction about myself. First, I'm a graduate student from carnegie mellon university.

C

My major is like electrical and computer engineering, I'm a newbie in the jenkins open source community and I'm grateful that jenkins can give me the opportunity to realize the idea in this project, so uh jenkins found rounder action for github actions, provides the customized, containerized environment and used for github actions for users to run the jenkins pipeline inside the github actions. So in more detail, if using these actions, any github project, which has a jenkins file, can execute its workflow in the github actions runner.

C

It aims at applying jenkins in the github actions in a function as a service contacts. This feature is based on the junkies file runner, which is a command line tool for the jenkins pipeline execution engine. So look at the picture here is a basic architecture of this actions. So I combine the jenkins core. A minimum required set of plugins to run the pipeline, plugin installation manager and the jenkins spell runner into a base images, and then I set up the entry point, shell scripts, to start up the container workflow.

C

So how can you use our actions? I I provide two jenkins file runner actions in my project. Now uh one is called a jfr container action and another one is called jfr static, image action. You can access them in your github action, workflow definition by using the following url.

C

Oh, as you see in these high-level diagrams, uh they are similar with other github actions. The basic example is using actions checkout to set up the workspace and using our jenkins file running actions to round the jenkins pipeline.

C

It is pretty simple so, as I provide two actions now, so there are some subtle differences in our actions, so the root cause of these differences is the starting time of uh jenkins container in jfr containing action. The jenkins container starts up before all github actions, execution and the ends. After all, github actions have results, so all github actions will have influences on on the container. This means you can use other actions in the marketplace to set up the environment as well, so in jfr static image action, the jenkins container postpones the starting time.

C

It will start up right before the jfr static image. Action starts and end right after the jfr static image action ends.

C

uh In this case, the jenkins working working container has strong isolation with the host machine, so users cannot use other actions to set up the container environment except actions, checkout, so look so actions. Checkout has higher support, has higher native support by github actions.

C

So look at our basic function interfaces our function, interface is pretty simple. You only need to point out the running options and the relative paths of jenkins file, jcs young file and the plugin installation list file in your repository. So the command means the how you want to run the jenkins file rounder. The default command is wrong and another support one is linked and the jenkins file uh input means the relative path to your to the jenkins file. In your repository, the default command is jenkinsfile and uh the third one is pluginstxt.

C

It means the relative path to plugins list file. In your repository and the final one is jcsc, it means the relative path to jenkins configuration has called young file so, as some people might not be familiar with the tooling our actions, I listed some basic commands about them. Basically, we used the plugin installation manager to install the plugins specified by plugins installation list file and the jenkins file runner to run the jenkins pipeline.

C

So after some explanations about these actions, I want to share some demos for you, so you can apply that into your github uh workflow definitions. uh I already added some hyperlinks, so you can click them.

C

So these demos aim at applying jenkins a real fas contacts, so the wine time used jenkins controller, won't store anything in the file system. So the first one is integrating actions cache to cache installed plugins, so actions cache is, uh is a kind. It's a very popular action to cache your dependencies, which is provided in the github action marketplace.

C

By default, the plugins specified by plugins.txt will be downloaded from the internet. Every time you run the workflow in order to accelerate the workflow, you can use actions cache outside and give its cache sheet status as input in in its plugin cache sheet.

C

So one thing to note is that this feature is only available in jfr container action, because in another action you are not allowed to integrate uh other github actions with with the jenkins, with the jenkins file runner actions.

C

So next one. So the second one is about uploading, your pipeline log to github action pages by integrating actions, upload artifact.

C

So, as you can see after these two actions- and they will leave the pipeline logs in your workspace- which you can access, so you can use the action spotlight, upload artifact action to upload your logs in this path. So, as you can see, uh so, as you can see you can, uh the only difference is the path of the pipeline locks.

C

The this one has, let me see.

C

So you can download or delete uh those artifacts in the related github action pages.

C

You can see that the architect artifacts on the downside, so the following examples are still in progress, but their experiments are are successful, so you can play with them and you can actually, you can directly copy the jenkins file or gcsc config configuration uh I'll put that in the official demo pages later. The third demo uh example is caching, the maven dependencies by integrating jenkins dropper job cacher plugin. So this this one is pretty similar to the to the function of actions.

C

Actions change, so the difference is. Is that this one used jenkins plugin. uh In this case I use aws s3 as an example, but you can use any similar object, storage uh services. Firstly, you can set up the cache in the jenkins file and validate the cache freshness by specified key. So, for example, if it is a maven project, you can cache your maven dependencies by prom dot by your prompt.

C

uh Then what then, what you need to do is installing job cache plugins specified by plugins.txt configuring, the aws access key and the s3 storage path in the jcsc file and then declaring your aws key as environmental variables in the workflow definition.

C

You can also find the example in the link, so the next example is uploading uploading, the artifacts to aws s3 by integrating jenkins artifact manager as three plugin. uh So this this exam, the function of this demo is pretty similar to the to the actions upload artifact, but you can use your own cloud services.

C

uh As you can see, the steps are pretty similar to the previous example. All you need to do is managing the artifacts uh in the jenkins file, configuring, the aws access key and some related storage path in the jcsc file and then declaring the environmental variables in the github workflow definition.

C

And the final example is about triggering the building at another permanent jenkins agent. So I think this demo is pretty interesting, because sometimes you might need to collect the results from different agent buildings, because maybe sometimes you don't you don't want to only build the results in uh only one machine. You might need to use another agent to build your job again to see the differences.

C

Oh it's just my guess so. The previous example only gives the single agent example, but you can integrate with this one. Firstly, you need to create the darker agent in your machine and remember to set up the related security policies accordingly.

C

So if you don't know how to set up the darker agent, you can click the link at below.

C

I think it is specified in the ppt url. uh Then you need to call the different agents in your to to process your building steps in the jenkins file. Finally, you need to set up the agents uh in the jcs dfl and declare your declare your machine access key and the public dns name as environment variables in the workflow definition, so the jenkins file, so the jenkins file can uh use the ssh remote control to access the uh the another permanent agent.

C

uh Let me see so I also give some jcsc example here you can directly copy them, so in our project, the command line. I have to click declare that clarify that the command line jenkins pipeline execution engines powered by the jenkins file runner project and the plugin installation part is powered by the plugin installation manager.

C

So so thanks these projects I built on- I built my project on that. So that's it. Thanks for listening.

B

Thank you very much uh yaming for this. This very interesting presentation. I see that you've done quite some work, experimenting different techniques and the potential of it. Thank you very much just checking in the q a.

C

B

No, that's not the right button here, no question in the q a so back to the mainstream presentation, so the next one is the plug-in health scoring system with the diraj brush the floor is yours. Now thank.

E

You so I'll now share my screen.

F

So you might be able to see the screen right. The front page.

B

F

Awesome so let's move forward, so this project is called plug-in health scoring system and uh myself directing yoda, and my mentors are adrian and abita. So with their help, we were able to do a lot of work on this. So let me present you how the project looks like so far.

F

So this is the agenda. We're going to talk about what the project is about. Then we will be going through a demo, then we'll be then I'll be talking about. What did I learn and then there's going to be some discussion on what is next for this project and then there's going to be some questions and answers. If there's any now, let jenkins community, as we know, has around 1800 plus plugins. That's a lot. So, if you think about all of them not each and every plugin is at the same maturity level right.

F

Some might be like at the up at most power that we have for a plug-in maturity. Some might not be at that level. So there's like some dis, uh you know, there's disparities between them, some, so what we need is there has to be a system in place which tells us what exactly is a position for a given plug-in in this maturity level aspect of perspective, so we are aiming to do that by putting a score a numeric score on a given plugin, so that score is going to tell you how mature a plug-in is.

F

So that is what we are trying to do with this. So in simple words, we are just going to rate a plugin. It's going to help the users it's going to help the the you know, maintainers of the plugin. It's also going to help the contributors, the new contributors who are interested to come and contribute to the jenkins ecosystem right. So having said that, let's just quickly jump into a demo, and I think you should be able to see the intellij screen right.

F

My id are you able to see the ide screen? Yes,.

G

F

Awesome awesome thanks a lot, so we would be implementing and running this project and just before that, this is how the readme file of our project looks like so I'll, be following these simple commands that we have here to set up and keep the project running so going back to the intellij I'll, be running this command called maven package and we'll be skipping the tests.

F

Okay, it's going to generate a jar file for us and then we'll be running the dock to compose up command to start the services. So talking about services, we have two of them here. This is the java project service and this one is service related to our postgres, so both of them work together in parallel and now you can see, we have a message called started- plug-in health scoring. So both the services are running right now, so you should not believe me. So I'll show you if they are really started, they have really started.

F

So let me go and click on this connection button. So what we are doing right now is trying to connect with the database that we have just started with the help of docker composer command right. So we have set up a connection by for filling up some details and now I'll be running. Some queries to query. What's inside the database, so there's basically one table called plugins and if I query it you can see these are the fields that it has and if I query it and arrange the document, I mean the columns.

F

It would look like this, so I just wanted to organize them well. So this is how it looks like we have for a given plugin. We have its name it. ha We have its same url, we have its release timestamp and we also have a an object called a json object which is named as details all right. So this detail is fetch from the jenkins update center, which is hosted, and it's available for everyone to view now, there's a very good thing about this database that we can do right now. That is, you can see.

F

Some of the fields in the scm column are missing any value. So that means they do not have any valid stm assigned to them. Now we can query and find out how many of the plugins in the uh are there, which do not have valid scm at all. So if I run this query last one, it says that there are 52 of them, so this is one of the data inside that we got with the help of this database in place.

F

Moving forward, we're going to have lots of very important details inside this database so that we can run very specific queries to find out what's going on on the whole plugins as a whole right. So that is how this is going to look like as we move forward.

F

Having said that, uh let me quickly show you the flow, the how the code looks like not going too much into detail. So this is where the this is the import plugins here, crl uh clr file where it all starts. So we are uh calling this method called read: update center as its name symbolizes.

F

It's it's reading the update center and uh we are using this update center url to access it, and then we are using jackson's object, mapper api to read the values, and then we have these specific records in place all right, so these records are written by us so that it tells jackson. What exactly do we want to fetch from the json object that we have from update center? So we are pressing, fetching, plugins and uh deprecations, and for them we have these two other records to tell them exactly.

F

What do we need in both of these entries that we are fetching? So that's how it looks like and moving back. We are then storing those results it actually it actually returns list of plugins. Then we are storing them in the database. Using this save or update method, we just call plugin repository.save method. So that's how it looks like this is the first and like the main, important flow of the project.

F

Now moving back to our slide.

F

Now that we are done with the demo, these are the key details that we have getting the plugin list and storing them into database, and then we have this open structure to record details. So, as I told you, we have the details, column, which was storing json object, so it so. It has json object because that way, it's flexible in future, we might have different different plugins. Their details can be stored inside that json object as per. However, their properties are so that's a very conscious decision that we have taken now.

F

These are this list can go on and on, but I've put in this few of them. So what did I learn?

F

So since we have a ci in place for our system for the project, whenever we run the ci uh starts working and we have some integration tests as well, so all of them are run acro around the new contribution that we have, let's say a pr, so it builds up and runs the integration test on it check whether it is up to the mark or not it's failing something or not, and if it is failing, we would have some reports. Some spot books, reports, some uh check, style reports.

F

So there's these other things that I learned and writing best practices and clean code. So this these were able to help me. You know with this project so far, so let me talk to you about the next step. What do we have next for this project? Now we are working on designing the probe engine and this probe engine is going to determine and it's going to store the logic for each and every probe like how it works. And how is it helping to?

F

You know analyzer plugins repository, let's say so: that's going to be other probe engine and then we're going to be using a process which will apply the weights of the weights to this probe values to generate the final health score that you're looking for and that final health score is going to be delivered uh via a json file.

F

uh Just like how update center file is being hosted and using that json file, it will be rendered. The scores will be taken from that json file and that score is going to be rendered on the ui of let's say plugin site or jenkins plugin manager. It's uh it's a uh it's. This is a decision we need to take far in future, so so this is how the plan looks like and uh if you want, you can check out these resources to know more about. It contains everything.

F

And uh yes, so if you have any questions, let's, let's address them.

B

There there is one question that appeared somewhere: I don't know if it's in q, a or so a very intriguing decision and uh to store the information about the plug-in health as a json uh field. What were the arguments for and against? What were the alternative options for that field?.

F

Sure so I can try answering few words on this question. Then I'll pass it to adrian who can share more lights on that, so we chose json, because when we talk about details as the name of the column suggests, we don't know what details a plugin might have, because each plugin has different properties right. It might not be in gradle project. Some might be a maven project, so there are some differences between them. So we need to have something in place which is flexible enough for us to store anything depending on the plugins properties right.

F

So that's one thing and I think I'll then pass it on to adrian. If he wants to add anything else,.

B

Adrian go ahead.

G

Yeah, can you hear me.

B

Yes, loud and clear.

G

So um that's a very good question. The idea of having a colon by in the database a colon by a probe uh could have been a solution. I actually I not quite excluded it. I I went through. uh I went to a json column because I wanted to be more flexible, not having to restructure the uh the table each time we inc, we create a new probe.

G

I wanted to have one one column that would be able to hold all the probe results and also uh to be mindful that not all the probes, like uh dr said, will be applied to every one of the plugins. For example, we can have we have in mind that maybe some uh we will have a probe that will determine if a plugin is using a jenkins file or not, and then we can have a secondary probe saying if the plugin is using a jenkins file, is it using a custom?

G

Is it using a custom, build instruction or is it using the recommended, build configuration from the from from the infra team, and then we can see that some probes won't be. We won't apply that secondary uh probe to plugins that don't have a jenkins file, and so we would have a lot of um colon that won't have any data or usable data and also the the the the prob.

G

uh The the probe result um is something a bit more complex than just uh keeping the uh uh then to use a varchar column, for example, on uh for each one of the probe. So that's why I I went to uh json. In the end, maybe we could have uh used a new nosql database entirely uh because of uh the main data point of of the project will be started in a nosql kind of uh colon, uh but uh uh after discussing with the infra team, they they have knowledge about uh postgresql.

G

uh They uh have a way to provide it uh easily uh on their cluster um and they have.

G

They know how to manage uh that kind of database, uh and I I I know that the json uh data type on the postgresql is also quite performant. You can have um a good index on that uh data, so, even even though you, uh the the query system, might not be as easy uh with uh jpa and and stuff like that, as with others, it's totally feasible to have custom queries with either.

G

Functions from from from postgresql or direct language syntax to query uh to two colleges and objects, so this this is why uh I I I wanted to to go to uh json uh to a json conan. uh In the end, the way the the way we are storing the data in the database is not uh strictly um the core the core product of that project.

G

uh We will be able to, of course, um review that in in six months one year, if we want to change uh that might be difficult, but again the the the database is not keeping data for the long term. It's something we can for sure analyze, uh reanalyze the update center and all the plugins uh from scratch. uh When we want so we won't have really.

B

G

Yeah we can have we don't we. I don't see any problem having to uh start from scratch if we want to go to a nosql or to a a colon, but I I I foresee more data structure issues with having a colon-based result. Storing system.

B

Thank you very much adrian for for this very detailed explanation. I I think there has been a lot of thought into taking that decision very, very interesting. I'm looking forward to see the the results of that and how it progresses. There was another question. uh This one is for the raj we're a little bit running out of time, but the question was here: do we have already something to share about the different probes that will be used to decide what is a mature plug-in and how this will be measured consolidated?

B

Do we have already a good view on that? I don't forget to remind about the survey going on.

F

B

F

Yes, so we do have a survey going on to find out the weights for the different different probes that we are planning to have. So to answer the question: do we have anything just now? uh I would not say anything this working probe right now, uh but moving forward. Let me give you an example of a probe. Let's say uh one probe can be. Does the given plugin has jenkins file or not that's one probe, so we can and to give you uh to answer like how are you going to do that?

F

It's just uh going through the repository plug-in and checking the presence of a jenkins file or not so that can be one. Another can be just trying to find out whether the given plugin has the recommended jenkins base version or not. That can be another probe. So we are going to have these list of probes and there's going to be specific uh functions that are going to.

F

You know address those probes like scraping the repository and the palm xml file and checking what the version is. Is it matching with the recommended version or not? If not, then that means the probe is not passing so based on the weight that the probe has. We will be uh assigning a score for that probe and basically this is going to be happening for each of the probe and combining all of these probes values. We would be having the final health score.

F

So, okay was I able to answer.

B

Yeah great I'm I'm watchful in looking at the watch so that we keep uh uh this this meeting uh on rails. Thank you very much for this. This explanation, two things there that I I take away of it is first we're building for the future and to be able to refactor uh what we're working on in the database design in the probe design. This is work in progress and just the start of the story that we're writing there.

B

There will be details, uh maybe on the on the meet up, page or elsewhere on the various channels where the survey is don't hesitate to give your opinion. What is for you, what makes a trustable and healthy plug-in and what not we're listening listening to the community, the raj. Thank you very much for this very interesting uh presentation and some uh um interesting challenges that you're you're facing in this project.

B

So thank you. Well done. So, let's walk to the next presentation and we're going to listen to uh hushy cash, who is going to explain us his work about automatic gift. Caching,.

B

The floors is yours, oh.

E

Can you see my screen.

B

Yes, perfect: okay, go ahead.

E

Let's get started.

E

B

E

It's indeed my pleasure to be amish to you all today, I'm going to present you my work on automatic, get cash maintenance, gsoc 2022. So here we go before we get started I'll. Take a few moments to introduce myself.

E

I am rishikesh rao and I belong to the land of unity and diversity that is india. Currently I'm pursuing my third year engineering in computer science, my hobbies are coding gaming and watching movies. The things that interest me the most and keeps me occupied are blockchain distributed systems and system design.

E

Let's have a glimpse of the components of today's presentation that is from where it all started and how it's currently progressing.

E

Git is the most widely used version control tool. Many of us use it in our day to day lives. Any changes made in our project are stored, using a blob, a tree and a comet which are known as get objects as the software progresses to its development life cycle. The increase in the number of comments is directly proportional to the git directory.

E

The increase in the size directly affects the performance of the git commands, if not maintained properly. An unoptimized repository mainly contains many loose objects, no commit graph in efficient search of data in pac files.

E

My project automatic my project is automatic kit, cache maintenance. It means I'll have to optimize the git caches on the jenkins controller. What exactly are caches caches are directories on the jenkins controller. They are created by the kit. They are created by the get plugin when jenkins job is configured typically a multi-branch pipeline.

E

Each cache contains a bare kit repository.

E

A bare git repository is a clone of the original repository, but only contains the git folder multi branch pipelines often include many branches which results in the checkout of the same git directory multiple times. To avoid this duplication, caches are used.

E

What are the advantages of caches, as caches only contains the git directory, it reduces the network io and the local disk usage the disadvantages. Currently in the git plugin. There is no way of periodically maintaining the get caches. The caches has to be maintained manually using a script. Hence the caches remains unoptimized.

E

B

E

To this, yes, there is in gate version. 2.30 kit developers introduced a new api, that is the git maintenance api, the api comprises of various tasks. Each task has its own speciality.

E

The aim of my project is to integrate this api into the git plugin and the git line. Plugin. The integration of maintenance api provides a mechanism for administrators to periodically schedule maintenance tasks using cron syntax on jenkins controller.

E

Few maintenance staff, such as gc, commit graph were implemented in older versions. We support git versions up to 1.8, which was released all the way back in 2013.

E

So let me show you a glimpse of my uh demo like my project.

E

E

So can you see the terminal on my screen.

B

Yes, maybe is it easy for you to make it a little bit bigger? Although oh wait, I don't? Okay, don't try it follow. You.

E

Yeah, I'm not sure okay go ahead yeah. So what are caches uh caches is a directory present on jenkins. So if I go into the caches directory, you will find get repositories. These are bare git repositories.

E

So if I move into a git uh direct, uh if I move into a cache you will see, it only contains a get repository. The git client plugin and the kit. These caches are unoptimized uh caches right now we can look, we can look. uh We can say that they are unoptimized by looking into the objects folder. So if I see the into the objects folder, you can see so many directories present in them. Each directory contains various files.

E

They are known as get objects, so if I these are the various get objects present and inside your uh inside the get caches uh so e, each git object can be a tree, a commit uh or a blob. So let's see how much space does this uh directory take.

E

Okay, this directory takes around 127 mb. Can we do better? Can we optimize this? Can we reduce the space this directory takes? Let's try it out by running to get cash maintenance on jenkins.

E

So let me start the project jenkins software by running this command.

E

This command uh compiles, builds, and you know yeah starts the software.

E

So yeah, I think the project is up and running so it's running on localhost 8080, slash jenkins,.

E

So this is the jenkins ui and if you click on the manage jenkins uh section you will be, uh you will see a git maintenance system configuration when you click on the get maintenance system configuration. You will see the ui, so the ui is not that good right now, so I I'll have to improvise on that. So bear with me. uh The current get version is the current git version on my computer, which is being used to run. The maintenance task is 2 37.1 uh in this demonstration.

E

I'm going to explain to, or you know, go through two maintenance tasks. That is the commit graph and the lose objects. So I'm going to schedule a commit graph every minute, okay, so uh every two minutes so.

E

Yeah, this command schedules a comment graph. Every two minutes feels schedule a prefetch every hour.

E

The scheduler gc every week it is recommended to use gc with care because it it takes a lot of time to run, lose objects, let's schedule it every minute and okay, so jenkin, it is recommended not to you schedule any task every minute, as it is resource intensive. So this is a warning, but uh let's do it for the sake of this demonstration and incremental repack, let me uh run it every 61 minutes.

E

Okay, wait! What 61 minutes jenkins is throwing an error, so you can't schedule everything or maintenance tasks for 61 minutes. So, let's, let's uh change it to daily daily yeah. So jenkins is happy. Let's save the data and yeah it has been saved successfully.

E

We can look at it by looking into the logs, so there is a log system logs and the git maintenance, so yeah the data has been saved successfully. We can also confirm it by looking into the uh by looking into the xml file present in jenkins, so I can see it by you know changing plugins.

E

So this is the data which has been stored successfully in jenkins each maintenance task, along with this cron syntax and whether it's configured or not. So, let's start uh yeah before clicking on the execute button. The execute button schedules the maintenance task so before scheduling the maintenance task. Let's try watching whether this medium stars has any effect or not so we'll go into the caches directory, we'll go into the get client, plugin and we'll try watching will try watching into the info director.

E

Okay, so right now there is nothing other than an exclude file. So let's try uh scheduling these maintenance stores.

E

Okay, the maintenance task has been scheduled. Let's look at the logs. The maintenance tasks are scheduled for execution. How does this work internally internally? Every minute jenkins checks, whether this uh cron syntax is a valid chrome, syntax or not. If it is valid, it is scheduled for execution. That is it. It uh takes that maintenance task and runs it on all the caches.

E

So, let's refresh this page yeah as you can see, the commit graph has been added to a maintenance queue and lose objects has well been added to the maintenance cube because the cron syntax states to run it every even minute event minute. So two four six eight so currently on my computer, it's 9 18! So that's why uh the loose object has been added to the maintenance queue, so the maintenance task, currently the commit graph maintenance task- is being run. So let me refresh yeah the commit graph. Maintenance has been run successfully.

E

Let's look into this uh folder. Okay,.

E

B

Something unexpected.

E

Yeah something unexpected one minute. Let me look into it.

B

It's never a good omen if a demo works immediately, so I'm looking forward for the final so.

C

B

Don't want to pressure you there, but you have two more minutes to the max.

E

Okay, uh okay, so yeah I'll I'll. Try ending this demonstration by you know talking about this comment graph. So if you look at the comment graph command, you will find two comment: graphs which has been created due to this command. So now, if let's try looking at the performance optimizations we have gained. So if I run this command, which you know disables the commit graph and tries running the log, uh get law command to get the latest 20 comments. Okay, it takes around. You know: okay, there is there. Are things going wrong on my side.

B

So, don't worry, don't worry, okay,.

E

Okay, oh one minute, so if I go into this directory.

E

And now let me try the same command. Okay, I think it should work yeah this command as the commit graph file is turned off, so it is taking some time to get the last. The latest 20 comments, so I think it would take around 10 seconds to you know, get all the latest comments like to execute this command yeah. So, as you can see, it has taken 16 16.8956.

E

So let's try running this thing using the comment graph turned on wow. It has taken 0.025 seconds that is around 600 times, optimized, so yeah that that is one performance again, which we have seen using a comet graph, and one more thing I want to show is whoa. Where are these directories present on my git plugin on my caches? All of them have been banished and what exactly is the size of this directory 8.1 mb before it was around 128 mb, and now it is an 8.1 mb.

E

So you can see the directory has been compressed by around 15 times, so there are various maintenance tasks as well, but as we are running out of time, I think I would yeah stop this.

B

Is this is an impressive way of doing maintenance and a nice way to conclude your presentation, hoshi cash, that was impressive. Thank you very much. This was a very interesting presentation, very lively and you dare to do a full demo of your product. So congratulations uh for that. I'm sorry to rush you a little bit. I just would like now to give the word to avian for the uh the the next presentation.

B

D

Hello, everyone it's great to be here and I'll start in the moment. Once I share my screen, so I am doing the project um for a pipeline steps, documentation generator and my mentors are christian and harshad I'll start by giving a brief context about my project. What is it about so pipeline steps? Documentation generator is a tool that generates step documentation for our pipeline steps, reference page.

D

um What what it exactly does is it queries all the jenkins plugins for the documentation of steps formats it as an ascii doc and then feeds it into gen keys, and that way the users can see the entire self documentation at a single place and then use it. The way they want.

D

Now we'll come behind the rationale behind this project, why this project is important.

D

As of now, we can find around 600 plugins on the pipeline stress reference page, and these plugins provide us more than 1500 steps, and if we dig further, these steps have several parameters which have their own hierarchy own help text, and this makes the entire documentation enormous.

D

User feedback: there has been a lot of feedback about this page and it suggests that it's not very easy to find a particular piece of information amongst this documentation.

D

For instance, if we look at a pipeline, multi branch, plugin page look at the ascii dock for that it comes out to be around 150 000 line, long ascii dock, and that is not good for the developer or the user. Both of them will find it very hard to deal with it, even the loading speed of that the page becomes quite low.

D

So we have to deal with that right and whilst we were finding solutions for these, we also came up with some additional uh improvements that were not initially a part of the problem statement, but we thought could be beneficial for the website, so I'll start with the work that I've done under gsoc 92.

D

um Firstly, the things that I did under the community bonding period so like everyone, I spent some time sharing the code base of jenkins at io and the psdg repository. These two repositories were the main repositories of my project's concern.

D

Then I created a pull request on genk's attire to fix an app barber, and this actually led me into understanding the css and the layouts uh that are being uh that are working on that case title and help me work towards my first full request and then, um as my mentor question, suggested, it's better to create an epic on jenkins, zero to keep our work organized, and this really gave me a corporate type of field to get the project organized and I listed all the tasks under that story and it's working out really well for us and then finally, I designed some wireframes for the ui, the layout part, to get some feedback of the community so that they can understand what I'm talking about and get the idea on paper.

D

So, let's begin with coding phase one. uh The first improvement that I did was change the sidebar scrolling behavior. So this was a completely ui based uh change um so before the sidebar uh scrolled, along with the content. So let me just move on to that page.

D

So this was the ui that was present earlier for the jenkins documentation, sidebar and, as you can see, the sidebar scrolls, along with the main content so in. If someone wants to scroll down and then they want to shift towards a different page, they'll have to scroll back up or maybe just press the home button, and then they can see the sidebar and they can click the link that they want to.

D

But this is not an effective way to. You know work with the things parallely you need to have both of them together, and that is the standard convention that is followed by most of the documentation nowadays. So uh I went ahead and did that and now the updated look is something like this. So both the content and the sidebar have independent scrolling.

D

You can scroll one of them, the other one will stay stationary and if your screen is smaller than the height of the sidebar, the side button is, it says, scroll, and in this way you can scroll, you can navigate through your entire content and the major change about this was this has been affected, affecting the entire engaged documentation, the user documentation, so not only pi by sales reference, but the entire documentation was could get the benefit of this change.

D

Moving on to the next change the presentation, so these are the things that I mentioned. um The second thing that I proposed using the wireframes was listing the plugins in the pipeline steps reference.

D

So what I meant by that was to in fact list all these headings that you see over here inside a sidebar, so that the user can actually see what all plugins are there and then click on any one of them to find their individual pages.

D

But after I implemented them, we found that it might not be very useful for the users to in fact use this change and the page was becoming more crowded and we felt that it was dragging us down more. It could be more than it could benefit us so from some the community's feedback. We found that that including a search filter for that would be much better and that would be the way to go.

D

So we scrapped that change of listing plugins, and then we moved on towards working on our search filter and that source filter was more successful and it was the change itself was much more effective than the list could ever be. So I'll show you what I mean by that, for example.

D

If I want to search about git, um so user just wants to see all the steps or plugins have git in it, just as an example and as you can see over here, the page has 101 search results for that and, for example, if you want to toggle between them here in one line, you have two places where the gate is present and basically you'll have to navigate a lot in order to see all the search results and seeing the relevant results at one place is not possible with this method, but with a new search filter that I've created.

D

It's very simple: you can just type in git over here and you will immediately see only the results that have the keyword get in it, so maybe the plugins or the steps. So, for example, this step has a word git in it, but the plugin doesn't so it the plugin shows anyways so that you can at least see what again it is coming from, but it's not the other way around. So if a plugin mentions the word get in it, uh the steps won't be shown and only the plugin will be shown.

D

So, for example, if I type scm step over here, it will just show me the plugin and not these steps that come along with it. So this gives us a very minimal minimal data on every page, and this essentially makes it very easy for the user to filter out the data that they want to see.

D

So this was the search filter improvement that I did moving.

B

D

Next one, so I've linked all these pages in the before and after section, if the. If anyone wants to see the these uh previews, so the after is basically the jenkins website. Since all these changes are merged and are live and the before are some previews deployed on netflix.

D

um The next change was separating the declarative steps from the main class. So what happens? Is um this generation of declarative steps is separate from the generation of the scripted steps of the pipeline, and because of that, we felt that it would be useful for the developer to see these two as a separate class.

D

So why not separate the declarative steps in a separate class and then use it inside our pipeline service start the main class in the way that you want to so this would make a make it much easier to refine or restructure the step. The way in which the declarative steps are generated, and currently they are present only in the declarative heading on this page.

D

So if you want to remove them in the future or maybe put them at a separate page itself, it would be much easier for us and the biggest benefit for this. Was it made the main class for pi pencil extractor much cleaner. um So that way we have only have the relevant functions that provide the maximum documentation and not the minimal things on the main class. So this was some code based related changes and then came the shifting parameter data types change.

D

What this was about is um what you can see that um whenever you visit a parameters documentation, what happens is for example, let me open the scm step uh plugin and let us see the checkout step and scm parameter within that. um Let me open one of these dropdowns and, as you can see, these types are present in a new line and the branch is there and then you have a type. So there is no help text over here, but still the data type is present on new line.

D

um One more example would could be opening up the git scm class and seeing the branches example. So here for name, we are not able to see the data type yet, but if we expand it, you'll have to scroll down through the entire help, and then you will find the type sitting somewhere over here at the very end.

D

But now, with this new change, you can find these data types in line with your uh the names of these parameters. For example, let me open the clear case again and, as you can see, the difference is clear. The data type is present in line with these changes um and seeing the git scm thing for the name.

D

This the string is present here itself and you need not even expand it to the first place. uh You can only expand it if you want to see the help, but it's not required now so at all levels of hierarchy. These types have been replaced by inline data types, so this ultimately reduced the height of a page, the scrolling height of a page. So if we expand many of these, you can see that the scroll bar height itself, the scroll bar size, is different on both these pages.

D

So the scrolling length of the new page is much lesser, which in turn reduces the size of the ascii dock and everything the loadings loading speed. Everything is improved cycle, okay, um so this was the so all about the parameter like shifting uh public quest then came the last, but not the least task of uh coding phase one. This was releasing a pipeline method by utils.

D

So what this uh does is it provides you, a localized, plugin manager and a mock jenkins instance that you can load in your program and which can be used to query plugin manager, different plugins, using the plugin manager. So the way we use it in the pattern set extractor was diagramming, the plugins feeding that into a reactor that react initializes, and once that is done you can.

D

You can find the components that have the type step descriptor inside a plugin, and you will get a list of these and from that list you can generate the documentation that you want, but we realize that this can not only be used for extracting the step, help but other types of things as well.

D

So let us look what what all it can do so, for example, if you want to use it in a rest, api documentation project, we can use this artifact and we can use it the way we want the plugin manager gets initialized, and then we can query it. The way we want. So there are many options available.

D

uh We currently only use three of them for partners direct starter, but there are many more because the hyperlocal plugin manager itself inherits from the main plugin manager of cycles, um and you can find this artifact on the jenkins maven repository. It is released and all thanks to jama the release has been done and we are able to use that in partnership extracted now, so we have removed the local dependencies and using this published version.

D

So this is, you can find it over here and linked it in my presentation and the some things that I did under this release process were, first of all, completing the java code of the uh hyperlocal plugin manager mark jenkins, mock extension list, updating everything with the main class, and then I updated the form of this repository to use the parent perm, which is a non-plug-in parent bomb. And then we moved this to java 11.

D

It was earlier on java 8 and then I wrote some tests to see if the green path for hyper local plugin manager is working, fine or not. So under that I initialize the plugin manager using the reactor that is used in pipeline step extractor, and from that I can query my plugin manager for some tests and we can assert those cases to see if the tests are working. Fine and finally, we release this as a maven artifact, which can be used as a dependency. By simply copying this down and pasting it in your palm.

D

B

Running out of time, so you have one more minute to conclude sure.

D

Sorry we are using this in our pipeline documentation general repository, so this was all about the changes that I made for a till now bid evaluation, and these are some tasks that we have proposed for coding phase two. So many of them are obtained from the community and first of all, the task would be to label deprecated plugins on respective steps page that mark suggested.

D

So that would be the first thing that we'll do and then we want to separate deprecated steps from the advanced steps. Currently, all the advanced steps, some some have optional. Some are deprecated and it's really very hard to distinguish between them.

D

So that's the thing that we would want to do and then comes a bigger change in which you would want to break down these pages, the with larger information into smaller ones, so that we can solve the ultimate issue, which was a proposed in the problem statement and yes, uh any more changes about the layout, the uh the ui, the documentation.

D

They are welcome and you can get in touch with us in the docs twitter channel or the europe edition of the docs office, or you can find me there and any feedback is welcome um and these are some links you can find them on my project page as well.

D

Yes, this was it from my side. Any questions, uh please. Let me know.

B

Right well well done vyan, so and and sorry to have had to to rush you uh at the end great great work um I just going to conclude uh and q a can go uh uh through uh this course or getter if they're there any there. So I just want to to share the jenkins upcoming events uh that we have so the jenkins community of jenkins project will participate to scale scale conference in los angeles.

B

uh The end of this month, mark waite and elissa tong will be there and probably.

B

Kosuke, the initial writer of jenkins will attend also and will be present at the booth. The big event that we're going to hold is end of september devops world, as, as you know, I'm not going to do more explanation. What it is because we're completely running after time and be prepared to hear information and looking for volunteers for this year's edition of oktoberfest in the jenkins community, so stay tuned for details.

B

People on the west coast may be looking forward to see you at the jenkins booth in scale and stay tuned for information for the two other events, I think that's it alyssa did I wrap it up. Thank you very much for everybody's attendance.

B

The world is back to you. Lisa.

A

um Thank you to our gsoc contributors. Thank you to our mentors. um Couldn't have done this program without all of you so well done we're out of time. So we'll see you next time.

B

Bye bye, everybody.