Jenkins Google Summer of Code Office Hours, 2 Jul 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Jenkins GSoC 2020 Coding Phase I Demos - Part 2 (Jul 02, 2020)

Description

Part 2 of the GSoC project demos in the Jenkins organization. At this meeting GSoC students in the Jenkins will present their projects. Every student will do a quick project overview and a live demo of the current project status. Agenda:

* Martin d'Anjou - GSoC 2020 introduction
* Loghi Perinpanayagam - Machine Learning Plugins for Data Science
* Buddhika Chathuranga - Jenkins Windows Services: YAML Configuration Support
* Sumit Sarin - External Fingerprint Storage

Full presentation abstracts and links can be found here: https://docs.google.com/document/d/1F9JVUEQyTL_JI8-_WPV-vMbuX94ZENjffsVlPOptPKE/edit#heading=h.ibikyjoj0fur

A

Okay, hello and welcome everyone to the jenkins google summer of code year 2020. This is the coding phase. One demo part two. We had part one earlier today which you can find on our youtube channel. Once we publish it yes, this year being a challenging year, we have come together and we participate in the great movement called open source.

A

The jenkins community is mentoring students under the google summer of code program right, so we're now we're now in part two of the student presentations, so I'll give a brief introduction to jenkins in google summer of code or as we call it as we shorten it.

A

Jsoc then we'll have our project demos by students followed by questions and answers, which jsoc organization are we we are the one on the left pointed to by the arrow we are um operating the jenkins and the jenkins x summer of code project under this banner and jenkins is part of the greater arc of a greater organization called the continuous delivery foundation and that organization runs other summer of code projects. Please visit their websites for more.

A

The jenkins community is in its fourth year of participating in the summer of code program. This year we have 20 project ideas and we've been able to accept proposals from seven students.

A

As a reminder, our capacity to accept student projects depends greatly on how many mentors we can recruit, and this year we are very fortunate to have with us an average of two to four mentors per project, as mentioned before, we have projects for both the jenkins open source project and the jenkins x, open source project jenkins x is is not the same as jenkins, but the difference is uh for a late. uh The explanation between the two is for a later time to learn more about our uh how we operate the program.

A

Please visit our main page under jenkins, dot, io, slash projects, slash gsoc,.

A

You can reach us through our different channels, so we have a mailing list. Under a google group, we have a gitter chat under getter.im, slash jenkins, ci, slash, jsoc dash sig and the dash sig stands for special interest group. We hold regular office hours on wednesdays, on wednesdays online or as video chat as needed.

A

We have project specific channels which you can find by visiting our jenkins jenkins website. We have links on each project page to those specific channels and the jenkins x operates the communication using.

A

Slack: okay, q: a during uh this presentation, you can use the zoom chat. You can also ask your questions on gitter and after the presentations. If you have further questions or want clarifications, please don't hesitate to reach out to us on getter or on our mailing list.

A

We also have a code of conduct that we're updating pretty soon. Essentially we are. We just want to be nice to one another and that's what we are codifying in our code of conduct. If you want to see these slides after this presentation- here's a short url at the bottom of at the bottom here, where you can find these slides.

A

So before we get started. I would like to thank our formidable students for the work that they're doing on their respective projects. I want to thank our dedicated mentors who are assisting students with their projects, and I want to thank our org admins, who are making things possible behind the scenes.

A

Earlier today we had part one demos so now we're gonna have part two demos, the part one demos were the following. The part two demos are the machine learning plugins for data science.

A

It will be followed by a presentation for the jenkins windows services, yaml configuration support and, lastly, our last demo today is going to be external, fingerprint storage. So before we start, would anyone like to add something to the introduction.

B

Hi this is marky. I would just like to uh congratulate all of the students uh all of the mentors and the org admins for yet another great google summer of code year and I'm very excited to see uh what these crop of students have done.

A

It is very exciting indeed, anyone.

C

A

Sorry I didn't get that.

C

Just plus one um yeah, we have already delivered a lot of stuff. Many projects have previews for jenkins users and yeah. I'm really looking forward to all these features to be delivered, and I am proud to see all the json projects on the public jenkins robin above this year, which is a kind of new thing been compared to previous years.

C

I'm excited to see that.

A

Thank you markie and oleg for your comments, all right so.

D

A

D

Quick comment just to thank uh martin oleg for coordinating this whole program. I know it's huge amount of work and just sitting on the sidelines here and getting the emails and all the organizational stuff that you guys are doing doing a great job. Thank you. So much looking forward to the presentations.

C

Thank you. We invested a lot in uh automatic uh automating and document things or conjunctions. So maybe in a couple of years we will be able to replace ourselves with changing consensus in a few pipelines for sending emails.

A

All right, okay! So now I would like to invite logie to present coding phase one of the project, machine learning plugins for data science. I will stop the share so that you can start your share.

E

Yeah, thank you. Martin yeah awesome. uh I'm gonna share my screen right now. Okay,.

E

Is that visible trailer.

E

Yes, yeah. Thank you. uh Welcome to my presentation on machine learning plug-in for the data science. So uh a quick introduction to myself. uh I was selected for gsoc 2020 with jenkins, uh I'm doing my major in computer science and engineering uh from sri lanka. So next we move on to the presentation of coding phase one.

E

So here we are, we are addressing a problem with a common data science discipline in machine learning field, so this can be related to a real world story. uh When I was working in my previous internship, I had a chance to work with jupiter notebook with a postgres sql for a nlp machine learning project.

E

So in that project uh the jupiter notebook creates a postgres sql to database to store all process data and every single data processing will be reflected in the database. So I need to so every time I have to check the data is up to date manually, while I try to build the model using my python code in the jupyter notebook.

E

So this is even worse when this happens in the production line when we are creating a machine learning model for in the industry. So this inconvenience could be resolved efficiently using cdci.

E

So here what we are saying is alone: interactive notebooks are not capable for building a production level machine learning model, but but they are very, but they are very excellent in both data, analyzing and coding. At the same time, you know that we can visualize visualize interactively from the jupiter network. Jupiter notebook is, is giving a great support for machine learning in this software world, so we cannot simply ignore notebook uh to build our models, so our goal is uh to automate these challenges in machine learning in machine learning workflows from end to end.

E

So in this slide here it is a simple uh way of using cdci principle uh applied to machine learning problem. uh This can give a sort of insight why we are here with this, our plugin. So, first of all, if we take a production line, uh first data engineers are the responsible persons for the data processing.

E

Then the data scientists will build high accuracy models using processing, processing the data and the code, so they have to run iteratively to build high accuracy models. So using this model ml engineers and software engineers will do the rest of deployment works as usual in the software development industry.

E

So this is a simple way of explaining how cdca can be applicable in machine learning, production environment, so why we are using the jenkins to automate these changes these challenges, so one of the reason is there are lots of plugin have been developed and maintained that can be very useful for automating machine learning, workflow in the production, uh for example.

E

If if you want to push uh one of what one ml model to a s3, aws s3 storage, uh there's a s3 publisher for that, so likewise there's a lot of plugins can be integrated with the jenkins. So we don't need to do a scratch work from uh bottom to up to uh to satisfy this.

E

uh This work loss so in the next slide, this slide after I have selected for gsoc uh during the community bonding period, we have spent a little time to name our plugin and designed a high level architecture that that gave good insight to mentors to guide me later. We created a repository and we prepared for the coding phase funds and a lot of interaction happens uh in the community bonding.

E

So so far we are done with these features in the phase one coding, the first one is uh configuring.

E

uh Configuring, the connection between ipython kernel and jenkins instance uh in the next user will be able to copy their notebooks to the workspace in jenkins uh if they want to convert the jupyter notebook into python files and json files, uh json files, which can be used to run uh the notebook by cell by cell, that will. That will be very useful when we are running the code, we can understand what's happening in the code and we can easily get get the knowledge of code.

E

So the next feature is uh we can build uh jupiter, notebooks, uh python or json. So these are the features we have done with the according phase one yep. So next one is demo before going demo. I will give us a short description about. What's what I'm gonna do in the demo, so first I'm gonna configuring, the local ipython server in my local machine and then I'm adding jupyter notebooks.

E

Then I'm going to build a model with the diabetics data sets with logistic regression. It's a it's a simple linear model. We can use it very simply. There are maybe 20 lines of code. We can build a model so for the demo purpose. I have created that then we can use another notebook prediction score. We can use it to predict uh how the model predict the model accuracy. So the accuracy means how many patients are classifying. Who are data diabetes or not?

E

So we will move to the demo.

E

So first I'm gonna configure my server.

E

A

Here, uh instead.

F

A

Let me interrupt you logi. We still see the slide of the demo with the different bullets. Maybe you want to change what you're sharing.

E

Yeah, okay, okay, okay yeah, is that okay, right now.

B

Now we can see it low here.

E

Yeah, thank you. Thank you, marty yep, so I have already configured.

E

Okay, here, okay, I I can configure a new server here, so you can add a name like uh it could be, it should be alphanumeric and it can contains only uh underscore like I can set a local server.

E

So next one is, you have to add the ip address. You cannot give any more from the ipad, so the validation will save. It will say that smartphone, so we will to uh simple localhost- or maybe you can add as a localhost to this server. So the launching timeout is about uh the how long it would take to launch the ipython kernel in the server and the next one is. The results is.

E

How many lines you can get you can gen you can get from the results. So sometimes, if you, if you, if you know about pandas in the machine learning uh it will, if you print the you can print your whole data sets, you can see. So if you add three, you can only see the three three rows of data, so maybe I can add something so in this next. I will test this connection.

E

So it's creating. Okay, the connection is successful because it is in my machine, so it is very speed. So I'm gonna save my configurations.

E

So yeah we done with the global configurations. So now we are going to create a job, a free style, job, freestyle, job, okay, yep. Here we go so in this case you can. You know these are the descriptions panel and you can select any service you configured in this global configurations, so it will show every service. You can get the global configurations.

E

So for this demo I would like to add a database data sets to this machine learning machine learning demo. So I'm gonna use a file parameter, so I'm gonna use the name as diabetes, diabetes, dot, csv cfc is a famous format for the data sets. So in the next build environment. I would like to add my notebooks in my local system, so I'm gonna give my absolute path for this.

E

For this files demo, I'm I have my uh notebooks in my lock my my computer model.

E

So I'm gonna add my next.

E

E

Demo, test model.

E

So if one user want to convert these uh files, these notebooks to a json or python, so they can easily easily use this uh checkbox. If you, if they check this, it will show a simple uh menu to convert these files to json of python.

E

So in this demo we can do both of one, but we will choose json, so he can also name it to us, a different different name like train.json, so in the next step he has to use this this file path to run this this this notebook.

E

So we will let this file as it is, and in case, if you get error in python codes, uh you have to view it's in a rich formatted view, so you can use a nc color plugin. That can be very useful uh when you are seeing the console in a very rich, very rich, formatted text, so I'm gonna use one of these, so next is the build step, is one of the important this project, so we are created a builder, so we can.

E

We can give the name here to train.json for one of this and we have two options. Like text parser, you can directly add your code here in this uh text area, so it will also uh uh run the code from this from the ipython kernel. So for this demo we will add a train.json and we will add another name for that uh test model.

E

E

Dot ip yeah here we set up with this config job configurations, so we gonna save this.

E

So when we are running with this job, we already have added a parameter. So we have to add the data set to the workspace. So here I'm adding the uh data diabetes csv, so it will.

E

It will reflect in the workspace in the later so we're gonna build.

E

A plugin, so here you can see here it it. It run the course by cell by cell, so here it saved the model here and the second notebook will get the model and uh give the results and the accuracy of this uh accuracy of this model.

E

Yeah. That's that's all about the demo yep! So we'll move to the presentation back, so I will share the screen.

E

Yep, so that's all for the demo and yeah, uh so there are exist. There are some exciting features will be implemented in in the future.

E

We are currently exploring to connect to existing python kernel from jenkins and uh we will add code editor for the for edit for editing, notebooks and python files in the code base according to according phase two and you can find more information about our design, our design in the design document and the machine learning project in jenkins, dot, io, so yeah.

E

So these are some helpful resources. You can find my github repo that will have some very detailed readme. You can follow that in that paper, so these are the some resources, yep yeah yep. That's all my presentation and finally thank you to everyone for joining in my presentation and thank you for this opportunity to work with with this project with my mentors and work at beans. Thank you for special thanks for maki, anes, shivai and bruno and also other organizat.

E

You give great support yeah. Thank you.

A

Thank you logie. um I.

B

A

I would like to uh invite the mentors to ask questions or comments.

B

uh I would like to give a comment, first and foremost to lohi loi. You make this extremely easy. uh You are an extremely talented and bright individual.

B

I am so happy that you are a student for this project and I I know I've said this before, but this particular project has so so much wide reaching uh implications not only for machine learning and data science, but but in the medical field, in the imaging field, there's so many things that can be used now because of what you're doing- and I hope you understand the the the awesomeness of that.

B

uh I would also like to thank the other mentors, our lead mentor bruno, who can't be with us today uh because of the time difference it's super early in the morning for him, but uh I bruno especially ionis chavez. Thank you for all uh that you do on this project. That was all for me.

D

Thank you, marquis and and logi. uh Thank you as well. uh Those was a very good presentation and- and uh I can certainly reiterate many of the things that uh marquis said about the development of this of this project- um I'm particularly excited about this uh because, as marcus said, he has applications to uh life sciences, and I don't know how many of you remember, but a few years back, I managed to uh um get a name of just for that um for jenkins.

D

We have now an open source project called jenkins for life, science, continuous integration, and I'm certainly hoping that this plug-in will be a big component of that as well. So great, thank you.

G

Thank you lucky. Thank you, nice, for your words yeah. Thank you. So I mean uh loki has done a very good job, or you know so far and though again, as you know, as mark has already said that the prospect of this uh plugin is, you know, really useful for a lot of the different machine learning workflows that you know people are having because the more we are you know moving towards uh more digital age. You know we are seeing.

G

uh You know, like uh data science being involved in multiple sectors and having this kind of automated workflow will definitely help. You know uh to actually move more and more. You know machine learning, models and kind of studies to a production environment, so armenia logy, has been quite brilliant. You know with his work uh ethics as well, and I mean the losses that he has applied so far and within the plugin, and it has been you know, free to work with the other mentors and you know be part of this project.

E

E

Thank you for your compliments.

H

C

So um do we have uh some time left for questions?

C

I think we do yes, okay, so yeah. I have one technical question. So in your presentation you presented how to use the giving with freestyle projects. uh Do you actually punch support changes pipeline in your project.

E

Yeah, I I I was presented to in the last meeting uh but uh ev, but mentors preferred that you, you can do a free style project that will be very uh details to audience. Yeah.

C

But technically it supports pipeline now right.

E

C

Oh great and the another question is how to try uh your project out so how to get it installed and how to get it running.

E

Yeah, so the project is uh now we have published alpha release. So if we want to, if you want to download- uh or if you want to use this plugin, uh you can uh use the experimental update center. So that would be very easy to download your plugin in your jenkins, workspace.

A

A

A

Do we have other questions.

A

Okay, so let's move on to the next presentation, which is the jenkins windows services, yaml configuration support and this will be presented by putika.

A

I will now stop my share and you can you can share your screen. Yeah, okay,.

I

Thank you, martin.

I

I

Yeah, I think you guys should see my screen now.

A

We can see your screen. um Could you check your microphone volume see if you can increase it? A little bit happen every.

A

F

Hello one: can you hear me now, yes, much better! Thank you, okay! Okay! Thank you very much. Okay,.

I

So uh welcome to my project presentation. So what I'm going to do uh in my project is uh under the ig sock 2020, uh uh I'm going to do. uh uh Yammer configuration support for uh jenkins uh windows service wrapper. So, okay, uh okay, uh here in the presentation, uh I will go through uh these topics.

I

uh First, I will introduce myself and uh then we will talk about uh you know stripper and uh also uh current configuration session and yaml configuration support, and uh then we will talk about uh phase, one updates, what I have done uh so far and uh okay and uh there will be a demonstration and uh initiation. I said okay, uh so I am buddhist ranga hi. I am from sri lanka and I am finally undergraduate of university of morocco and faculty of iit.

I

And uh yes, this is my first time in google summer of code awesome experience and I have a little bit of experience in web development actually with angular and css. So basically I wanted to move into computer science things. So that's why I uh select this project uh which have not much html things.

I

You know okay, so, okay, when it's come to windows service wrapper, you know uh we can run jenkins uh server and client in windows machine as we know, service which will provide uh more robotners so and actually those uh that feature is also bundled into the jenkins core as well. So currently you know when we are deploying windows service. We need to feed a lot of configurations. So currently the those configurations are uh free to uh windows service wrapper by an xml file. uh So you can find the project uh the given uh link.

I

So, okay, uh let's talk about uh current configurations a little bit so, as I mentioned before, the configurations are configured from uh xml file and uh the xml file should be in the same directory where the executable uh place, then the uh xml files name should be the xml files name should be same to the external file names and also a user can't specify the xml file at the file path from command prompt separately and there's no xml schema validations and there are limited configurant checks as well.

I

uh So uh if you are interested to find some uh sample configurations file, definitely you can find the sample uh configurations file which are minimum configurations and and all the configurations as two files from the uh given link.

I

Okay. uh So uh now we are looking at the uh sample xml file, uh so you can see uh it's uh much, not verbose and not human friendly, and uh you know it's really difficult for newbie uh to edit uh those things, and uh so uh what we are going to do is uh we are going to provide those uh configurations as a camera file. So that's uh what I'm going. uh That's the uh solution we are going to uh present in the project. Okay.

I

So while we move into uh yeah, so yaml is less focused and much more human, readable than xml, and definitely it's a lightweight than xml json, because they are not using extra delimiters and also the yaml is becoming more popular among configuration management tools like build tools, it see. So uh that's the reason we are moving into uh yaml as a configuration file. So now we are looking at uh sample aml configuration files, so obviously we can see it's uh much readable and human friendly. Anyone can edit this and use this as so.

I

This is a kind of a comparison between those two files. Obviously you can see yeah. My demo thing is much readable, so it's really easy to handle anyone uh even without a good programming experience I mean yeah ml experience is needed, but yeah, it's still uh readable, okay. So uh this is my project scope that I am uh I'm going to. I do during a google summer of code in 2020, so obviously, with that yaml configuration support, my project and uh also yummy schema validation will be added and a new cli.

I

I will talk about these things in later. Slides and also sml uh schema validation. Okay. uh This is how my timeline is uh structured now, but I I'm not going to talk about it uh more. However, I'm happy I'm uh all the time.

I

Okay, uh let's come to uh phase one updates, and uh uh there are a few updates uh which are uh has a new cli and yaml support and html schema validation, which are not realistic, but definitely you can find uh those pull requests in uh given links and uh this w project structure, reviews published and uh with the help of all it.

I

Thank you so much uh so if we talk about uh new cli asi, uh as I mentioned before, uh we can, we user can't specify the xml file from externally uh income online, but uh in in in uh three phases we are going to uh keep the xml support tasks. So definitely you uh usually need some. uh We should allow user to uh uh separate uh specify the configuration file uh separately. uh So uh actually uh cli thing was not in my plan, but definitely to move in uh this project. We need this.

I

So uh all cli commands uh was something like that we can uh okay, ocli command or something like that. We can specify the executable name and then redirect where lock should be redirect and then the redirect path and we can elevate the command from tesl and then we can uh specify the command. So it's kind of only a commands manner. We can't I specify any options here. uh If uh I mean user want to specify things, uh it's not a kind of structured day in new cli.

I

What we are going to do is uh when it's kind of uh commands and options manner like here's, the executable and the user can mention the command, and then he can specify the uh options for this. Like uh he can use this c or the config file uh option to specify the configuration file and then he can use this uh command to elevate the command prompt and he can use this uh configuration uh uh sorry uh option. Two uh uh specify the uh redirect path available should be redirect.

I

So uh there are these options in uh shorter and uh longer manner as well. So uh okay, uh so I think I described these things and- uh and I use a command line library to passing those command line, arguments and I charge about 2.2 k stars. I think it's a really good library, and so you can find my put request in the given link. Okay and then okay uh means come to ml configuration support.

I

As I mentioned before, yeah the configuration will be delivered as yaml file, and in this case uh the configuration will be provided in more structured way than extent, and I used yaml.net library to dc realize lml file into an object draft. So that's a really great library and in previous library I had some compatibility issue with uh because it was not support. uh It was not support for uh donate four point: uh zero upwards, but uh later um windows, size, wrap.

I

Also, obviously, the support for uh windows- I'm sorry.net 2.0, so it won't be a problem at all and uh yeah means complexion, schema validation. There wasn't a schema of validation for xml before so I I had a xml schema, validation and so user can use that schema file in order to create the xml configuration file and xsd file will be uh shipped as embedded resource. So uh in uh I mentioned there that I am going to do. uh uh uh Okay, I mentioned uh I will add.

I

uh Yaml configuration support as well in phase two hopefully, and uh the my plan is. uh Current plan is used to uh json schema to validate uh my file essay. So, okay, uh those are the things that I have to mention about my project. I think it's clear so now I can go into my demonstration and.

I

I think you can might see my screen uh come on. Come now. Yes,.

A

We can see your.

I

Terminal. Okay, thank you, marty, okay, so here's my uh uh windows service, dropper executable. I will use this and this is the jenkins dot profile, uh which will run jenkins and in this presentation, I'm going to use this config under all underscore new dot tml file in order to provide the configurations within those sizes.

I

I

Let me show my configuration file lessons so, okay, uh this is my configuration file, uh so you can see the id is as jenkins and uh those are basic configurations. Things are here, and this is the same xml file for that and okay. Now, I'm going to use this uh yaml file in order to create this uh windows. Size wrapper, okay. First, I will copy the file part of this configuration file because I mentioned that now user can uh specify the file path. So, okay, sw first okay.

I

Before that I will uh introduce about my new cli uh a little bit uh so in previous uh uh windows, stopper command line. We can do something like that and then.

I

Part will be there and we can uh do the command so, but uh in cli what we are going to do is uh now uh we have uh so okay uh before going to that demonstration. So I will uh show what are the commands available only so easily? You can use help command in order to see uh those old uh commands available here and also you can use.

I

This help option in order to check the options which are available with uh this uh command that sell okay. Let me show you, okay, uh install and yeah. This will show you all the uh options which are available with uh this particular install come on. Okay, okay! uh Now uh let me move into the demonstration first, I'm going or what I'm going to do is I'm going to install windows size wrapper with that configuration file, and I can specify it in this manner.

I

Okay, now uh windows server service is installed. uh Let us check whether it has installed uh correctly. Okay, now the id and caption knows finnish, w demo. Okay, let us check let us refresh this before okay yeah.

I

uh It's uh uh should be there and it's uh difficult to find all the time.

C

There's no particular search and service manager is still quite complicated.

C

F

I

I

Yeah, I could find it.

I

um Okay, let me refresh it again and uh let's see whether it is here.

C

Maybe it actually wasn't installed due to whatever reason.

I

um But it's shown: okay, uh let's try it again: okay, I'll use a shorter command line, this time uh yeah, but so it's saying that already exists. I think.

C

So that's a good question. Maybe try refreshing it again.

C

D

uh Extended, rather than standard.

D

Oh sorry, you have two tabs at the bottom. One says standard and one says extended services. Is it possible under the extended.

I

Yeah I was finding with the caption yeah uh yeah it's here and uh status is not running yeah. Thank you so much yeah, yeah hi. You know uh demonstrations always work.

I

Okay, uh now it says installed, and uh you know what I'm going to do is start uh jenkins. With this configuration file. Okay, uh then you can use same configuration file for that. Okay, it's starting and uh uh let me check whether it's state is changing yeah. uh Now it's running. Okay, let's uh check the browser.

I

Yeah, I think it's working yeah, it's fine, so uh thinking started as a windows service now, so what I'm going to do now is, and I'm going to stop this uh jenkins server and I'll use the same configuration file.

I

Okay, now we should be stopped by now. Yeah, it's fine, okay! uh Finally, I will install.

I

This okay, uh so let me uh show some uh invalid commands, so invalid commas, uh I network, and uh it will uh show here and also internally the options as well uh so yeah. So if, uh as I mentioned before, uh we can uh at the at the moment, uh uh we are providing uh support for xml and ml uh both. So if you want to uh boot uh windows, stopper with xml file, then it's totally okay.

I

We can do it xml file as well, and if we don't have an xml file and ml file both then it will automatically go to find the same xml file with the same name of executable file in the in that directory and uh okay. uh Let me check, I think there was something yeah there's xml file. So let me just install this.

I

I

uh Yeah jenkins is already there, so let me do it.

C

I

And yeah now it comes file is not specified. Now uh it will start with xml file uh the same with the xml file, which have the same name of executable.

I

So yeah, that's my demonstration up to now, and so I took that I have implemented the xml uh schema validation with xsd, so you can find that pull request. Yes, so uh this is the particular xml file that I have created. Yet so user can use this xd file in order to create the xml file. You can use this schema to create the particular xml configuration file. uh Yeah. Okay, that's my demonstration. So let me go to my presentation back.

I

Okay, yeah! That's all about my demonstration! So please uh there's any correction. Now I'm open, you can ask anything.

A

Any questions for.

A

C

Okay question to the audience: who does you see windows here?

C

D

Pick me pick me pick me: I do use windows, this is yani's quite extensively um and I never actually run jenkins as a service. Unfortunately, because I need access to some local paths and other things, and I don't think services allow that so I'll usually run it as a scheduled task.

C

But yes, still services the most recommended way to do that for common setups, because it provides a lot of reliability. It provides a built-in, failover and other features, just ported by windows, service wrapper and now we shall support it by yaml configuration so yeah and by default jenkins, installs agents and jenkins uh server as uh windows services. So this project provides a lot of value for such setups, um especially when it comes to configuration management, because it's also a thing for windows um and having yaml uh support uh unblocks.

C

A lot of things which are not possible with the current configuration management tools for xml.

C

Personally, I think that it's a really useful project for those who use windows and, according to our statistics, a lot of drinks, users run on windows and thanks a lot for dubudika for working on that, because we've already got a lot of great features and I'm looking forward to see them released and to become available to all windows service reporter users, whether they use jenkins. I use other services, because windows service wrapper, has millions of users and jenkins is just one of the way it's used.

I

Yeah, thank you olek. Thank you for the description, so uh I think uh that's all so I want to give a huge uh thank you for my mentors, oleg mike and uh next turn. He did a great job uh to uh make the project success and yeah. Thank you alex. So much you were helping me when you are traveling. So, as I mentioned before, I was not into comprehension, so mentors helped me a lot of things, a lot of things. So thank you so much and thank you.

I

That means uh it's a great opportunity to work with your genius minds and awesome people. So thank you so much. Okay,.

A

All right, thank you. um Could you please release the screen share so that we can move on.

C

While mike, would you like to add.

A

F

A

Mike, if you are talking.

C

A

Okay, so let's move on all right. Oh one comment in the chat so mike says that his microphone's not working, and it will not unmute. So if you've got comments, please share them via the normal communication channels. For the for your for your for your project, all right.

A

So our third presentation today is the external external fingerprint storage by sumit, so sumit. I will stop the share and you can grab the screen thanks. Martin, uh let's just start.

H

H

Can you screen me, can you see my screen.

A

H

Awesome uh so uh hi everybody uh uh thank you for uh joining us for the external fingerprint storage presentation, uh which is one of the jenkins uh g-sub project this year. um So without much further ado, let's start uh so. This is our agenda for today's presentation. I'll start with the first short personal introduction, I'm sumit sareem, so I'm the student for this project and I'm currently pursuing a bachelor's in instrumentation and control engineering uh from nsit delhi.

H

uh I started contributing to jenkins in december 2019 I made some small contributions to the fingerprints engine and that's how you know uh today uh you know so it eventually led me to taking this project over, and uh I really love it so and thank uh I'll just uh thanks for everybody, all the all the org admins mentors uh so for this project, uh oleg uh mike andrei helping me out.

H

So that's our project um so I'll first start talking about what are fingerprints actually right, so fire fingerprinting inside jenkins is just a way for dependency tracking right so say you have artifacts. You have files uh that you want to. You know basically track across jobs. uh Fingerprinting engine is what basically allows you to do that. So I just uh so like a small example say you have a team, a which builds, you know an artifact, a dot jar and there's a team b that works on b dot jar which uses a dot java.

H

So now uh you might have that someday. You know, b reports that you know we have. We are having some problems with a dot jar, and you know please fix that. So now you want to know that you know which version of a dodger. Are they actually using right uh to fix that problem? So that's how the fingerprint engine comes in. It tells you that you know they are on this version and how uh that how?

H

Basically uh you know, which version you have to fix right so uh I'll just show a small live example of this uh fingerprint engine right. So I have configured here uh basically using an instance, and I have two jobs there right just like explained uh job a and job b.

H

So yeah, so, as you can see, uh what purjabi does is basically um yeah, so it creates an artifact.txt.

H

uh It archives that artifact and it's uh it's recording those fingerprints which it gets from a dot, a dot, txt right uh and what b uh the job b does basically is that uh it copies this uh a dot, txt uh artifact from the job, a right and it just uh it. Also uh fingerprinting enabled right. So now, if I go back, uh I can you know, start a build for a and.

H

Right and then uh so, if I go to this build a right, I can see that here I have see this is c fingerprints right uh and I can see that this, a dot txt fingerprint, was actually the original owner. Is this build right and if I go click there, I can see that it was used. uh Where was it used and producing job is build number four right simple.

H

Now, if I go start a build for job b right, uh what I can see is I can go and type this uh build and see the fingerprints. I can see that this a dot txt original owner is a uh build four right, so I can see that what are the usages uh where this particular artifact was used? I started a simple uh live example of the fingerprinting engine behind jenkins um right uh now so yeah. I showed that fingerprint ui in jenkins right so I'll just skip this over.

H

um So so then we come to the problem right. What's the problem, so the current fingerprint storage engine, it's uh saving these fingerprints in xml files inside the storage of uh inside the local disk storage of jenkins right. So now this is a problem, because you know you can't configure uh p as you use cloud storages you're dependent on the local disk storage.

H

uh You know you can't create replica sets that might allow you better reliability and availability factor management is harder and plus, since these fingerprints are stored locally, you can't track them across inconsistencies right so uh to solve all these problems, uh and- and just to you know, uh show you you can see the screen also right. The code editor.

A

Yes, we can see the.

H

The code yes awesome so uh just now I had showed you an example right. So I'll just show you that in the work directory uh I have, this is what this is. The fingerprint that was actually created, it was a is build. Four b's build four, so this is actually the example find that you know runs behind the scenes um and I'll just stop this instance uh right. So that's the problem we face. So now.

H

What's the solution, so we uh are working towards building a pluggable storage so that uh basically plugin we can, you know, provide an api inside jenkins core uh which plugins can build upon, and then you know so somebody can, you know, build a mongodb plugin, a mysql plugin. They can just come in and you know these fingerprints can be stored inside these external storages. uh So that is, you know the goal of our project and this uh the entire story of uh uh you know making uh this pluggable architecture is a common.

H

uh Is something common in jenkins as we move towards the cloud native jenkins, uh and you can see these stories inside the cloud native sig. So there are stories around externalizing log, storages, externalizing, artifacts and so on and so forth right.

H

So then, uh with this project, what are achievements till now in phase one? uh So for so uh this api was released in jenkins, 2.242. uh You can see in the change log. So basically in 242 we have this api, we introduce methods, and you know so. Basically the developers can build this and it has 73. uh I good good build. So I I hope I didn't break anything, that's good!

H

uh So that's uh that's our api incentive, it's code uh and then we are building a reference implementation around it and basically this reference implementation is backed by redis. uh So so this is our uh repository. The redis fingerprint storage plugin, uh you know, go and give it. uh You know just try it out, uh give us a feedback. That would be awesome. I believe installation and configuring the plugin everything is there in the readme. So that would be awesome um right.

H

So that's what we achieved and uh so I'll now move on to the uh you know, and this uh the plugin is released at 0.1 alpha one, uh so you can download it from the experimental update center right. So I'll move on to the demo now um let me just uh configure jenkins instance.

H

I guess it should be just a few more.

C

C

Everything takes longer when you run it with zoom.

H

uh That's true, sadly, yes, so yeah, so we have an instance uh the thing about this instance and the earlier differences earlier, as I showed so now I should have uh you know I'll show you that plugin installed. So I go to manage balance and see what means.

H

H

C

H

Oh sorry, install it sorry yeah uh so yeah, so I have registering open source installed on alpha one release, all right, so um I'll just go to management. So let's go to configure system. That's how you can configure your latest instance. uh So you know you have this tab. The redistribute storage configuration you can specify host for uh if there's a necessary connection, uh that's configured on the on the connection so that this uses uh integer index databases. So you can, you know, specify a database number.

H

This connection, timeout socket timeout that you can give, and you know, if there's a credential involved. You can just uh add a credential right so and plus. If you do the test, it is which will fail right now, yeah, because I don't hover, it is instance, configured and I'll. Just do that um right. So I'll, just spin up a data server and I'll also spin up a command just.

H

So I now have a server running at port number: six, three, seven nine! So now, if I do a tester, this success, voila uh and now I'll just uh apply this and I'll. Just save awesome. Now I'll start a new job demo official project, okay,.

H

And I'll add a build step so I'll just create. You know something that this uh job will create uh once it's run or demo.txt file and I'll add a post build action that is uh record fingerprints, so I'll record demo or txt.

H

Hey, you just say: check decoder, then yeah right, so we're good I'll, just apply save right. So now, uh once I hit a build- and I hope it does not fail. Thank goodness uh so and I go to this build right. I can see the fingerprints. I can see that it's recording demo.txt and you know we have a it's. We found it in build number one. So, interestingly, now, if I go here, you'll see that in this work directory I don't have any fingerprints folder, because these fingerprints now went to the greatest instance.

H

And how can we verify that uh so yeah? So this particular key is for this particular fingerprint and now I have that exact same fingerprint information stored inside that. So that's pretty much it about further demo slides.

H

Yeah, so that is the level. So what's next um we plan to extend the api. You know the more methods, so we need to. You know uh we need to extend this api to implement tracing across instances, that's something for the next phases, so this fingerprint cleanup so currently uh in the local instances of the then in the local xml based uh storage.

H

What happens is that uh those fingerprints which whose builds are no longer present on the system, are cleaned up on a periodic basis, but uh so we need to uh you know, uh have this functionality, for our external storage is also migration. So currently you know if a user installs this and has you know uh already present fingerprints inside the local instance they won't get transferred at the moment. So that is something they're gonna work on savable listener is something uh it's a it's a it's a. Basically, it triggers something around.

H

uh It basically triggers itself when some changes happen, and you know uh that is something we might want to. uh You know do as a next step uh to support uh virtual files, or you know so that might need a rebirth and uh tracing. As I explained across you know, tracking these fingerprints across instances uh and before the q and a I have these links at the end of my slide. So you know uh that would be awesome if uh you just you know, give this plugin a shot.

H

Let us know any issues come up and there's a greater channel, also link. So you can find me there always and I'll be happy to help uh so I'll, just open the floor up to any q. A uh thanks a lot everybody for having me.

A

All right, thank you, questions regarding this project.

A

I know I have a question.

A

No one has takes the floor. Okay, my question is: when you store the fingerprint in the database, is there links back to the instances that consume the fingerprint.

H

So by instances are you referring to the the particular jenkins instance, which created that fingerprint.

A

Yes to the one that created the fingerprint, but also to the other jenkins instances which may consume that fingerprint.

H

Right so we have so. What we do is basically because these fingerprints are id'd by uh md5 hashes. uh So we have to ensure that you know an instance. Id is also associated with them, so uh to maintain all the current functionality. Yes, we do have an attached instance id uh whenever we save it in the redis instance, but uh the api does not enforce anything.

H

So the api is built such that it's up to the plug-in developers to decide how they decide to store those fingerprints, but as far as our reference implementation of radius goes. Yes, we do still.

A

Okay, I was asking in the context of cleaning up the fingerprints, so if they are consumed on multiple jenkins instances, it's.

A

If you have two jenkins instances consuming fingerprints, then you cannot run the cleaner on on one without consulting the other jenkins to see if they, if it has references to the fingerprints. That's the context of my question.

H

Something to think.

A

H

I guess no, I got your question, so uh that's not something that will happen because uh we saw basically there's a you can think of it like as a barrier, uh so the fingerprints are pretty much isolated on an instance level. uh When we talk about tracing uh we we have not built tracing yet so uh that might come in the form of a separate plugin. As far as our current implementation goes.

H

If, if say fingerprint in jenkins instance, one is cleaning up fingerprints, it will only clean its own fingerprints, it won't touch the fingerprints of other instances, so they are isolated in a sense.

A

Okay, that's clear, thank you any other questions or comments from the mentors for this project. Oleg andrei michael.

C

uh Yeah, I would uh just like to thank uh sumit for the hard work on this project, uh because externalizing fingerprints is one of the items required for providing full pluggable storage in the jinx project.

C

It's really important for future architectures for running can say in the cloud, especially in public clouds, um and this project helps us to move towards this capability in jenkins.

C

Also, I would like to say thank sumit for contributing to the jenkins core, because this project includes a lot of complexity, because we operating basically in with a low level uh code base of the uh jenkins core. So there is a lot of uh various uh aspects like computability like performance and also retaining apis and make this apis efficient for existing use cases without breaking binary compatibility, so also it was able to study this part of the code base and to quickly start contributing to that.

C

Moreover, we have a jenkins enhancement proposal which documents the specification, for example, fingerprint storage, and I believe that it's a really great progress and I'm looking forward to see what we will achieve uh during the next phases.

C

So thanks a lot uh for your work on this project.

H

H

Without your help and guidance, thank you.

G

Yeah, I just want you to comment. I mean uh you know when you know someone had raised the point about it being uh part of the cloud native, so I guess that is uh in itself. You know very next step that you know uh it's today becoming of real importance to be actually becoming part of the cloud native part, so I mean yeah. I really loved the entire idea of the of the plugin.

H

Thanks if you are interested in you know cloud native stories, so you can go ahead to this. Actually, I did provide a link somewhere. Yes, so there are a lot of uh pluggable storages uh other stories, also in this cloud native side. So I would welcome you there also thank you.

A

All right any other question regarding this project.

A

Okay, let's uh conclude the session today, thank you for releasing the share, so here we are, and so this was our last presentation today, um sister. This was part two of the project, demos. If you're looking for more resources on the jenkins google summer of code website, we have the link.

F

A

There are recordings about the jsoc meetings on our youtube channel and you can also visit our blogs. They, the tag for the jsoc blogs are jsoc, is jsoc actually and these slides here's the super, long link to these slides and it's um it's. It's truly wonderful to see that um people are enthusiastic about contributing to jenkins during the summer of code program. We have other programs that jenkins the jenkins community participates in. So please visit our website for more on that.

A

So on this, I want to thank again all our students and our mentors for their great work and for your participation in this open source project, and I wish you all the best for coding phase two all right and the link to these slides, here's the short url to these slides. If you need to see them offline and on this. Thank you very much and have a great rest of the day. Everyone.

D

Thank you also, and thank you to our students, wonderful.

C

Thanks all and yeah thanks to mark and martin for hosting the gsoc demos, and it's much appreciated and thanks to everyone who contributed to the projects, whether you uh student, mentor or just contributor interested in these stories.

C

It's an effort which includes dozens of contributors and it's much appreciated.

B

Awesome I am going to stop the recording.