Jenkins Google Summer of Code Office Hours, 30 Aug 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: GSOC 2020 - Machine Learning Plugins for Data Science project demo

Description

Presentation by Loghi Perinpanayagam, a GSoC 2020 student. The main goal of this project is integrating Machine Learning workflow including Data preprocessing, Model Training, Evaluation and Prediction with Jenkins build tasks. This plugin will be capable of executing code fragments via IPython kernel as currently supported by Jupyter Notebook. Kernels which are already installed can be configured for each build step and dumping visuals is an added feature in the plugin.

Machine Learning has evolved rapidly in the software industry for recent years. Jenkins CD/CI can be a good practice to deliver a high reliable product in the end. Machine Learning plugin can be used to build Jupyter Notebooks and script files with proper kernel configurations. In addition, the build wrappers could be used to convert Jupyter Notebooks to python/JSON and/or copy the files to the workspace for more actions.This Machine Learning plugin will endeavour to satisfy the data science community together with the help of other plugins. Success of this plugin will definitely serve much benefits to the community and Jenkins.

References:
* Project page: https://www.jenkins.io/projects/gsoc/2020/projects/machine-learning
* Plugin page: https://plugins.jenkins.io/machine-learning/
* Repository: https://github.com/jenkinsci/machine-learning-plugin

A

Welcome to my presentation on jenkins machine learning, plugin, uh I'm so excited to present my work. uh I have done for this uh jitsock and also. I would like to thank my mentors bruno ironic, marquis and shivai for a great feedbacks and silences and also thankful for our jenkins, or that means that who makes this event successful.

A

So let me introduce myself, uh I'm I'm just student for machine learning, plug-in jenkins for 2020 and I'm doing my major in computer, science and engineering. So before we move into this uh technical aspects of the machine learning plugin, uh I would like to explain a little bit about what is machine learning operations. So if you, if you take any machine learning operations uh continuously continuous analyze and continuously training, is a very essential part to improve this quality of these machine learning products.

A

So, uh as you see in the diagram uh data scientists will analyze the data sets and then they build their model and train their model and finally, they will hand it to the devops engineers and they will test and deploy and they will do the rest of uh production works using the application code.

A

So, uh let's let me explain about uh how it's possible, uh how they are achieving using this uh using the plugins of the jenkins in the uh in the ml operations. So if you take one machine learning operations, there must be uh multiple steps like uh training, pre-processing uh and validation. So there's a lot of intermediate steps. Are there so for the clustering tools?

A

You can make it possible to use one of those or docker plug-in or a club kubernetes plug-in, so there's a lots of clustering process available in uh jenkins and also if you want to manage your source codes, uh you have a jig plugin and if, if, if you have a large size, large file size like uh large data sets uh large models, you have to manage that also. So we jenkins has some advanced plugins, like s3 publishers.

A

You can easily publish your models datasets to this aws s3, and if you want to uh analysis, if you want to submit your analysis and if you want to publish your reports in somewhere, you have some analytic plugins, also available in the jenkins and jupiter. Notebook is widely used by data scientists to uh to analysis and training models. So, in addition, uh we are giving this uh jenkins machine learning plugin uh to leverage the all the plugins to make the ml operations successful.

A

So let me explain uh what are the features we have implemented during this gsoft period? So the first one is a very data science community friendly configuration about uh about the kernels. So if you want to check uh what are the kernels you have installed in your system, you can just go to your terminal or your command prompt. You can just type jupyter kind of kernel spec list, so it will uh display all the kernels you have installed in your system, so I will explain a little more when we're doing the demo.

A

How to do that and uh let me pass to the next slide- and we have this wrapper included in our plugin- uh that will helpful uh for converting your python or converting your normal books to the python or json, or if you want to rename this notebook, you can do with this wrapper. You couldn't plug it, and this is our main feature of this plugin. So we have to.

A

We have to give the ml task name, which will be helpful. If you have any artifacts like uh images html generated by the notebook, it will be saved under the name uh like, for example, it will be saved under the training folder. So it will it it's the purpose of this uh giving machine learning task name and the next one is uh you have to uh you- have to select one of kernels.

A

You have configured in the kernel configuration so, for example, I have selected for this short as a javascript and- and you can give this uh code or if you can write the code directly or you can uh give the notebooks notebooks name or a script name in the file processor.

A

So uh so, finally, uh we have a features like if you have any uh any images or html generated in the jupyter notebook, uh it will be saved under workspace. So there's a action in the left panel. You can see image and html, you can click it and, and you can view what are the images generated in the jupyter notebook.

A

So we will see a very detailed in the demo so yeah I will uh go to the demo directly.

A

So before we going demo, uh I have a github repository which contains some uh csv data sets and some uh python enabled notebooks and are enabled notebooks. So for this demo, I'm going to I'm going to uh run this uh python enabled notebook and our notebook.

A

So you can see here. This is our notebook. You can see uh they have or they have used our codes here. So there's a por code, so they have using a gtp plot to make the graphs in this notebook and the other one is uh the issue. It's using the python code directly. You can see that so for graphing we are using the matplot library, so yeah. So first we going to create.

A

uh uh We are going to configure our kernels here, so you can go to uh configure systems and then you can find uh you can find kernel configurations here. So you can just add new kernel here and I will tell you how to uh how to uh how to find what are the kernels have been installed in your system. You can just type jupyter kernel spec list, so uh these are the uh list of kernels that I have that I have installed in my system.

A

So I just choosing a python 3, and there are so I just configuring my item, 3 language, I just typed by them.

A

Sorry, uh for what is uh so the launching time mode I just setting as a default 10. Second, uh so it takes time uh to connect it. The launching time mode is uh how long it would take to connect to this uh kernel from the jenkins and and the max results is uh how many lines can be uh sent back to these agencies from the kernel.

A

So you can here have a test connection button. You can test your uh kernels from uh from here, so so I'm adding also a new kernel for ir like we have. We have seen that there is ayaka so for this r. So I'm just setting this hundred and I'm testing this uh kernel here.

A

So yeah we are good to go with this kernel configurations, so so we're gonna create a simple freestyle job for this. uh For this demo.

A

Yeah, it's a final okay, just collecting this uh yeah. So for that uh I'm going to check out my repository here. So I'm just getting my repository.

A

Click here I just open them and for building these notebooks. uh This is the essential part of this uh plugin. You have to give. If you want, if it's training, you can just add training and you can select either one of these, so we are going to run here. uh Python enabled notebook, so I just uh selecting my python notebook, okay, the name, so I just giving the name of the uh python because it is already accessed it. Has.

A

It will be checked out from the github to jenkins workspace, so you can directly use the name of this notebook and uh also there's- I will add another builder pro. If so I just and I'm selecting so for the file. uh I just uh giving the name of our notebooks.

A

Yeah so yeah, we are all good to go with this job configuration. So we just one step left. uh I will run that.

A

So here uh you can see uh this. uh This is a checkout console and you can see. There's htmls added to the training, uh folder and there's a lots of images have been added to this training. So if you want to uh check whether it is added to this workspace, you can just check here and the generated images and the other arc plots are here, so you can even uh try to view your html generated in the notebook yeah.

A

So if you want to publish this uh results to somewhere, you can use html publisher, plugin, oh you can use. You can leverage the other plugins to make the result viewable in somewhere, yeah, so yeah. So that's the demo part I'll be shifting to this uh uh main presentation.

A

So uh this is the link to our uh github repository and if you feel any issues about our plugin, uh uh I forgot to say that uh we have released our plugin uh version 1.0.1. You can directly uh download from the main update center. So if you, if you when you are using, if you feel any issues, if you feel any errors, you can report it here.

A

We are happy to help you and if you have any uh any questions, you can ask directly in our jitter channel dedicated 4g submission learning project, and this is my demo repository. uh You can use it if you have. If you want to try that demo I have shown before, and I will show you. uh I will show you uh what are the kernels available in the jupiter kernel project, so you can see if you want to run a julia code. If you want to run haskell haskell.

A

If you want to run ruby, you can just install this kernels in your system, and you have to configure that in your jenkins and you can directly use that. So that's the best part of this machine learning, plugin and yeah.

A

So uh for future future improvements uh we are trying to implement. uh We will try to implement the issue like uh at this moment. We have to give this uh the kernel features and we have to type it manually. So if we in, if we, if we, if you implemented this automatically adding a label to the agent based on the kernel features, we don't have to type that we can have a drop down list or something like that or we can uh leverage that thing to easily uh configure our kind of kernels.

A

So that's the thing we are concerning right now and uh and the video we were also struggling when we uh testing this code testing this plugin, because uh everything is uh in the back. There's lots of python work, so uh we also expecting to write more tests to the plugin and in ml operations uh we have to. uh Can we have to concern about the performance of the plugin, because uh performance is a one of the uh essential thing in the ml operations?

A

So that's the thing we have. We are concerning right now and yeah. So uh thank you once again. Thank you for joining with with this presentation and thanks for my mentors and thanks for martin to hosting this presentation and yeah.

B

Thank you, logie. um Do we have comments or questions yeah.

C

Not that I see so far in the chat. So if anyone has any questions, please ask using zoomcon a and we will ask presenters and in a while, while we wait for questions um if mentors or other parents, so would like to give some comments.

C

D

Well like, if I may comment, this is janine's muchachos. I was one of the mentors and um you know we have.

D

It's uh uh work now um twice last year and this year and finally, we succeeded- and I think it has uh progressed quite uh well. Loggie was a very uh excellent student and he was able to um bring together a lot of technologies that are required for this particular plugin, which is a little bit challenging. But uh you know he rose to the challenge and I think we have a really nice version. One, oh now, that a lot of us really, uh you know as users now are excited and waiting to to use.

D

So um I am really thankful to the gsoc um program for making this a reality and- and I think, we're moving into a great area uh with new jenkins applications.

E

Thanks for watching yeah- and uh I mean like it's fine I'll also just like to you know, speak a couple of uh votes for louis, because you know it has been really amazing. You know working with loki on this project because you know first of all, working on it from scratch. You know and logie you know from the start. You know he was always you know up to the task and always completing all of the tasks before you know uh the deadlines, and that was you know, probably the most.

E

uh You know the best uh you know work that we saw was, you know, always maintain that consistency and also coming up with you know great solutions for solving all the different kind of issues that we run into with the with the plugin, and uh there were also instances you know where we did take in some. uh You know: suggestions from external people as well, and those are also implemented- and probably you know like one of the best features at the end, was to add multi-language support.

E

You know not just python, but also the other kind of you know: data science languages that people will be using a lot of them. We are using, are julia. So having that you know, integration as well within this plug-in makes it so accessible for all different kind of you know: data science related uh work, workflows right, so it was, you know like a tree to be working with logi and all the other.

E

You know uh mentors as well, both ionis marquee and starting- and um you know bruno so I mean in all you know, uh loki made it completely. You know successful and I'm, like so proud of. You know whatever work he has done for the plug-in.

A

Yeah yeah. Thank you. Thanks for your events, thank you.

F

I would just like to uh echo what everybody else has said and also just say: you're an amazing person. You not only put together a very futuristic plug in here, but you did it during a pandemic, which is no, you know, that's a pretty big deal. So thank you and thank you to all the other mentors.

B

Great thanks for all your great comments, everybody in let's move on.