GitLab MLOps, 21 Feb 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: MLOps SEG Update Feb 21 2022

Description

Today we talk about Citer, a POC we are working on to make the experience of iterating on pipelines within GitLab faster

A

Hello, everyone and welcome to another update for the single junior group uh for mlaps. My name is eduardo. Today is the 21st of february and today we'll be talking a little bit more about pipelines and iterating on creating pipelines.

A

So every engineer everyone that has worked with creating a pipeline knows this experience.

A

To create a pipeline, usually it's a try and an error until you get it right and each try and error step takes quite a while so usually make a small change.

A

Commit push open, gitlab find the pipeline wait until it completes find that there's another error and then go back to one and then make a small change, commit and push open the gitlab, and this takes quite a while um opening pipeline and so on and so forth, and we want to see if we can make it a little bit faster uh to do that to by avoiding git entirely on this process by avoiding uh commit and push by avoiding open, git lab by avoiding fighting pipeline.

A

So why is this important to email ups? um It is important to every devops engineer everybody working with cicd, but there is a gitlab use case for pipelines that goes beyond ci. Cd within mlaps pipelines are part of the create stage.

A

So when you think about creating a machine learning model, you can think about training the model uh the same way as you think about compile, compiling a an app or a software.

A

So, but is a compilation time that takes that can take really long, and sometimes you need to run this on a different machine so usually for data science. Applications pipelines are part already when you are creating the model before you deploy before anything else, you're already setting up your training, your your your, uh sometimes your even your pressure type on top of pipelines and depending on the git flow for this makes experience a lot of lots lower than it could be.

A

So we are setting off here to try out a new idea. uh I've been working on a poc for uh citer, which is ci iteration, which is how to make cia. How can you use these pipelines uh that we already have for the create stage without, depending on uh the on the gita, on git commands or on git itself? So let me show you what I'm talking about. So let me show my terminal- and here I am- uh I am on a folder that is not on git.

A

This is not on any repository at all, so suppose that I have this file over here. Cat.

A

A

Yeah, so let me make it wrong uh so that I can show what I'm talking about. So, let's remove this line. This uh make a type over here. So now I have this new ci uh file that is not on git at all. It's not on any repo. It's not on git lab. It just lives on my machine and I want to test if it works or not. I set up a repository which will show a little bit later uh that I can now just run citer minus watch for watching the pipeline execution.

A

I will pass a variable test. Citer hello equals world and I will try to catch that file so, like I said we had, I made a mistake, but I want to show the hello there hello. There is also not on git hello. There is a local file and then I pass my gitlab side of key that I want to test it out and citer will pull. I will upload. The files will create the pipeline for us and we'll start to watch the execution.

A

This will take about 20-30 seconds to complete because it still needs to create the runners and so forth. I'm using dynamic uh parent child uh dynamic burn style pipeline to get this working, and I will do another video explaining the technolo that is in a technical way. uh This is more of an overview of what I'm doing, but it's running it's running it's running and it failed. uh I can see why it failed.

A

uh This is something I want to improve, but for now I need to click on the pipeline, which is already there, uh and then I can see on the pipeline why it failed. um Well, no such fire directory us well, that's because we did it wrong, so I can just uh I can just go over here.

A

uh And then now, if I were on the regular flow, I would have to commit push weight, but now I can just run the same command again.

A

I can just run the same thing again and it will reupload and you create another pipeline, and you will start watching uh the execution of the pipeline. It will take another 30 seconds over here until you get it done, um so you can already see how this makes up a lot faster, and you can think this right now. We are running this on the command line right. We run this as a bash, but think about this on, for example, our vs code plugin or our jetblinks plugin. You are working on it.

A

You can run the gitlab ci no need to open gitlab at all. uh Okay, here, like I said we have success now, so this doesn't. This makes a lot faster to do it. I can be on on my vs code and just run this, and I can just have that done so, just a quick overview of of how this is this achieved. Let me hide my uh my terminal uh and then okay, let me open up over here.

A

For example, you see on this one uh that it was completed you're here on the test pipeline frankly completed. It shows the the message, so it called the hello there. uh It showed the content of the file as we expected um so yeah, just a quick overview of how this happens. It's two parts, one there's a cli. uh There is a tool that calls uh everything, but for this to work it needs a pre-configured uh ci uh repository.

A

So this already has a gitlab ci that uses dynamic, parent-child pipelines and does all the things necessary for that for the magic to happen. uh It just needs to fork this set up some um some environmental variables and you're good to go about using cipher.

A

um So let's go back over here so like I said it uses two uh ripples for this.

A

The good thing about this is that it doesn't touch uh the code base at all. This, doesn't I didn't have to add anything into gitlab. I've just used the apis that are available.

A

uh However, I used some apis that uh are not designed for this. Let's say, for example, for me to be able to upload the files that are not on the repo for the hello there file. For example, I use the issue upload api yeah, so you can upload files to your issues to create attachment attachment files uh like images or anything. You can do this through api, and I use that to upload the files uh for the for for this, and then I download these files.

A

When I- and I I I extract these files when running the child pipeline, so there are things that uh well, not I'm not proud about it works, uh but this is not something that should be used on uh on scale or anything. It's it's very. It's a proof of concept of what can be done, but it already shows. How can this experience be?

A

Be a lot easier and a lot faster? For example, one thing that we can improve upon right here is instead of waiting uh when we show when you see that it failed. Instead of having the user go and click at the pipeline and move over there, we can just use the jobs api and show the output of the job that failed, for example. That would make a lot faster and then I don't need even to open gitlab at no point of this whole process. That's a very exciting thing to achieve.

A

Another downside here is that it requires a maintainer token. So for me to trigger uh pipelines, I need a maintainer token, and sometimes it's not a token. You want to share- or I don't know create for everyone and not everyone can create on most places, um but it could be something we can work within gitlab itself, allowing this within gitlab not only not requiring this workaround that I'm creating over here, but something a little bit more. Let's say official.

A

So that is what I had for citer. Some other updates um swapping between raw and clean diffs for jupiter notebook is already in the mr and it's going to be hitting 14.9 for sure at the very least, the code for the commit page soon. This will be on the mr page as well, but this will be hitting uh production very soon and we are starting a book club uh for for different engineers in different uh groups within the company, uh we'll be reading the book hands-on machine learning.

A

uh So if you were interested about that go into the issue uh and join us for this book club, and that's all of the updates that I had thanks for watching and have a good one.