GitLab Package Group, 8 Jun 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Proof of concept: CI pipeline for packages

Description

Related issue: https://gitlab.com/gitlab-org/gitlab/-/issues/324196

A

Hello, this is david from the package team at gitlab and in today I wanted to show you a proof of concept that I've been working on. So let me share my screen.

A

Okay, so the idea is quite simple: we can trigger pipelines when git push is committed to a git repository and the idea is what, if we can trigger a pipeline when a package is pushed or uploaded to a package registry.

A

So I wanted to explore this idea- and I created a small demo for that. So I'm going to show you that first world of caution, what you are going to see is uh still work in progress. It's not final and yeah. Some changes uh are not meant to be uh used in production. Yet so it's more an exploration of this idea.

A

So let's say that we are a company where we have three projects and those projects create. Each of them creates a npm package that is pushed to a fourth project that is public, so this package registry is public. Anyone can access, but the code source of each project is private.

A

Note that in this case, those three projects and the package registry are in the same place, which is my gitlab instance, but they could be separate, meaning that the packages could be built anywhere else and they are pushed to gitlab.

A

Now, let's say that I am a a developer that I want to use those three libraries, so here is my npm application.

A

I set up the registry for the scope of those libraries pointing to my gitlab instance, and so I'm going to try to install those three.

A

A

Boom, it doesn't work, so one of them is missing, so we can't pull it from the package registry.

A

At this point, I could write an issue on the issue tracker for this company and now let's say that I'm a developer or devops um from this company- and I read that issue, so I know that there is a problem with this package, so the first thing uh we will check is going to the package registry, where we publish all those packages.

A

And in this screen um there is a change from the standard one.

A

There is a pipeline status and we can see that yeah for the package bananas. We have an issue on the pipeline, so we are going to open that pipeline.

A

And we see that we have some validation jobs and all of them are failing. So let's try to fix that.

A

So I'm going to do it right from the gitlab instance, so I will open the project and simply open the web editor.

A

So let's do this one by one and check each job.

A

So the first one called license. It's a simple one: it's simply checking that the license is the mit one.

A

So it seems that it's not correct nope, so we are going to fix that.

A

Let me grab the mit license text.

A

A

Okay, so that's our first check: let's see the other jobs, the second one oops, the jobs of the pipeline, not all the jobs.

A

Okay, so in this one we are running a linter on the package.json file to require some fields so that it's compliant to our needs, and we can see here that the author is required.

A

Let's check the package json file yeah, the author is not there, so let me fix that. Also.

A

It's me, of course, wait. That's true. Let's see.

A

A

Npm audit so npm audit, it's actually a command that is run when you install a if you not. If you don't disable it, it is checking the dependencies of the package and checking for vulnerabilities.

A

So we can see here that there is a dependency on low dash and this version has a high severity.

A

Let's open the package json again yeah, we do have a dependency on lowdash and let's fix that with pulling the last version.

A

Okay, um that's good! Moving on the last job, so the last job is a special one. It's running a.

A

Software being developed at gitlab for checking what happens when the package is installed, and in this case we can see that it's failing, because there is a outbound connection that is made during the installation of this package. So this is some something super fishy and we don't want to publish packages with fishy scripts.

A

So, let's review the package json file again and with close attention. We can see that there is a post install script and that script is trying to.

A

Send a environment variable value to an external server: that's bad! Okay! So let's remove that.

A

Okay, I think we're good. So we are going to publish this version since it's the npm registry, we can't publish the same version twice, so we are going to bump this number and commit the changes.

A

I'm going to come into the main branch which you shouldn't do you should use, merge requests, but in this case I'm going to use that shortcut fix the failing package pipeline.

A

I'm going to commit this change, so what I will have here is that this project has a pipeline on git pushes so open this commit. It will create a pipeline that will take the call tools and build the package, and then it will publish the package to the public package registry and once the package is uploaded, a second pipeline is triggered there, which is the package pipeline to verify the the package.

A

So, let's see, if we fix the issue, okay, we have our pipeline working this one. It is building the package and deploying the package to the package registry.

A

Okay, good now we go back to the public package registry and yeah. We can see that there is a pipeline running for this version. This new version we created and we find our four jobs.

A

Three of them are already green. Let's see the last one.

A

Succeeded so it's green, I guess we have a green pipeline.

A

Yeah we do have a green pipeline. So now I can reply back to that issue. Hey we fixed the package. We published a new version. You can pull that version again. So let's switch back to the developer, trying to install those.

A

A

And it worked so we have those three packages installed to the npm in into the npm application.

A

Okay, so uh to summarize, uh we can see that having pipelines for packages can make a lot of sense to validate the package and to run some jobs to make sure that the package is properly set up and the files is are properly configured. The way we want like the license is the mit one.

A

We do want some metadata on the package json file and we can even have some security aspects checked, which is the the fourth job running package hunter from gitlab and- and there is a lot of things we could do by having jobs on packages.

A

I guess that's it for this video. I hope you enjoyed it and I will post the issue link in the description of this video. So if you want to tune in and post a comment feel free to do so.

A