GitLab Package Group, 28 Sep 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Importing NPM packages to GitLab with a CI pipeline

Description

Artifactory import investigation: https://gitlab.com/gitlab-org/gitlab/-/issues/298726.
Importing packages with a CI pipeline idea: https://gitlab.com/gitlab-org/gitlab/-/issues/298726#note_1078330418.

A

So hello, everyone, I'm David from the package team and in today's video I'm, going to present a demo of an idea uh that I encounter, which is importing packages to the gitlab package registry using a CI pipeline. So I'm going to share my screen uh here.

A

Okay um is the phone size? Okay? Is it readable? Okay? So um we do have an ongoing investigation about importing um packages from artifactory. Artifactory is a third party package registry, where you can post many different package formats and we have uh quite an extra amount of customers asking for a way to import those packages to the gitlab package with history.

A

um While doing this investigation, I was wondering um well uh some package formats.

A

um They have apis to query the state of the registry, meaning that you get the list of all the available packages and their versions, and we do have the same in the gitlab package registry.

A

So could we have a script um that will look at both States, build a div and then actually trigger another pipeline where one job will do one operation, which is actually either adding a new package or a removing one so that at the end of all of these operations, the state in the package, uh the state in the gitlab package registry is the like the exact same one. Then the the one of the registry that we were targeting.

A

So the short answer is yes, we can do that, but we are going to uh dip a bit into details. So um uh I created um I created a ruby script uh to help me with this. uh This script is basically built around a configuration where we Define a Target registry and uh well actually a source registry and the Target registry and the whole script has commands to build the the diff between the source and the Target and then build all the the um jobs for the children pipelines for for all the operations.

A

um So we are going to use that um the other thing I'm going to use is the dynamic child pipelines that we have on CI pipelines, so Dynamic pipelines is, um is a tool where, instead of having a static, yaml file with the configuration we can have a script that will build the configuration of the child Pipeline with all the jobs we want and then that this Dynamic config will be used for the child per plan.

A

um Yeah. It might be confusing now, but it will be uh clearer on the in the UI, so um I have here the no. This is the script. Here is my uh project with the configuration where all the pipelines are running, um I decided to have the Target registry on a different project.

A

It's here, so we can see. If we go to the package registry, we have a bunch of packages.

A

And that's okay, um but the the jobs are. uh The pipelines are executed on this job. So it's like. We have a job that is the orchestrator and, uh and we have a Target package registry which is in a different project.

A

It could be the same that could work too, but I wanted to show that we can Target a different uh package registry, so um yeah I'm, going to show the the configuration of the import, a quick note, I'm using gitlab.com here so I'm, not using my local gitlab instance, I'm, not using any feature that um I added locally or something that is under development. It's just Center things that are already available, um so we are.

A

The static configuration of the of the pipeline, so this one is basically pulling the the script from the first project that I show and set up the script and just starting the script and and defining the the files that will trigger the dynamic child pipelines so basically uh uh yeah. We can ignore this step. This is just pulling the the gem or the script and and doing some stuff for setup.

A

uh The main one is this one: so basically, we generate a div as I said between a source package registry and the target one, and we are going to generate two pipeline file, not one actually, two one for all the the operations that add the package and all the operations that remove a package um we declare those files are artifacts and here are.

A

The here are the the triggers that, on those files will create the dynamic Child part time, so we will have not one but actually two one for adding packages and one for removing packages um yeah.

A

uh So, let's see the configuration of the import, um the script since I I worked just a few hours on that is coped on npm packages. Only for now and the Target registry is only git lab, but uh we could imagine to have different Target types but yeah for now it's on big Islam, so uh to keep things super simple, this the source package, which history is actually npmgs directly. It's not a private registry.

A

uh uh The target one is gitlab with the with the project ID, since we are targeting a different package registry on a different project, and then this is some configuration for for the child pipelines. We can just keep that the most important thing is this one. um This declares that we want.

A

We actually read the state of npmgs and this defines additional filters on that state, because, obviously we don't want to import all npmgs on that project. It would be perhaps too many packages, so we have additional filters to select which npm packages we want from which version and yeah I implemented the uh like. We, we can specify a range of versions, um so these ones are actually the exact packages that are currently living here. uh You can see npm no uh Redwood, so uh they are there. uh So I'm going to update this configuration.

A

uh Let's say that we remove uh this one uh and we are going to add a new one, so we are going to go to npmgs.org.

A

Do you have a preference for any npm package can be anything.

A

No okay, um yeah, let's um I, think it's you or ugs, no I, think it's you yeah! So we go here. We see all the versions history, let's say that we want from this version and all the above that one. So we want it's real right. It's not huge! Yes, yeah! It's you, the name so view from this version, so I'm going to commit this change. This will trigger the pipeline and that pipeline will execute the script.

A

Compare the states apply these filters that we just defined, uh generate a div and then generate the dynamic child. Pipelines.

A

A

Yeah it's here, so um this is the pipeline running, so we have the pulling the discrete and and doing some setup. This is the main uh task where we are going to generate the the div of the states of the two package registries and then out of this div, we are going to generate two uh uh two, two yaml configs for two pipelines, one for adding packages and one for removing packages and those jobs are waiting for those files to be created so for now the they will not get uh they're in waiting mode.

A

If you want um what we should have is that um we removed it. Was uh uh you know it's actually this one? So I can open it.

A

While it's running we'll remove uh it was bubble, call yeah bubble, call should be all bubble. Tall packages should be removed and and packages for view. Ges should be added.

A

Yeah, it's running the diff. um Actually, the this the div command will output a debug login that will show all the add and remove operations that will then be used for the dynamic uh children. Pipelines.

A

Yeah, that's it. um So here is the summary. We are going to remove all the bubble core packages and we are going to add all the view. Npm packages um and yeah you can. We can say that the the CI yaml file for adding packages has been written and the same for removing it's here. So now that we have those two files, the triggers yeah I need to refresh the page. The triggers have been executed and those are two new pipelines that are children pipelines of this one.

A

And if we expand it, we can see that we have all the remove jobs. So each job is removing one package. In this case it's removing one bubble, call version, and here we have the same, but for adding so each one will add a version of The View package.

A

So you can see the nice thing with um splitting the important steps and and make a single CI job execute a single step.

A

This brings a lot of visibility on the import operation, um because each step has its own job with its status, and we can actually imagine that one of those jobs fails. We can actually retry it uh uh or we can actually see like very, very quickly which one fail uh because well here they all pass. But if you have a job that fails, you will have um a new tabs here with failed jobs, and so just clicking on that tab. You have a summary of which packages which steps of the import failed.

A

uh So it's interesting the amount of visibility that we have on the the whole import operation that we are for free just by using CI pipelines.

A

um We are yeah. We are going to verify that yeah. It worked as expected, so the package registry um well I'm not going to search for view since they're there, and if we look for bubble call they shouldn't be there yeah, it's not there anymore. So we just um we just have the the exhaust. The exact package registry state that we wanted and that we defined from the uh config file that is here from this. This configuration um that's good.

A

The other thing that is interesting by using a pipeline is that this pipeline was triggered by a change on the on the config file, but we have some users that wanted not only importing packages, but they won't um like a step further, which is mirroring.

A

So actually you have some changes in the source package registry and- and you want those changes to be reflected in gitlab and actually mirroring. We can do this very easily if we have a pipeline because we can have scheduled pipeline pipelines so pipelines that runs on a defined schedule like I, don't know each hour or a daily Pipeline, and and so if we let's say that we run this pipeline nightly well, this python will check on every night.

A

It will check the state of the Target The Source package registry uh generate the blah blah blah generate the the operations that are necessary to bring the Target registry. In this case, the gitlab package Regency to the to the state that we configured in the config file, so that's another bonus of using pipelines- is that mirroring is just one setup away from happening. We just have to set up a um uh a scheduled pipeline that could be recurrent so could be implemented very easily.

A

uh So a lot of benefits of this approach, um one benefit that we didn't talk about- is that these jobs are potentially executed in parallel. So it's not like we are executing them in sequence, we get some parallelism in the input steps and they are all independent. So if one job fails, that doesn't mean that another another job will fail too, like they are not related at all.

A

It's each one, it's its own job, its own world, so we have a more reliable import if we want uh so that's good um on the downsides, because I'm selling this idea way too much. So we have to talk about the downsides right.

A

The downsides um I would say. uh The challenge is to really have a way to get the state to the list of all available packages from The Source package registry, and this is not something you can do for all package formats.

A

um I I choose to start with npm, because I knew that there is an API that is defined by the the like npm client uh API, with package Registries. There is an endpoint that will list all the packages with all the available versions, so for npm is, is possible, for others could be more challenging to to get that.

A

That would be if we can have an example.

A

Yeah, if I I'm, not saying something wrong, not get packages would be more challenging because on nuggets um you queried the package registry with a given package name. So you you ask the package registry: hey, give me all the versions that you know about this package, but in this case for the import.

A

What we really need is hey give me all the packages that you know and all the versions that you know so could be challenging for some package formats, um but yeah um that would that would be to be that that would need to be worked on on a on a case-by-case basis. I guess uh there are other package formats where they don't work with um uh package artifacts or package files, such as composer composer packages, are actually git tags.

A

So it's not exactly the same as pulling a file from from somewhere and uploading that file anywhere else. It's not exactly that so um yeah I'm I have some doubts that this approach can be applied to all package formats, but it could be worth to expand the list of formats that we could support with this, and the other downside I had in mind.

A

uh Yeah we we would need to to update the script to support. Obviously, we need to support private um Registries, which means we need to support credentials so, for example, artifactory, like a user, want to import all the npm packages from RT Factory, we will need to support the credential. So there is a bit of of work to improve things, but I would say that it's it's doable.

A

And lastly, yeah one benefit I'd like to uh list here is that we could have um a project in in Gate lab so like this one that is importing or moving packages from package Registries that don't live in git lab. This is I, don't know strange a strange use case or weird, but we could imagine having a gitlab project like this one moving packages from I don't know artifactory to something else, yeah. Why not? That would be possible.

A

We could also imagine this project moving packages between two other gitlab instances. um Why not? That would be possible. Imagine that we have to self-manage gitlab instances and somehow they want to move all the packages from one instance to a different one. We could have this project uh handling that move or import or mirroring, depending on what the the users want. That would be possible, so uh I would say that it gives a lot of flexibility using this.

A

This approach, which is which is nice, um okay, uh yeah, I, guess this is the end of uh the demo. I, don't have anything else um to show I, don't know. If you have questions.

A

No not really but I mean it looks really great and I think it opens some opportunities so, as you described, yeah um yeah I agree with that, um and- and it opens like it's due to its flexibility, we we could even have more opportunities to use these these.

A

um This approach on, like some weird scenarios like the ones I, was discussing.

A

uh Okay. Well, if you don't have any questions, um I guess this will be the end of the of the demo and I will stop the recording. So thanks for watching I will uh um add all the relevant links to the video, so it will be somewhere. The video description and I'll see you on the next one.

A