GitLab Package Registry Group, 3 Feb 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Packages Importer

Description

Packages Importer CLI app from the Package Registry team.

Project: https://gitlab.com/gitlab-org/ci-cd/package-stage/pkgs_importer

A

Hello, this is David from the package, video Suite team and in today's video I want to present you a tool that we created to import packages into gitlab.

A

So the idea of this tool is to um you have a configuration file where you describe your packages that you want to import and this tool will generate a pipeline config. So why a pipeline well, because then we can create as many jobs as they are needed and each job will import one package file into gitlab.

A

um Currently there is only npm packages supported, um but we plan to add more so, let's demo this tool and how to use it so here I have a brand new project empty.

A

So first thing we will create a gitlab pipeline, uh config, really simple. We just include the template that is included in the in the project.

A

Please note that I'm using the remote keyword here, which means that you can include this template from outside gitlab.com, so any self-managed instance can reference this file and and have this feature working no problem. Then, as I said, we will need a config file here. It is, and how are we going to organize Imports? Well, we will have blocks.

A

Let me add one, so this is an import. There is the type so npm, that's the only value available. At the moment there is a source package registry and there is a destination which is okitlab.com Project. Then we have a description of all the packages that we want to import. We can describe them as a mapping package, name and version, or we can have a package name and a list of versions.

A

Of course you can have multiple Imports, and so this tool will use npm the CLI tool, the CLI, the npm tool itself, so it can interact with any npm compliant package registry. So here is another import. This time around, it is from a gitlam.com project a different one. But since the gitlab npm package registry is an npm comply and package registry, we can read packages from from that notice that for the destination this is yaml, so we can use a yaml anchor so that I will not repeat everything.

A

uh uh So this is nice but let's add another npm package registry provider. So again, this one has implemented the npm apis of the npm package registry, and so we can read packages out of it, just fine and import them.

A

And lastly, this wouldn't be fun if we can't import packages from the official registry, so npmgs.org um last thing notice that we can also import packages from a CSV. So this way you can have a different tool that will create that CSV and then feed that CSV into this importer. So let's create that CSV.

A

Here it is so very simple: there is only two columns package name package version: that's it no headers, no titles, nothing! Just two columns, um yeah I, think we're good. So let's Commit This and see what happens.

A

So we will have a look at the pipeline side, and here we have our pipeline. So our template that we imported is describing uh so one job, that's our importer tool that will take the config file and generate the pipeline config uh it. Well. It's already done. uh You can see that the python config has been successfully written nice, so a pipeline config has been created and now, if we go back to the pipeline, there is a downstream pipeline here. It is, and we do have all our Imports um organized by the import block.

A

So each import block Maps into a stage for the downstream Pipeline, and you can see each job is importing one package, uh it's all going nice. So all the packages were imported successfully.

A

If you have any error of on on any package, of course, you can click on the job and you get all the logs uh you can see. As I said, we are just using the npm client to pull and publish the package like nothing magical now, let's have a look, and here we have all our packages into gitlab. So we successfully imported packages from one two.

A

Three four different sources so um have a look at this tool: try it don't hesitate to open issues for ideas bugs or feedback and I will leave the this URL of the project in the video description.

A