GitLab Package Group, 14 Jul 2023

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: GitLab Packages Importer: PyPI Packages

Description

Packages Importer CLI app from the Package Registry team.
- Project: https://gitlab.com/gitlab-org/ci-cd/package-stage/pkgs_importer

This tool uses the pip pull packages and twine to publish packages. It can interact with any PyPI compliant package registry.

0:00 Intro
0:10 Quick refresher
0:57 Demo with an empty project
1:20 The .gitlab-ci.yml file
1:49 The config.yml file
2:00 Artifactory to GitLab example
3:25 GitLab to Gitlab example
3:59 pkgs.csv to list packages
4:30 PyPI.org to Gitlab example
5:06 CI/CD variables
5:28 Commit to run the pipeline
5:43 Pipeline explained
6:04 View imported packages
6:45 How packages are pulled and published
7:24 Checking the project's package registry
7:43 Closing words

A

Hello, this is David from the package of Industry team and in today's video I want to show you the package importer for Pipi packages, so, first of all, a quick summary of the packages importer, that's a tool we have been working on that allows users to import packages from a source, packaged registry to a destination package registry.

A

Basically, you have a config file for the tool where you define all the Imports and all the packages that you want to import, and then this tool will run and create a gitlab pipeline configuration file, and we will take that config file and and create a downstream pipeline. That Downstream pipeline will carry out the import with like one job per package, so you will have a very nice view of of the import.

A

So here we have a empty project. I'm going to open the web. Editor.

A

So to pull the Importer tool with the config file and and kick the downstream pipeline, we created the template within the Importer tool project and we just need to reference that project. So I'm going to create the gitlab CI EML file, and here it will be just two lines you see I'm, just including the the template from the Importer project.

A

One thing to notice here is that self-managed users can also do this and reference the template that is available on gitlab.com. So next we will need the config file for the Importer. So here we are going to organize the Imports by blocks. If you want so the first block, it will be from artifactory to a gitlab project. That's actually this project that we are editing, but it can be any gitlab project. So we have a type here. It will be API.

A

We have the source with the URL and credentials so for Pipi. The authentication is done by basic auth, so you will need a username password which will translate into this configuration as the username and token we will have a destination in this case. It will be this project and again same thing. We will need a username token.

A

For the gitlab token, it will follow the gitlab API package registry, so you can use a personal access token, a deploy token or a CI job token that's up to users and then, of course, we need the list of packages, so you can list them one by one with the name and version. If one package has multiple versions, you can list them right below and that's it for this import. Let's create another import because you can Define multiple Imports in the config file.

A

Okay, so this time around I'm going to import Pipi packages from a gitlab project, so same thing: we have a source with the URL username token and for the destination it will be the same one. So we can use a yaml anchor so that we don't need to repeat this configuration for the list of packages instead of having them here in line. We can also Define them in a CSV file. So let's create that.

A

Here it is so very simple: there are only two columns: the package name, the package version, no headers on the CSV file, nothing very simple, just package name and version, and you can fill the the file with the list of packages.

A

Okay, that's another import. Let's add a third one. This time around, we are going to pull from the pipi.org.

A

Which is the official registry for API so same thing? Again, nothing magical, but this time around the source. We don't need any credentials, because this source is public, so we can pull publicly packages out of it.

A

um I think we are good.

A

uh um Oh yeah, one word on cicd variables, four credentials. Obviously you don't want to store passwords within the file directly, so what we can use is a CI CD variable and this variable will be resolved. Just fine, so I have variables for credentials for accessing the different registries.

A

Okay, let's Commit This.

A

And now, let's check.

A

The pipelines, okay, here it is running so the template has been pulled, so the the first job will run the Importer tool and generate the downstream pipeline config. Okay. Actually it's already done so you can see now here we have a downstream. If we open that pipeline, we have as many jobs as packages and they are organized by import. So we had three Imports, so one two three and each import defines a list of packages, so we will have a list of jobs.

A

Well, they are all successful, so we were able to import all the all these packages. If one of them fails, you can really easily just open the job log and and have a have a sense of what is happening and and and what could be the the error. The Importer tool is using, um as you can see, uh pipe download for pulling packages and it's using twine here. It is to unupload for publishing packages. So we are using like the tools that are available within the the python ecosystem.

A

We are not using any Custom Tool or whatever, so this importer should work with any Pipi compliant package registry.

A

um Let's check the package registry of the project, we should have all the packages and yeah here we here they are uh so all published and now the these ones they are all available for pulling from this this project, using the gitlab IPI package registry as as usual.

A

um Okay, that's it for for today. If you have any comments or any feedback, you will have a link to the Importer project in the video description, so don't hesitate to open issues there. All right, I will see you in the next one bye.