GitLab Package Registry Group, 23 May 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: GitLab Packages Importer: Maven Packages

Description

Packages Importer CLI app from the Package Registry team.
- Project: https://gitlab.com/gitlab-org/ci-cd/package-stage/pkgs_importer

This tool uses the mvn client to pull and publish packages and can interact with any maven compliant package registry.

0:00 Intro
0:13 Quick refresher
1:15 Demo with a new project
1:50 The .gitlab-ci.yml file
2:40 The config.yml file
2:52 GitLab to GitLab example
5:10 Maven coordinates
6:08 Artifactory to GitLab example
7:39 GitHub to GitLab example
8:15 pkgs.csv to list packages
8:56 Maven Central to GitLab example
9:37 Self managed can also use the importer
10:07 Commit to run the pipeline
10:25 Pipeline explained
10:49 View imported packages
11:45 Checking the project's package registry
12:01 Limitations
13:32 Submit feedback and questions

A

Hello, this is David from the package widgets with him and in today's video I'm going to discuss how to import packages into digit lab package registry. So we have been working on this tool that takes a configuration file that describes a source package registry, a destination package registry. At the end, the list of packages that you want to import from the source to the destination.

A

This tool will then create a gitlab pipeline configuration file and you can create a pipeline out of that, and the pipeline will execute the import. You will have one job per package to be copied, um so you will have all the visibility on the on the import. On the first version, we had support for npm packages. On the second version we had support for um nuget packages and this time around I'm, going to show you how to import Maven packages.

A

So here is a brand new project that I just created on gitlab.com I just set a few CI variables in the settings. You will see why we need them so I'm going to open this project in the IDE.

A

So first thing we will need uh gitlab configuration file and for this one it will be quite simple, two lines. So what these two lines do is importing the configuration template from the Importer tool project and it will set up a job that will pull the tool and executed within the pipeline. Then it will create, as we discussed it, will, output a gitlab configuration Pipeline and it will start a downstream pipeline out of that file.

A

So, on this file we are all set up. The tool will need a config file which we are going to create now, so how this works.

A

um We organize the configurations by Imports, meaning that um here we have an import well from gitlab, so it's from another group in on gitlab.com, and so you have the source, the destination and the list of packages to to import um notice that the gitlab maven package registry has multiple endpoints from which you can pull packages. You have the project level, the group level and the instance level for this one I'm using the group level endpoint to pull packages.

A

This group, being private I, need credentials um to pull them.

A

So I will just describe them. We will need a token. That's where we are using the CI CI CD variable that I defined in the project, and then we will need the header name, so the header name. If we open the gitlab maven package, which is free documentation, we can see that, depending on the token type that you are going to use, we need to send a very specific name as the header.

A

So in this case I'm using a personal access token I will need to send a private token, and here it is for the destination. The destination will be this project that I'm working on. So it's the project level endpoint again, this is a private project or well. Actually we are pushing or publishing packages towards that project.

A

So we need credentials to do that and same thing: I'm using a CI CD variable and the header name is again private personal access token now for packages, so we need to list which packages I want to import on Maven packages. It's a bit challenging because Maven packages, if we look at the reference um they are described by coordinates that have three pieces. The first one is the group ID.

A

The second one is the artifact ID and the third one is the version so to describe that we will need to put together the group ID and the artifact ID together to build the package name. If we want and then below that we can describe the versions, the versions can be in their way right below the package name or it can be a single value that works too.

A

So, as I said, we can describe multiple Imports in the config file. We are not limited to one single import, so let's try to describe a different import.

A

This time around is from artifactory, so um we will use the URL from your artifactory repository URL and we will need to pass credentials so for artifactory we can use the basic OS authentication, and for that we will need a username and token again it's a bad idea to put password in config file. So we are using the ACI CD variable destination.

A

The config file is a yaml file, so instead of rewriting or duplicating the destination definition, what we can use is a yaml anchor. So this is a yaml anchor and I'm just referencing it here, meaning that this destination will be exactly this one. So we are importing packages from here from a gitlab group to this project and here from artifactory to this project again, so the package list similar. We have the group ID the artifact ID and then the version.

A

um Let's put another import.

A

So this time around from GitHub very similar to artifactory same thing, we need to describe the source, the authentication there is with a username password. So we need to use username token. So same thing, cicd variables, this time around for packages I reference, a CSV file, um because the tool supports that. So, let's create that file.

A

And let's put some um packages, it follows the same rules. We do have a group ID, the artifact ID and then the version. We only have two columns, that's it it's the package name and then the version, that's it. We don't need anything more.

A

So we are good to go here and the last one I want to show you today.

A

It's, actually, an import from Maven Central, so Maven Central is the Like official public Maven registry to access this one. You don't need credentials, so we don't Define them. We still Define the destination, of course, because we want everything to be imported on this project and we list the packages that we want to import.

A

um Well, that's good I think we are all clear.

A

A

Okay, oh one thing I forgot to mention here is that we are importing the template using the full URL, which means that self-manage instances of GitHub can use this tool directly by using the exact same lines in the the gitlab instance will pull the template from gitlab.com from this URL and then execute everything so that works for self-managed too.

A

All right, uh let's commit everything and have a look at the pipeline.

A

So here it is running, so we can see that we have the job to generate the downstream pipeline config file, so yeah. It seems that the config was properly generated.

A

If we look at the pipeline again here we have the downstream pipeline. If we open it.

A

We have all the Imports so, as I said, the Imports are organized as blocks, so each stage that we have here is a block import block that we defined in the config file and each stage will have one job per packages and you can see that there are all running in parallel, more or less so it's pretty pretty fast.

A

um If you have any doubts on how the package the maven package was imported, you can open the job and see the details.

A

So, let's check the package registry of this project, and here we have all the packages that were imported into this project from the different Source registries.

A

um One thing I want to mention here is the limitations, so the tool itself has some limitations, because when we, when we have a downstream pipeline, the config file that we can pass has a limitation on its size.

A

So there is a there is a limit on the amount of configuration that you can put in the config file, but the tool will check for that amount for you, so you will see the tool failing. If the limit is the the amount of config is above the limit.

A

The other limitations are for Maven packages. So at the moment we don't support importing artifacts that have a classifier. So, for example, Dash.

A

Javadoc.Jar or Dash sources.jar, those are notes about it at the moment like we only support like standard Javas files for for the import.

A

um Yeah I think that's the only limitations I wanted to walk you through so um yeah I, guess, that's it for the demo. I will leave a link to the project in the video description below feel free to visit the project and open issues for any kind of feedback.

A

Yeah I! Guess that's it for for this demo and well see you next time. Bye.