GitLab Package Group, 8 Sep 2023

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Package stage: Google Artifact Registry Integration

Description

Sync meeting on the discussions around the technical analysis of the Google Artifact Registry Integration.

https://gitlab.com/gitlab-org/gitlab/-/issues/421632#note_1543899485

A

Hello, everyone I'm David from the package industry and um today's we are going to uh discuss the Google artifact registry integration. uh With me. We have joal from the container registry and Rahul from well the package stage. I guess the front end for the front end side, um so yeah, I guess the first thing uh I want to go through is the approach. So I did uh yeah some like a small technical analysis and yeah I guess aim for a too high goal to manage too many things.

A

um I think we are all more or less in agreement to on the first version on the first version to support one single set of credentials and one repository of from from the artifact registry, but um think about supporting multiple repositories.

A

uh Yeah I'm, pretty convinced that we are going to have this need um due to the other formats that we have in the repositories, and we will have users saying: hey: I, have Docker images and and Maven packages and npm and a can we can. We have everything in the the S be laab project, so um yeah I think we should aim for that. First version uh am I accurate with that.

B

Scope, yeah I think so uh there are a few things that that we can do. For example, yesterday I was we were talking about the the environment variables. uh So probably it's not a good idea to provide an environent variable which includes the the repository name, uh because if we are going to add support for multiple repository names, then we will have to let the users decide. What's the target reposit repository name for a push and a pool.

B

uh So maybe we should just fill the other parameters like the project, ID and and location, uh and leave that part outside so yeah. There are a few things that we can do to make sure that uh we can add support for multiple repositories without breaking changes, yeah without changing environment variables, without changing apis.

A

Yeah, what I want to avoid is okay receive the need of multiple repositories, and now we are kind of stuck or blocked with an implementation, and we have so like so big barriers that we need to work to towards lowering them, um be it I, don't know, data migration or or something if we can U at least think about supporting multiple repositor from the start. That would be nice, um okay.

A

uh So then, the next question, uh yeah I, think we discuss where to put the yeah I call it the gitlab Border it's like from from which point we start controlling things on the gitlab side. So we have a kind of a hierarchy. We have uh Google projects, each project has many repositories and each repository has many uh artifacts. So where do we put the the Google The Google, the gitlab uh border?

A

So there was this idea of um yeah: let users just fill a list of projects and we dynamically get the list of repositories. That sounds um great, but yeah, as I said, as I commented. I think I would leave that for a followup rather than the first version and start with uh simple things like they need to provide a project and the repository. That would be the first version and then, if we want to open up- and they just provide a list of Google projects yeah, why not that?

A

But that would be the second version, and I would do that when all the formats of the repositories are.

C

Supported yeah I think that's SS good to me. I think your explanation. There uh was uh uh good and we I we were I was discussing where we put that like boundary and so I think in agreement there we can go with. U like having the repository as the layer.

A

Okay, okay, that that works then um so now digging a bit deeper in the implementation. We have this uh part in gitlab projects which is called project Integrations and so well. Logically, that's where we want to put our settings right of the artifact registry, um the issue: it's not an issue, it's more a challenge! The challenge I see that how this part is implemented.

A

It's it's like a super smoth framework where you have a integration object and you have actually like a name like some kind of common data and then you have uh a props and that that properties, it's a simple Ash, and so you put whatever you want inside of that hash, so you can put like like URLs API Keys, whatever um The Challenge I see, is that well we are going to put the credentials in there because it's only one set, but for the repository we need to put a array of at least uh I guess strings.

A

That would be the repository names and then the problem is that I'm not sure if, um if an array value in those props is supported in the in the UI, because right now the props are the it's like the persisted field in the database, and then on top of that you have many s small help of functions that will help you to describe how the fields within props are displayed in the UI. uh So you can say: hey in the props.

A

I have a, um for example, for the the the Google project ID, you can, you can say: hey in the props, I have a I, have a project ID and that's a string, and it needs to be displayed like this, and so then the UI will understand that okay, I, have a props object and inside those props I have a project. I did that I need to display. The problem with the repository is: is that if we want to support multiple repositories, we will need to support.

A

Hey I have a a key that is named repositories. The props and the value is an array of string and I'm, not sure that this is supported, um I'm in discussion with um the team behind this small, uh like framework or well integration, object and we'll see we'll see how we can support that those multiple repositories. If we knew that it was only one repository, it's easy. It's like a simple hey. We have props and inside the props you have a repository name or repository and that's a string field and that's it.

A

But if we want to prepare things for multiple repositories, then we need within an array somewhere um yeah, so I'm in discussions with them um yeah. Hopefully we will have some some answers: um I I, even I I'm, even exploring we have the integration project, which is a persistent project, I, even wondering if we could have a like a side table that would hold additional settings.

A

And this way you can, you can hold like um the repository names or something like that, but it seems that uh the really the recommended way for them is really having everything in that props uh field. So yeah I'm, still in discussion with them and I'm at I, might be missing how to do this properly. So uh perhaps there is a there's a a way within this integration uh object, um so we might have or not some frontend work on those settings.

A

If everything is supported by the integration object, um yeah, I guess a lot of front and work is already implemented, and so uh that would be a a good news. The additional nice thing about this um integration object is that they have a test feature or test aspect, meaning that you have the integration uh screen. You feel all the details, and then you have a test button, so you can click test and the back end will try to contact the the external service.

A

So we could use that for the for the first version too, so that that they can users can verify the.

B

Credentials yeah. We could also use that to make sure the repository is of type locker before yeah.

A

Persisting but but it's uh from what I saw it's really a test button like it, it has to be um interacted by users. Perhaps there is a way to trigger the test before pesting.

C

A

Yeah that could be possible. I guess.

A

um The other challenge is the the Jon key, so so Google we, we are going to use a service account and uh and the related key, and that key is actually a Jon structure. It's not like an API key, which is like a very long string. It's it's a Jon structure and the challenge here is that we need the Jon form like the string. That is the GM structure for the official client. We need that and we need the B 64 version of that key for the cicd variables.

A

uh um Yeah I, don't think it's a good idea to put a Jen structure as a as an environment.

A

Variable um I'm concerned that this could bring some issues, whereas with a B 64 string, it's just a giant string and that that's okay for for environment variables and and so there is a way for the the docker login command to send um to Google. To say: hey here is my key, but it's encoded in B 64, so uh we can use both.

A

So the challenge is that we need to ask users to fill the integration settings with the J and key, but we will need. We will need the B 64 version of that, um and ideally we want to um encode in B 64 only once and not each time that we are going to set the cicd variables so yeah, probably we need to store it in the props of the integration object, Tool, uh but that's another callback that we will need to uh Implement. um Okay, that's uh that's the like lowest level. I would go.

A

I will not go into more details. uh Otherwise that would be like writing code.

A

um I guess yeah. Is there any questions on this integration? Those it's like the the settings side of the of the integration. Oh and the other thing about using the integration object. Is that yeah I guess pretty much. The permissions are already handled, like only maintenance can access the project Integrations. So we wouldn't need to add a new new permission for that, uh so yeah. If we can reuse that part, that would be.

B

A

C

B

We we can't it's probably. It would probably be useful to use that for as long as we have a single repository y uh and we could change later, because we already have a lot of new stuff to to build and doing that. It's probably not a breaking change in any way. So.

A

Yeah, uh it could be possible that to handle multiple repositories, we need a a side table that we will link to the integration object. A bit like we do in the package registry, where you have the the the package table, that's the common fields for all formats, and then we have dedicated tables for additional informations um and so yeah that could work so that the first version is only the integration project, uh the integration object with the props um um field and then to support multiple ones.

A

We extend that with a dedicated table that that could work but I'm still in discussion with them, and actually once we are done with the technical analysis, I'd like to ping one of uh the member of that team, so that they can look at what we want to do. This is to avoid on DMR having things like hey. What are you doing? This is all wrong.

A

um uh Okay, and actually one of them is a graphql expert, so that could be useful in the Mr reviews too.

A

uh Yes, all right, uh so the other aspects uh yeah one of them uh I, found I, guess we we were discussing it yesterday with Jo. It was about the the graph C thing, so, in the current analysis, I thought hey. Well, we will Implement a a way to get the repository of a project and then a way to get the darker images of that Repository and and if we are looking to support multiple formats, I thought well perhaps having a graph query that is get Docker. Images is not such a good idea.

A

What would be better is have a query that is get artifacts from the repository and then that graphql we return a abstract object and so, depending on the type of the repository, we will have a dedicated type. We have the same thing in the package metadata type where the the metadata type doesn't exist. What we have is, we have npm metadata type net, metadata type and so depending on which package type we have traversing to the metadata type.

A

We return the right concrete type, so we could do the same here, have a get artifact query and that query will return the right objects depending on the type so well. Obviously, on the first version, we we we are going to only support Docker images. So if the repository is something else well, we either return an error or an empty result.

A

Perhaps an error is is a better thing, uh but then, as we start implementing the other types, it's just a matter of adding the concrete type and the graph Q query will not change change more or less. The result would change, but not the the query itself and the the inputs.

A

So it's a it's a bit of more work in the sense that we need to implement the abstract type and then the docker image, concrete type but I think it's it's worth it because then the query we we are not going to touch it to implement the other repository formats.

C

A

C

B

C

I left a comment um like based on your discussion. Yesterday, I just left, I've left a comment.

A

C

In that same issue and yeah I was trying to Envision where, if we can validate when we are adding the uh repository, if it's a type, it has the format type docker, um then um in the UI for the artifactory registry we can just have the docker images list page and the detail page. uh We don't need to show the list of repository page at all.

C

In the first instance and in the next next iteration we can have the list of repositories and those can have the filter like filter from um and those will already be filtered by Docker, because when you're adding it um in the integration settings, um it's validated that it's a Docker format.

A

uh Yeah well, we will still need to handle issues because uh yeah as Jo suggested yesterday. We could have this crazy situation where someone uh destroys a repository and recreate the another one with the same name, but a different format.

C

A

Doesn't change the Integrations on the gitlab side and so yeah? We might need I. Guess it's it's better if the back end still verifies that we have the right format when we ask for the for the docker images. Actually, the the official client will end up in an error. If you ask like the official Cent, the method is really get Docker images, that's the name of the function. If you call that on a repository that is on format, I, don't know npm. That will end up in an error.

A

So it's it's not like. We need to verify it like. We will get an error and we need to handle that.

C

Yeah I think we need to still um yeah if, if it's, um um if it's required, that you need to call the format thing yeah um that will that that'll actually scale better when we add like multiple formats as well right so I think having that support in the back end is probably useful, yeah and I'm trying to think from a front end perspective to keep like the most uh like minimal implementation, yeah and not have like an extra page.

C

um So if you have like one repository, we show the repository like the list of Docker images. um If it fails, then uh you will handle it on the back end and show like no Docker images or something I know.

C

The API will fail with an error but probably you'll, handle that with like an empty array and then we'll show that um in the integration they can probably test their connection or whatever, and this check that it has the right format, if not we'll, show some error and then ask them to update it or something like that, yeah and so in. So in the first version, we'll just um have the docker images list page and the detail page and the Integrations can have yeah the single credential and.

A

So I you want to avoid the page where there's an single rowle.

C

L, which is.

B

Single repository, yeah yeah, one more click.

C

A

We can put a small message there: almost there yeah um yeah that can work, I, guess yeah. That can work. uh So we could have like the query to list artifacts. We could accept a project Global, ID, L uh and from the project. The back end can get the integration, get the repository and get the docker images and return.

C

B

Yep correct correct y.

C

Yeah, okay, so so so um the front end will call the get artifacts API.

A

And then yeah for that project and then I guess other one query for get Docker image because.

C

A

When you will have the result of get artifacts, you will know that those are doer images and so to get the details, we will need another gra kill query which would be get Docker image and actually the back end can call get Docker images and get an error.

A

Because before that we need to know which well, it depends I get I, I guess, on the first version we could call get doer images on the repository without checking it, but on the version where we support multiple formats, we will need to know okay, this repository, which format it is. Is it doer npm or Maven? So we will probably need to call the get repository details. There is a method to get the the information of a repository, and so the back end will need to uh handle that.

A

But that's okay for the first version, I guess we can directly call get Docker images and if there is an error, just other return, an empty array as you suggested, or we could also return an error so that the front end canplay.

C

A

There was a there was an issue check your project, integration.

C

Settings or something yeah.

A

Okay, uh yeah that works for me, uh okay and then yeah.

A

um Yeah, the other aspect was cicd variables, but I'm not not sure. If there is something to discuss there or I guess we can discuss it in the blueprint directly.

B

A

Mainly the approach and the the gql and the project integration I wanted to discuss today. um Okay, so if we are all okay with that, I will update the analysis. um I will also get the answers from the uh for this integration object and and that props field and uh yeah update the analysis with that and I guess. I will ping you for uh second round of feedback.

A

uh Well, hopefully, uh like we see the the end of this analysis and then I will Port all the main points to the blueprint Mr so that we have everything in a in a single place. Does that work for.

B

You all right yeah, definitely so next week, if we can have that done next week, then we can ask the product designer to to review as well and start drafting yeah the.

A

Designs so yeah uh yeah, that's one! That's something! I wanted to start with, and I completely forgot in the analysis. The front end part is really not super accurate, because there are many ux questions that I don't know. We don't have the answers yet. So it's a bit hard to f figure out uh what what will be exactly the font and changes but uh yeah I guess what we we discuss today is um detailed enough to put that in the in the analysis.

A

Okay, uh right, we a bit of the time- apologies for that um yeah I will update the analysis and ping you on on that uh when it's done. Okay, thanks for attending and well see you in the issue. I guess all right have a great day.

B

C