GitLab Package Group, 20 Jun 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Container Registry and Dependency Proxy Walkthrough

Description

Container Registry: https://docs.gitlab.com/ee/user/packages/container_registry/

Container Registry cleanup policies: https://docs.gitlab.com/ee/user/packages/container_registry/reduce_container_registry_storage.html

Container Registry cleanup policies development docs: https://docs.gitlab.com/ee/development/packages/cleanup_policies.html

Dependency proxy: https://docs.gitlab.com/ee/user/packages/dependency_proxy/

Dependency proxy cleanup policies: https://docs.gitlab.com/ee/user/packages/dependency_proxy/reduce_dependency_proxy_storage.html

0:00 Intro
0:44 Container Registry
3:24 Container Registry Authentication
14:50 Container Registry Data organization
28:16 Container Registry APIs
39:28 Container Registry Cleanup Policies
49:39 Container Registry Data Migration
01:01:59 Dependency Proxy
01:10:37 Dependency Proxy Cleanup policies
01:11:52 Dependency Proxy UI and settings

A

Hello, everyone I'm David from the packet stage um and today with me in this video I, have moas and Rod. Thanks for joining, we are going to discuss um the container registry and the dependency proxy. uh It will be a like a walk through so now to set expectations right. We are going to look at the container register independency proxy, but on the rise side on the rail side, sorry, not on the container registry directly, so I will start with the container registry. Then we will go into the dependency proxy and uh yeah.

A

That will be it don't hesitate to interrupt me if you have any questions or anything so the container registry um yeah I at its core, it's quite similar as uh package registry. So it's like managing a bunch of images, files, no images and that registry is really targeted to Docker clients. So you have the official, Docker, client um I, guess some other clients exist too, and so those clients will push and pull those images from the registry.

A

Now. The difference here with the gitler package registry is that the container registry, the gitlab container registry, runs on its own backend. It's not on Rails. So it's you, you have the real score of git lab and then you have the container registry, the gitlab container registry backend running on on the side. So that's already a difference.

A

Now. Why do we have container registry features on the rail side? Well, um the thing is that the docker I will I will say Docker client, but it can be any client. The docker client will try to do operations with the container registry so directly with the container registry, without going through the rails back end and the container registry will ask for Authentication yeah. You can imagine that pushing an image that that's not something that an anonymous user can do. So the container registry doesn't have any authentication set up internally.

A

What what it will do is hey. uh You will need to get a token from here, and the here is the rails backend, because this way we can leverage all the authentication and all the permissions system that we have in the rails backend. So let me show you a diagram.

A

B

A

um So basically, you have the docker client, the gitlab container registry and the gitlab rails. um As I said, the docker client will try to like do an operation in this case. It's like as simple as as a ping and the container registry will reply. Hey you need to to have a token to do this. So look at this URL and get a token from there and this URL it's the GWT controller from the gitlab rails.

A

So, on the next request, the git, the docker client, will just do what it has been instructed to do, which is get a token from gitlab rails and then rails. Will reply a token. If everything was okay with the permissions and then we will. The docker client will redo this the exact same request, but this time around it will provide the the token the the Authentication, and so the container registry will uh accept this if all goes well.

A

So what you need to see here is that, um even though those are two different backends, we still use only one permissions authentication system and that's the one from gitlab rails.

A

You can easily understand why, because on the git libraries, you can have permissions on like groups, and you can Cascade permissions in in the contained projects and- and things like that, so we have a set of specific permissions for the container registry. I. Think it's read, read registry and write registry or read: read container image and write container image and those permissions will leverage what we have in the in the gitlab rails. Backend.

A

um Is there anything else? No I, don't think we. We can take a look at this uh uh JWT controller.

A

So this is a general controller that basically will create GWT um well Json web tokens um for for a given service. So usually, what we expect is a service parameter um that will tell that clients we use to say: hey I want a token to access this service, and you can see here that we have a service dedicated for the container registry and then we we have a service for for the authentication.

A

So the controller, but it's called the idea, is really read the service parameter and then distribute or locate the right, authentication, Service and then just execute the service. So we do have one for the container registry and we do have one for the dependency proxy uh yeah. We will see when we discuss the dependency proxy, but the authentication works exactly the same way. So that's why it's there too.

A

If we take a look at the.

A

Container registry authentication, Service.

B

A

Well, it's well, it's nothing too magical uh I mean we do have the permissions, and then we will use the declarative policies that we have, like the project policy or the Google policy to evaluate those permissions against the the provided user.

A

Do we see this here.

A

Yeah we do have the username.

A

But I'm pretty sure that the GWT controller needs to be authenticated.

A

Like a request, you do needs to be authenticated.

A

um Okay, so well, actually we can see it here. um We should have you see we have an HTTP basic, um so we will receive from the doctor client the username password. We will locate the user and then it's just, but it's a pretty pretty complex service. So it's not that just it's not that simple, but it's a matter of taking the user. Taking the.

A

We also received the the path that the client wants to access, so we can even locate the project, and then we evaluate the permissions, though this user has the the permission to like push a container image on this project. Yes or no, and then we we reply a token. Actually that's an important Point, even if it is a no that we we get.

A

We still reply a token, but what the the actions the allowed actions will be if the the empty array like this, this user doesn't have any permissions or any actions that that the user can do so, even if we don't reject uh failed authentication. That's that's the point.

A

We always reply a token, no matter what so either the token is properly filled with the actions that the user can do or the token is like empty, because at the end that's just a token uh the one that will decide if the request goes through or not is the container registry when it receives the token. It will well first um uh like it's not authenticate the token, but there is a shared secret between those two backends and that shared secret is used to sign the token.

A

So it will verify the signature of the token to make sure that yeah, it is a token generated, but the gitlab rates backend, and then it will look at okay. This token is now a valid one. Let's see what actions the user can do. Oh it's empty! Well, you nope. This request will not be a 200 okay, it will be something rejecting the the action, but how how we, how we reply to a Docker client that the action is is rejected. That's that's the container! That's the container registry responsibility.

A

So in this request this one, the five that Returns the token you will always have a token, even even if you fail, the the authentication like the user, doesn't have the right to do whatever he wants to do on on the project.

A

um That's a pretty important difference, uh usually in the gitlab rails. If, if like authentic, um it's not authentication, it's more authorization.

A

If, if the user lacks the the right permissions to do an action, we will just like bail out, reject the request and just answer: no, you you can do that like a forbidden response or something that's not the case here we will always return um a token uh uh okay. So that's the authentication uh which is already well different.

A

Then we have data, that's uh an interesting part. um But before that, do you have any questions on this Authentication, uh like interaction, schema or or anything else,.

B

I'm curious, why we always reply with the token. Is that a standard that we need to comply with yeah.

A

If I recall correctly, um the the docker like standard will require to always reply. Your token uh can do a quick check.

A

Yeah it's this thing.

B

Yes, I can put it here.

A

Yeah, that's tiny, let me zoom in so this is the like token spec or what what the token server needs to do um to be able to be used by a container registry as a authentication server. So you have the like the actions. You can do things like that, but one thing that is really important.

A

Is this one um yeah if like, if the, if the CL, because when, when we receive this um request to create a token, we receive the the array of actions that we, the user want to do so it can be pull, it can be. Push can be well whatever um if the client has only access to a subset of of this requested access.

A

That must. That must not be considered as an error, because it's not the responsibility of the well token server to to say that this is an error. Instead, what we do is that we intersect the requested actions and the permitted actions.

A

So if you request for, for example, for a pool and the user doesn't have access to pull, it has access to well something else but not pool. If you intersect, both you will get the empty DMT array and that's what you need to to return.

A

Does that answer your question yep.

B

Yep, thank you.

A

Okay, uh yeah data, um yeah I, will simplify a bit it's way more complex, but I will simplify the data on the container registry as um repositories and tags. So repositories is the the path of of an image, and then uh you can you can you can add tags to this path and you can create different tags to to a repository, so only two models that you have a repository that has many tags. It's way more complex than this, but for the explanation, it's it's enough.

A

The main thing here is that, as you can see, the only interaction that the docker client has with the gitlab rails, it's only for authentication and that's it. What happens here like the actual action of the docker client?

A

We don't know rails rails, doesn't know better said container registry knows, of course, because it has to like execute the action, but rails doesn't know at all. So there is a a challenge, and this is the the like. The core of the complexity of the data regarding the the container registry is that rails doesn't know about all the data that the container registry has, because we are only like raises.

A

Only called during the Authentication, so um it's before my time, I, don't know why it has been made like this, but I guess on the race, but we wanted to have like a best effort. So what we do is that actually, when, when you receive a request for for Authentication, you received like the container repository path, so you can locate the project and the the name of the of the container repository. That's actually the the path and we receive um the action.

A

So if we detect that there is a pull, uh pull a push action to a container repository in a project. Well, here we are going to create a container repository object in the database. You can. There is one container repository here. uh This.

A

Yeah, that's the this is the model.

A

The the main problem is that this is Authentication, and this is the actual action, so authentication can be successful and you could be creating a container repository with the path and everything, but the actual action could fade. For example, a network hiccup or whatever so again, I think this was done as a best effort, so that rails still know okay on this project. We have that many repositories, um but it's it's not guaranteed at all, because the action can fail.

A

So you can have a think issue between what are the really the container repositories that actually exists and the ones that are in gitlab Wales.

A

um This has always existed um most of the time it works, but just be aware that when we create a container repository in gitlab rails, that doesn't mean that it will be created in the container repository because rails is before the actual action that can fail due to well whatever uh problems you can uh like assume or guess. So we have. The container repository is um most of the time it's okay, what what we have in rails? It's what we have in the container registry, but it's it's not guaranteed. There is no guarantee to that.

A

So now, I said that there are container repositories and then they have many many tags. um Each container repository has many tags. This. These tags objects. They don't exist in rails rails, doesn't know when you like. If you load not this one, if you load the container Repository you, there is no way to know how many tags or which tags are available here on this container Repository.

A

The only way to know it and I assume that there is a death tags yeah it's here, uh the only way to know it is to we'll take this. Let's come out back it's to contact the container registry to say: hey I have this repository.

A

Let me know what are the list of tags, so the container registry declares an API that we can use to inspect container repositories like inspect tags, have information on the container repositories have information on on each tag and the git libraries will like heavily use that to build the data um so yeah, that's pretty the the core of the challenge. Is that anytime, that you work with tags here in Wales? It's challenging because we don't know the tags container registry knows but rails doesn't know and to know it.

A

We need to contact the container registry, so you can imagine that we have a few challenges. For example, when you open the uh project page and and you go to The Container registry and then well. Actually let me see if I can find well. We do have an example. That's that's for sure.

B

C

A

uh So that's the uh uh gitlab, uh no, that's not actually, where I want it to be here: okay, so container registry. Those are the the images and you can see the amount of tags. So just imagine building this UI. This UI can be built only on Grails, it's impossible. Why? Because we have the amount of tags and that amount is not known by rails, because we don't have the tax object so for to build discount. We like for each image, we contact the container registry and say hey how many?

A

What is the list of tags that we have here um and that's? Why you see that there is I, don't know if you noticed but like the tax count is delayed from the main loading of the page. That's because we have a. We have a we have to query the container registry, um and this is a challenge in in all the features that we build in, like DUI API and whatever in rails, because, like for example, uh I think one of the the things we wanted to to do is like filter the results.

A

Well, it's it's not simple in the sense that this is not a database query. It's way more. We need to locate the repository. Then we take the search input and we we need to get the tags from the container registry.

A

There is no search, API and pond there, so we we get all of them and then we apply the search on it. So this this action is it's actually like several or several interactions, orchestrated between rails and the gitlab container registry. So it's it's not like the usual rail search function, where you end up with a database query with the I like statement or something like that. No because tags doesn't exist in the rails database that that yeah, they are not there they're on the container registry. So we need.

A

We need to use the the API.

A

uh So to do this uh uh very specific class has been built, which is the hs3 client, yeah client RB.

A

um That's the client that will implement the container registry API. So we have a few endpoints uh like, for example, tags. You see the the list of tags of a specific repository. That's uh that's a focus on. We have other functions like the the digest or deleting things or well, it's a it's, an API endpoint that well it's a API on the container registry that we implemented in in a single signal class.

A

um Do we have anything else here? No, that's the main point. We do have the container repositories on Rails, but not the tags anytime, you you do something with the tags. There is a container registry access all the time. uh That's why it's it's challenging for those those features.

A

C

Yeah I have a question here: yeah or you don't just store the the count of text. For example, in our database, I mean it's easy to fetch those numbers and store them in rails.

C

So every time a user is going to the page of the container registry, then the count is in the database not to fetch them from the.

A

C

A

um I guess it's doable, but then it's it's again the same thing. uh If we look at the interaction like let's say that you see a push for a repository and the tag you can increase the count, but that count is not guaranteed because the real action happens after so yeah I. Guess it's it's possible, uh uh but it's like it will not be reliable. So how much of this reliability we want um I, don't know that would be a discussion with with product and the container registry folks, um but that that would be possible.

A

um We can. We could also even perhaps push further I, don't recall exactly if we see the tags here, I'm, not sure if you see the tags, if you see the tag like hey I'm, pushing tag latest on the container repository, we could create a tag object on the real side, but that might be not the case. Otherwise I don't see why it hasn't been done already, because we already do that for container repository. So why not do that? For tags too?.

C

A

um But yeah under on the account side and that could be possible I guess um you will see. uh Perhaps in some parts we have even tried to use some caching in the sense that for some requests we can just quickly like cash, a prior request to The Container registry and reuse. That result instead of pinging the container registry. Again um we tried that it worked quite well.

A

um Nowadays, we we do have something else, which is. uh Let me see if I have it here. No, no or with this one.

A

So I said that the container registry provides an API. That's the standard, Docker container registry API that the dockler owns in a way it owns that spec. So, for example, the the end point that returns tags. uh uh Well, it returns all the tags names like. If you have a massive amount of of tags, you will get all of them, but only the names. If, then, you want any details of on any tag, there is another endpoint where you you ask.

A

The container registry- hey I, want the details of this tag, so, like things are even more challenging in that situation, but that API is like owned by not own but like it is standardized by Docker and they have respects so it's not like. We can move around this API so for our needs. uh Yeah. We speak a bit on on the history of of the project.

A

We created the yeah thanks, I. Ask you to open this. We created another API that that runs like side by side on this API and this other API. Well, we call it the gitlab, API I, don't know if that makes sense, but it's basically the container registry.

A

The gitlab container registry has the like Docker API, like from the beginning, like it is even referenced here, and you can see that it is like they have a spec for it, but we decided to have uh like an API for any git lab need and, for example, if we take the tags example back on the on the docker registry API it was, it was returned in tags, all the tags, all the tag names more precisely. So it's it's like receiving a giant array with all the tag names.

A

So imagine that uh you asked the the amount of tags, the the list of tags for this one. That's uh that's a pretty huge array. So what we did we created uh like a gitlab API? We call gitlab because it has a gitlab prefix.

A

Let me see Lions check yeah, you see, there is a gitlab view, one prefix, so I call it a git, the gitlab API, but perhaps that's not that that correct- um and here you we were talking about the tags, you can see that we created the like, give me the list of tags, but this time around, like there is pagination available. So we can like walk the tags in in chunks and we do have more information than than just the name because I just name you it's not like you can do a lot of things.

A

So let me show you the body here. It is so for this example. We have two tags here on the docker API. That would return a single array with like this name and this name. That's it. If you want anything else like the digest or the timestamp the equation, timestamp, you need to Ping the content registry again for each tag. So having building this on Rails using just the like the standard, API or the docker API, we would have one request to get the list of tags names.

A

One request for each tag. That would be two, so you are already at three requests.

A

Yeah well apply that to this amount of tags, and you can say that you are like exploding the amount of requests down to the container registry in this API, the gitlab API we it's tailored by two to our needs, so it will return in one request, like the name, the digest the time step, and so it will be one request for everything and it's it's paginated. So we can work this amount of of tags in in like chunks. Instead of trying to return everything in one one go.

A

um Well, I will not go into much more details, but we do have like several uh API endpoints for gitlab needs um and on this API, since it's gitlab, it's like we are, we are free and and way more flexible. We can create any endpoint that we might need or or anything.

A

um Yeah will I speak about that yeah I guess I can so because we have a gitlab API. We do have you remember that we have a client to talk to this API on the container registry. Well, uh no, that was not yeah, it was so we do have the gate lab. Api, client, yeah I created this class.

A

So if that, if the name like isn't great, that's my fault- um yeah happy to two hours with a better name, but basically we are implementing all the endpoints I just showed you in the documentation, for example the the tags list. uh Where are you tags list yeah here? It is tags and this time around we have like a page size and and everything.

A

So we we have two clients.

A

At at this time, and at this time of the presentation we users can well not users on.com, we are going to use the gitlab container registry on self-managed. They have the choice of using any other container registry.

A

But that is a problem, because uh uh that container registry uh could be like um the Amazon one or something else we'll implement the docker API, but well, obviously it will not implement this gitlab API, and so that's why, for some operations you you will have a service that is a facade and that facade will select the correct implementation, depending on on which container registry is connected to gitlab rails uh example.

A

B

Think it's delete bugs.

A

Yeah, that's the one! So here it's a service that will uh receive a container repository and then it will try to remove tags, Oh Yeah from from a list of tag names, so you give a container repository and the tag names and it will send the delete requests.

A

Many or a single one, depending on on which container registry is connected- and you can see here that, um depending on which container registry is connected, we will use the git lab API. So we we have a delete tag service but under the gitlab namespace and then we have third party, which is like anything else and so for anything else. We can't use the gitlab API endpoints. We need to use the the standard, um Docker API, so I guess.

A

Can we see D here uh yeah? You can see here that we are accessing the client, which is the the very first uh client class I showed you, which is the docker API, whereas if I look at the gitlab namespace.

A

Yeah well, there is a a little function, but the the this function will use something that is only available on the gitlab container registry.

A

So you can see we have a bunch of things that are divided in two depending if we are connected to a gitlab container registry or or or something else, that's something to keep in mind while working on new apis or or things like that, um that's the current status today, if one day we are going to drop the support of a third party container registries- um I, don't know, I saw some discussions about it, but I don't know if that will happen.

A

What I know is that if that happens well there are a few things that we can draw up on the real side, but until then we need to support both both apis and well. Usually, you can easily guess that using the third party API, the docker API- is usually where it's it's less efficient than usually the than using the gitlab API endpoints, because those endpoints are tailored for our needs. So, for example, in this case, it's it's customized for deleting tags in an efficient way.

A

Instead of this one, so those Services below third party will will like trigger more requests to The Container registry or or do things in a different way that is less efficient than these ones.

A

Is that all clear.

A

And checking the time yeah I have two more things on the container registry and then we can go to the dependency proxy. But since we spoke quite a lot on the container registry, I guess dependency proxy will be a bit faster.

A

um One thing I want to show you is the cleanup policies.

A

um I will probably link and put a link in the video description of uh walkthrough of these uh cleaner policies for the container registry, but we we can have a look, uh a quick look here, so clean up policies so at its core, it's we let users Define a set of filters and those filters. We apply them on all the container repositories of a given project and, like some of the filters, will say, hey tags that are that match these filters. We can remove them.

A

um That's because the container registry to store to store the images and the tags it will use, object. Storage. So that's that's! uh That's a huge! That's a usage on the object, storage, so users need a way to manage that storage efficiently and one of them is the cleaner policies.

A

um Yeah, we'll not go over all the rules, but, for example, one of them is uh tags that are like older than two years old. We can remove them. How can we know that the tag is two years old? Well, it's easy in the sense that here in the git lab API, we do also have the the same thing on the docker API, but it's a bit more involved, but well.

A

The main thing is that we do have a created art timestamp, so we can know when this was created and we can have a rule, okay tags that were created like two years ago. We are. We are quite sure that we will not use them anymore, so we can automatically remove them.

A

So that's the part where you can Define filters. The other part is executing the cleanup policy, and for that we use several background jobs.

A

um They are all based on the limited capacity concern, but in a way it they detect that a policy is ready to be executed for a given project and then it will try to well not try. It will execute the policy.

A

The challenge here is that, and you can see what we were discussing in at the beginning is that we are working on the at the tag level. So there is no way that those background jobs can do the cleanup without contacting the container registry that that's impossible.

A

We do have the container repositories, but we don't know like how many tags there are behind and which are the ones that we can clean up. No way. We need to contact the call the container registry on the gitlab API, and then we will get back the the tags. Then we can analyze the tags and and select those that can be deleted.

A

um So it's it's a pretty complex job because of this interaction with the container registry.

A

um I I guess I will not go into more details. You will have all the explanations here and there is also a page on the development side. uh Let me show you I think we do have something for this.

B

A

B

A

Yeah I'm not sure we'll have it right away package yeah.

A

Is this one? Yes, it is so it's this page. I will I will drop a link in the.

B

A

Description, so this is the like the technical description of those jobs. You can see here, The Limited capacity job and what are the services that are called and and and anything actually we can see the delete tag service, which is the the first aid uh being called, because this job has to work in with both apis right. So um we are, we we use the facade, we don't use like the specific Daily, Tech Service for gitlab or for third party. We use the let's say directly.

A

B

A

I will let you check the technical description. um It's pretty complex because we do have quite many um parameters. Can I show you an example? Yes, I can uh very quick so that we are done with the cleanup place. Is.

C

C

A

Yeah yeah, so this is.

A

This is the settings of um and of a project. So it's you have a project settings. You have packages and registries, you have cleanup policies and you have the rules, and here are the the rules like it's basically in two parts: uh what are the tags that we will always keep, meaning that those tags that match those filters will never never be removed and even even if they match this, this section it's like protect a protected tag or something like that, and then we have the filters to select the tags to remove.

A

So you can see that we have the older than and we can match using the the names using reg X. So that's pretty complex uh have a look at the at the jobs implementation.

A

If you want to have more details, but that that's uh that's a big part on the rails side and it's challenging due to how the data is organized and we don't have everything on the real side, you can see the difference between the clinopolysis for container registry here and the cleanup policies for package registry where in this time around, we have everything in the database in the result. So executing a cleaner pool, it's as easy as like selecting all the packages on the the database and then just mark them to for Destruction.

A

So it's it's way more easier, um yeah, not too much anything else. For for this part, um do you have any questions.

C

Yeah, maybe a question on why they weren't implemented in the container registry instead of fields, I mean yeah, since it's easier to retrieve.

A

Yeah, a good question: um I would say that at some point they they will get moved to the container registry, because the container registry is like the source of truth of all the data, so it is way better geared to execute. Cleaner policies and and rails can dry them. For example, you can imagine having a DUI and everything to manage the parameters, and once you save the policy you send, you send that save to The Container registry saying hey.

A

This is a the policy that we want to use here, and then the container registry will do its thing. um I would say that it's uh it was a capacity and a focus from the team. That's a that's! A good bridge for my last Point um I wanted to show you what we we have done, uh the last two three years yeah on the on the container registry.

A

They have been focusing on uh garbage collection that was a very, very high shl subject, and so they like they did everything on. They focused entirely their efforts on on that subject. So, like policies were working on the race side, so we didn't move them uh at some point. I think this discussion will happen and yeah I would be in favor to moving them to The Container registry. The other thing is that the container registry is uh I, guess that's something I should have mentioned in the beginning.

A

It's a it's a back end that is uh different from race and it's a backend in golang, so they don't have all the like bells and whistles that we have in the rails back end like, for example, um like everything that we have around sidekick and The Limited capacity job like with uh with an application setting. You can easily uh manage or or tweak the amount of jobs that run in parallel for this execution.

A

um Things like that so like there is. There is work to do to implement this. It's it's not just as simple as hey. Let's do a background job and and let's execute those policies, uh it might might have more more work, but that's the container registry side.

A

um It could be a good question to ask them uh in a in a coffee, chat, I. Think at some point the discussion will will happen and yeah for me. Definitely the policies need need to be moved on.

A

The container registry side, the other thing, I guess the other problem- is that what do you do with third-party container Registries because they will not Implement them right? If you, if you have a self-managed user that uses the like Amazon container registry or the Google container registry, they will not have policies, um and these policies, at least they have the benefit of working with those third-party container Registries. So uh I guess there is also a product perspective here and some product questions that need to be answered.

C

Yeah, thank you.

A

All right, um yeah, the last thing I wanted to to discuss, is uh yeah what we did. It was mainly last year to took a good amount of time last year on the rail side for the container registry.

A

That is what we call the data migration on the gitlab container registry, so uh I will simplify things here to go a bit faster because it is things on the container registry. Basically, they need the it's a registry, so they need to handle files right. You they get not files, but like bytes, you can imagine bytes or blobs.

A

If you want, they actually have this definition like a blob object, so a set of bytes, um so they need to receive them, store them somewhere, which in this case it's object, storage and then, when we delete them well, it would be nice to remove them from object, storage right because it's uh uh it's taking space and well space on object. Storage is certainly not free.

A

So the problem is that at the beginning, the container registry, the gitlab container registry, which is actually the like container registry, it's a fork of the container registry that Docker opens host. So it's a fork of that um they work with um the file system like they were. There was no uh no release No database, nothing. It was a file system. So, like looking for a tag to delete, you need to go through the file system.

A

So well, no problem when you have a few like blobs to delete or to handle, but imagine that you have millions and millions and millions of objects on file storage, that traversal of going through the the file storage that that has a hefty Hefty price. So what they did uh uh and and I'm simplifying here things is that when they receive a delete, they will not delete from the the object storage directly. Instead, they will like Mark the object as hey this.

A

This has been destroyed, and so they will Mark different objects as destroyed, and then they have this command. That is called the garbage collection or garbage collector that you would run, and it will like walk through the file system to find those marked objects so that they can remove them from object, storage.

A

The problem is that this garbage collector has to be executed offline. There is no way to run it online, so you can imagine that this restriction will not work well or not at all with gitlab.com right uh and it's actually quite long. uh Traversing of our system can take a bit of time. uh Yeah, like a bit of time like a few days to execute the the command so having the container registry offline for a few days no way for for gitlab.com, so they needed or we needed. Actually we are all the same thing.

A

We needed a way to have an online garbage collector, meaning that the garbage collector could uh check what are the marked objects and remove them from object storage without requiring the container registry to be offline.

A

To do that, there is no way well I, don't I, don't have the details, but, like the file system will not go through it's, it was impossible. So the idea was to have a database and like properly organize data into the database. So what we did most of the uh last year was moving or migrating you. We heard like the container registry that immigration moving objects from the file system side to the database, so that then in the database, it's way easier to find those marked objects and remove them.

A

So we needed that database migration first and that's actually what we did last year that migration was driven by rails. So it was ways that that had lagged. It was the orchestrator. If you, if you want all the the driver of of this, so it would take a container repository and then ask the container registry. Hey. Was this migrated? No okay! So can you start the migration and the immigration was actually a two-step process for the container registry?

A

It was not like a in one go because it again, you see a container repository, but you could have like thousands and thousands of tags behind in the container registry.

A

So that takes time, and so the container registry would then contact rails to say, hey, this container will be positive, repository was migrated, and so uh uh it's like it's done uh and the the raised part would walk through all the container we put it there is that rails, nose to migrate them and that took a good amount of time, but you can see in the code we can still have we.

A

We still have some some things about the import like, for example, this function, try import, that's the function that will like send the the start: migration command to The Container registry uh yeah. You can see, we I think we used a State machine where it is here. You have a migration State because, as I said, the immigration is in two steps: it's not a single one, there was a pre-import and there was an import.

A

So we had many steps uh defined many events and you can still see the code of of this data immigration in the in the code on the right side, the immigration was completely fully done, for.com and for self-manage. If I'm not wrong, they will not use what we did. On.Com they will use something else, so I think it's basically a command like bringing down bringing offline. The container registry migrated in one go and then starting the registry again, which was impossible for.com right. So we did this all.

A

While the registry was online, uh we had fun. It was actually the first time that both teams worked really together, uh like we were investigating logs, like what happened on this repository. Oh yeah. This happened on the Wayside, what happens on your side and on the quick lab container registry, so it we, we had fun, but all went uh well. And.Com is uh fully fully immigrated and any new container repository that is created now it's created directly uh has immigrated like directly in the container registry database and that database actually unlocked several features.

A

One of them is the gitlab API endpoints the tags and points that we discussed with pagination. That was possible because tags are now in the database and so well by generation in the database is, is quite easier, well, quite more or less easy to to do, and and they were able to create this online garbage collector. So we have an online garbage collector that runs all the time and all the time it will like reclaim all the storage from object, storage.

A

They have a few dashboards for that on grafana, so on the container registry side feel free to explore them. If you, if you want that's, that's what the the main focus of the container registry team was and that's why the cleaner policy is yeah, we will we discuss about moving them, but the the there was never okay. We like we now have time to do this. It's like no. The the online garbage collection uh collector is is like the the highest priority, and that would be the end of the container registry side.

A

um If you have any questions.

C

Yeah just curious that garbage collector is written in go right. Yeah.

A

Yeah, that's that's all written like within the container registry and that's that's um specific to The gitlab Container registry, like all the other container, Registries I, don't know if they have an online garbage, collector or not. uh uh I have no idea, but the the open source container registry provided by Docker doesn't have an online one. It's an offline one. You need to like.

C

A

The container registry in read-only and then you can run the offline garbage collector.

A

Right, yeah, I, I, hope your heads are not exploding too much data no in in a short um amount of time. Okay, we take just a few minutes for the dependency proxy and then I guess we we are. We are good, so the dependency proxy yeah there is vastly uh um a pretty good confusion like the dependency proxy has something to do with the container registry feature and the truth is that nope not at all, but it's related. That's that's correct, but it's it's not the same thing so the container registry.

A

Oh, no, don't say me that my my mouse is dying now. No, we are still here. The dependency proxy is basically this a pull through cash, meaning that you will have docker. You will have well at the moment it's Docker hub and you have gitlab in the middle one Docker pulls an image from Docker herb. It will pull it. It will send the request to gitlab gitlab will send request.

A

Docker Hub will reply the image or tag or whatever object you you request. Gitlab C sees that the object and returns it to docker.

A

So you see there is no gitlab container registry in that interaction, but it's related in the sense that Docker will pull objects with the same API comment or whatever you want to call it as if it was pulling from the gitlab container registry. It's the same API because for Docker it's hey I'm, let's say that it's hey I want to pull a a blob or an image the commands that are triggered by this client, this Docker client, it's exactly the same for the dependency proxy and the gitlab container registry.

A

So we do have this on gitlab Rays the dependency proxy, and what is interesting is this word cash, meaning that we see the objects going back to the client, but we are not going to send the objects back to the client, because otherwise we are doing a proxy feature, but without like a lot of value right.

A

So what we do is that we send we send the object back to the client, but at the same time we save it in in a cache which is actually object. Storage, it's a file on object, storage. This way, next time we see the exact same request to the same object. Instead of like a pinging, Docker Hub and returning it from dokohub, we can just use the cache and return the cache uh that's a way to, for example, mitigate the docker Hub rate limits.

A

I'm, not sure if you are aware, but well dokohub introduced some pretty restrictive weight limits, meaning that if you pull too many images per hour at some point, they will not be happy with that. I guess! No! You will reach a limit and those pool will not work anymore.

A

So, with the dependency proxy, you can cache those heavily accessed images and you can remove them from the the rate limit computation of of Docker Hub, because you are pulling them from the cache directly so uh because we are using a Docker client same thing for authentication exactly same thing.

A

As I said, um the the client will try to like pull a thing from a Docker hub using gitlab and while GitHub will say Hey, you know you need to be authenticated, so use this controller, which is the uh Json web token a controller, and that's why you have a service authentication Service for the dependency proxy.

A

um There are again nothing too magical. It's it's really similar. uh The dependency proxy works at the group level, so we receive the group we receive the user and we'll receive the action, and then we will map that to a permission, I think it's read dependency proxy if I'm not wrong, but yes, we can just check really quick, pull AC, and here we have weight dependency, procs yeah, that's the permission, so it will. It will get a token from that and then it will ask the uh well.

A

It will use that token to to do the pull it's only for pulling right. The dependency proxy doesn't work for pushing images. Actually that doesn't make a lot of sense, but uh it's really for pulling only so I think. Let me see it's dependency proxy group.

A

Yeah, that's the one, don't ask me why they use this name, it's before my turn, but it's all done with a controller and that's it. So.

A

Basically, there is only two routes: the Manifest and the blob, which is the which is the um two like row, types that container Registries will work with, and so both work in the similar way, in the sense that we are going to check Docker, Hub I think there is a head request that we do to check the digest the signature and check if we have a cache with that object and check that the cache is the has the same signature.

A

If that's the case, then we like, we send directly the the Manifest and if that's not the case, we actually.

A

This is a super smart action that vocals can do. You can tell workers, which is the component that is in front in front of rails to say: hey I, want you to return this URL or this data that is on this URL, but at the same time you upload it to this URL.

A

So you can imagine the T command in Linux like where you can like split the outputs. Well, this is the same thing, but with uh with data that is located in a URL, and it is like, like pulled the data from this URL, and then you see it so one one stream for the for the actual client and another theme for the the upload and that's when you we are going to create the the cache uh object.

A

So we do have cash objects, so those are, if I'm not wrong. It's a dependency proxy blob.

B

A

It is here, so we do have a blob and we do have the Manifest and both have a file uploader, so that that means that we have a link to a file on object, storage, um that's pretty much, the the whole dependency proxy. The only other thing that you need to know is that we, like with time users, will start building a lot of cash entries and those crash entries are stored on object, storage, so same thing as the container registry.

A

If you want, if using it, is using object storage, so we do have like a simplified cleanup policy. It's it's super simple uh I, think it's only one rule and it's the amount of days um the amount of days that this entry was not downloaded. So, for example, here this means that hey, if the this blob or this manifest from the curse, was not downloaded after 90 days, you can remove it and that's that's it that that's the whole cleaner policy.

A

So this is way more simpler than cleanup policy for container Registries, uh because we we do have like everything in the database and it's just one one goal and that's it. There is also a way to like remove.

B

A

All the all the cash in one go: it's like uh a result, and you you can you can use that too. um Is there anything else I want to show yeah. We could have a look at the. Let me see if I find we do so. It works at the group level. Don't don't make the same mistake as me, like I was looking at the project and I was where are the? Where are the dependency proxy settings? I don't see them and yeah I will not disclose how much time I lost that day, but yeah.

A

That was a pretty high amount of time. So don't do that mistake as me? It it works on on a group. You need to be at a group, you see it there, um so you can see the the yeah, the images that were cached you you see here when it was charged, and the other thing is that we do have uh some settings which is basically like enabling the cleaner policy or not or and enabling or disabling the dependency proxy.

A

So it's very similar to the package registry and the container registry, but it's a very simplified version of it um yeah. Apart from that, it's nothing too magical. It's like it's really a cache like we put gitlab in the middle of a web request, and we use this opportunity to Cache responses so that we can save some requests to the actual registry.

A

My last word on this is that um we are currently at this point of time. We are currently working on a similar feature for package Registries and uh we are starting with Maven so same thing. We want to be able to Target a maven package registry and put gitlab in the middle so that when the nav and client asks for a package and that package is in the cache, we can return it right away. We don't need to Ping the the external package registry.

A

And that's about it for the dependency proxy. um Do you have any questions.

A

Yeah, it was quite quick, but really it's like you see, you see the controller. The controller is actually two routes. That's it like. Well, okay, no, two routes. It's like for clients, it's two routes for the rails back end, it's a bit more because you need the routes for um uploading objects, because uh this magic trick with Workhorse will upload things. So it needs a route to upload objects and that's where we put blogs and manifests to object, storage, but from a client perspective.

A

It's really like two routes, that's it, uh and so it's like nothing, nothing magical deal! Yeah! Don't don't be confused that hey the dependency proxy is a thing that is working on the container registry or you interact with the container registry. No, not at all you it's it's different, but it used the same, uh commands the same objects and, and you actually log in in the dependency proxy. The same way you log in the container registry.

B

A

C

A

Here you see, you need to Docker login and then you actually log into the to the base URL. So that's that's the thing that is a bit confusing you don't log in into the Container registry or the gitlab container registry, you actually log in into the rails application, not the digital container registry, because this is all with the rails. Application.

A

Okay, you made this far, so congratulations.

A

There was a lot of information, I went over time and yeah everything. So congratulations so well better said. Thank you for bearing with me um yeah again I will I will try to post the interesting links in the video description of this recording.

C

A

Okay: well, if there is no other questions, uh I guess we have.

B

I have a oh yeah, stupid questions. uh Probably it's way earlier. There.

A

Are those so there are no stupid questions that that concept doesn't exist? It's it's like no.

B

Container registry and uh dependency proxy, they keep their state in the same database that the rails application talks to no.

A

Or no that's the thing. The dependency proxy is really it's like a feature of the Rails back end, and so it will use the rails database and and everything and it's in in a way.

A

It is a container registry in Disguise, because we need to interact with the docker client um I guess we can have a parallel with the dependency proxy for packages we we are starting with Maven, so that that dependency proxy will need to speak the maven API, meaning that when you pull a maven file with the you know the structure that Maven clients we use that dependency proxy will need to understand that structure so same thing here. The dependency proxy for Docker images need to understand. Oh, this is a Docker, client or well.

A

A client doesn't need to know that it's Docker, that's a client that needs that is pulling a blob. Okay. So let me let me do my thing to pull this blog, so it's really all contained in the rails. Backend, it's a rails, backend feature the container registry. It's the the order, backend that is running and in particular that's actually a good question.

A

The database of the container registry is like specific of for the container registry. We don't have any access from the rails. Backend um yeah. We had some discussions. If it was, it would be a good idea to provide rails a read-only access to that database, so that we can. This way radius has a way to know the container repositories and the tags without pinging the container registry, um yeah I.

A

Guess the disc the discussion went like well, we are going to build a database first and then we'll see, uh but the problem is again: we are supporting third party Registries. That will not have this database part, and so you will need to get this part that you need to get this part in the rails, backend for the gitlab container registry only but yeah the the. If, if you look at the postgres dot, AI tool that we have, you will have git libraries and there is an instance that is for the container registry.

A

That's the container registry database or metadata database that they sometimes they call it that way.

A

But it's it's really disconnected. That's why it's challenging to work or to have those changes on those features with the container registry? Because, if you, if you look at the rails backend only, we only have the container repositories and that's it.

B

A

Okay, uh well I guess if there are no more questions, um thanks for for attending and yeah I guess, I will see you in the next one and have a great day bye, bye.