GitLab Package Group, 26 Feb 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: How to roll out Docker tag expiration policies to all projects

Description

No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).

A

Okay, so, as all of you, many of you know I'm sure that we launched a new feature in twelve eight four that will be that was focused on docker expert AG, expiration policy, so it basically leveraged the existing container registry API that we have the bulk delete API to allow users in a user interface to fill in which images they'd like to keep using regex when they would like to run deletions like what's scheduled, they'd like to run deletion and how many images they would like to keep.

A

So we launched this feature only for new projects, assuming that there was this performance issues with that both delete API and we didn't want to overwhelm the system or crash anyone's self managed instance. So we were only rolled it out for new projects.

A

The flip side of that is, we got some feedback right away from the community that they want support for this, for all projects and I was talking to those user, the community and saying. Well, we have these concerns, and so this conversation is a follow-up to that to say what has to be true for us to roll. This feature out to all projects and what are some of the concerns that we have and how can we help address those concerns?

A

So I had some items added to the agenda and I think Steve you had the first okay, so the thing that I brought up was: is it possible to turn this on for all existing projects and can we think about self managed instances versus get babcom separately and I? Think Steve? You had the first kind of comments in the dock yeah.

B

And so I was I was kind of just starting a comment from like you know. What kind of things can we possibly do here in terms of turning on from my perspective, which is just from the like? How can we turn this on for some people, but it doesn't necessarily solve the problem of overloading the system, and so from the non overloading the system standpoint for self-managed instances, we could a day configuration option something in the admin options, maybe for just being able to turn it on and off at the admin instance level.

B

However, we, you know, that's probably something that we don't necessarily want to do until we know what happens if someone does turn it on, and they do have a lot of tags that are going to be cleaned up right away and then forget lab comm from the standpoint of turning it on really there's just some validations in place that we need to remove in order to allow these projects to run it. But we do need to first address the concern about performance.

B

So I wasn't really offer any solutions. They're, just some insight into you that that one particular side of what has to be done is.

C

It possible to so it's only for new projects: I'm, not I'm, not on your person that share the mechanism by which we determine that. But is it possible to you know, ease that back and say, like you know, projects that are year old to you like, because they're what gradually do it so that we don't overwhelm the system.

B

We could conceivably do something based on time right now. The way it's set up is all new projects when they're created they automatically. You know, like a a database record for a container expiration policy, is also created associated to that project, and so pretty much we're saying. If a project doesn't have a container expiration policy, then we don't allow them to create one like it's only if it's created automatically when the project is created that we allow it to happen.

B

So we could add some additional rules there. That.

B

Would we Strix based on time not just based on whether or not it's brand new yeah.

C

And time is just an example: I mean some some way to introduce this in a in a phased kind of way. Just if, if the thing we are truly worried about is that everyone's gonna use it at once, and that's just too much load.

C

B

I think one of the concerns is just if someone turns it on on a project that has a large amount of tags regardless of age. So you know, for example, if we turned it on on gitlab, there's a very large number of repositories with a very large number of tags, and if we someone turned it on and said, keep 100, then that would probably break things.

B

So one of the questions that I have, which I think is question number one. It is what is involved in cleaning up the bulk cleanup process, which is calling that docker api endpoint for deleting tags.

B

There's two sides of it. There's one we're calling that endpoint, which is maybe known to be slow and then there's two when we call it where we're finding all of the tags we want to delete and then calling it for each tag, one at a time, rather than making a single request for all tags. So I don't know. If there's different approaches, we can look at to help with that. But that's something I was kind of curious about yeah.

D

So basically, right now there.

E

Is a difference.

D

Between deleting attack from the API or the front tanks, or deleting a tiger synchronous, which is the one that I believe we're talking here, so that's different in 12:9, we are introducing the performance improvement for the tech elites, both for the API and the UI, and that has moved from four to five five requests to delete one tag, just a single request to delete attack so that that's that's not much less effort on the network and on the on the systems 2 litre tail. But the improvement was not applied to the synchronously deletion.

D

The asynch resolution continues to use the old way of deleting text.

D

So if you are dealing with a lot of loads because of this feature, maybe it's worth to roll out the improvement to the to the asynchronous the elite as well is.

B

There any reason we can roll it out to the async version. Like is there.

D

So again, we would have.

B

To change yeah.

D

We have to change a class, but it's definitely possible. We only change if the V, API and and the front end, because that was what was being asked on the issue and we also wanted to do a control, a rollout of the new feature in case there was something going wrong with a new delete, at least the asynchronous part, with the would work as expected. But now that the new feature is live for more than a week and everything is ok, we can probably roll that to the asynchronous deletes as well.

D

So that's one of the follow-up things that I've noted on the on the merge request for the delete improvement. I can probably post a link here.

B

Do you think that if we implement that, like does that one change to solve this problem, where we could just say like we are confident we could turn it on or do we want to fix other aspects as well? All of you, yeah.

D

So that will only work if the container registry being used is the container registry, so that will only work for sure for get Lancome, because in the other places we are not sure what registry people are using the bit like what our registry or a third party one. So even with with with the performance improvements if people are using a third party, but the registry, it's a lot with your fault to the old behavior, which is the slow give it okay.

D

So even that wouldn't be like silver bullets. It would just be a little bit more help to ensure that we don't eat any performance constraints.

F

A question why, for all the project, we don't invert the policy and we say instead of three ten hundred like delete ten executed ones every second day or something like that, so that we can slowly chip away until it becomes to a manageable way that then you can run it normally so like we can use the policy to slowly attack the very old stuff in a controlled way, so that we don't float any server.

F

We don't store a drive and then, when it goes under a certain amount, then you can start using the normal and delete everything. Minus one. Will that make sense performance wise or it just doesn't work with this I.

D

Think that's possible.

D

And I think the key here is probably is probably to figure out an algorithm to let us roll out these progressively to more and more projects and make it available to more and more projects like every day or at least every week. Every week we could then lock the feature for projects that are older than a week before the release happening, and we can continue to do that for several months, for example, because yeah. If we turn this on for a lot of projects, we may have phones, but.

F

D

Not sure what yeah.

F

Sorry, can it be that one project still generate a problem, say the CNG project and get a lock on that? Has all our doctor container right? Can you be that one project generate issue? It.

D

Depends on the load I'm, not sure, on the limitations on the async use a little bit sidekick that we are using for villain and, of course it also depends on the number of instances of the container agency that will be under oath under those circumstances. So, to be honest, I don't think we will be able to answer that with a high degree of certainty before we act. We try and expose that future tomorrow tomorrow tomorrow, because.

C

This is an async job. Anyway. Is it possible to put these these tags into a common queue, and then you know whenever the job is run like we have a maximum limit that we know is is acceptable and that will produce an acceptable amount of load and just chip away at the queue, because I think this this address is the problem of well. What if there's one repository with that's gonna delete 10,000 tags at once. This might resolve that I. Think an issue with this is that you know get lab comm MA.

C

Somebody might you know. I could see users complaining, it's like well, I. I click delete you know week ago, and it's not still not done, because it would take some time at least fitting. They shall run I.

B

Think I missed you're, saying using a cue system, so you mean a cue for each individual run or because, like I'm thinking, if you have like someone that runs a 10,000 clean up and then someone that runs a 5, but it gets queued after the 10,000, then that you know the person that's just trying to delete the 5 tags. It never seems to happen. Yeah.

C

That is the that is the not necessarily a queuing by I'm requesting these mini tags, but queuing all the tags in the request, if that makes sense, so like each tag is, is queued up regardless of the the project but yeah that does have the limitation of like if it could be a while, especially for an initial job, on a big big instance like it left at home before it moves, it can be seen as being not effective in you know not doing the thing, rather than just waiting.

D

And I think another way would be to apply throttling on on the on the queue. So even if you someone requests the volitional for 1000 text, we could, for example, define that the queue will not be spatch more than 100 deletes per minute or per hour. Let's say so. Requests will just queue up on the queue and they will be served on a constant pace like in bursts. At least that would be a controlling approach. We would always have an expected behavior, because the loop will always be the same I'm. Looking at I'm.

D

Looking at the issues and the merge request and get love and there's, there seems to have been a change three years ago to a low sidekick jobs to be throttling I'm, just saying, link I'm, actually I'm, going to post it found on the document as well. So maybe we could attempt something like this.

C

Yeah sounds like a good idea, but.

F

Maybe we can directly limit the algorithm output right and it's not the phone, the queue we can say if the algorithm is spelled out that we need to delete the policy says, meaning to delete a 10,000 targets that is going to be 100 right 101. They will go back to 100 and we can like document this for the user directly on the UI and say we don't delete more than 100 at once, and then it will take a while before their retention policy does its job.

B

Think that makes sense so I'm just thinking about this from a technical standpoint then versus what we already have. So, let's say, like somebody did, have you know a thousand tags, so it's going to take.

B

10 runs with the idea, be that if we're running it once a day, maybe that if they say I actually want to run it once a week, I think like the tricky part is like if they only want to run it once a week, but that first time it's going to take 10 times to run through it like being able to tell it to run for 10 days in a row and then something they switch to well, you know only running once a week maybe be tricky.

B

At least I, don't see how that's easy to do.

D

The other option, instead of throttling like you, would be throttling the the service that is processing the the requests like saying you can only execute like five times per day or ten times per day and at most you can be executing like for ten minutes five minutes and whenever that line is reached, we just stop deleting whatever is left to be deleted, and we will continue on the next execution.

D

So that may be an alternative, but the the IV is basically the same. It's the only difference is where we are going to throttle application or on the cube.

A

What about anything like failing, gracefully like when you encounter an error and and not just you- know, getting stuck and running consistently, but that might be another thing we're considering, because I think we've seen.

A

Sometimes the process fails like when you we just change this in the garbage collection process where they delete it, where the corrupted manifest would cause errors. Would we need to would we need any special error handling for the delete process to consider I.

D

Think the main concern is not actually the others deleting something it's likely because since the container registry is sheriff for other services as well, so listing takes people that are pushing images either through the CLI or CI.

D

So if the registry is under really heavy load, it makes stop processing those requests or take too much time and eventually a timeout will trigger and we will start impacting other functionalities beyond the deletion of text under the expiration policy. So I think it is worth to find a limit this to make sure that everything else is not affected.

A

So so far we've been talking about separating out potentially get lab, calm and self managed and Schwalm. You had some concerns about that, because it's possible self managed is using external registries.

A

Is that can we move forward even and just like say that? That's okay and just make a note that this won't work for external registries like if you're using ECS it won't work. It's all. This only works if you're using the good lab container registry. You.

D

Mean the exploration policy feature yeah.

A

D

Limiting that to to our container registry yeah.

A

D

A

For all projects, if we did it I.

D

Think that may be excessive, because the functionality that the recorded functionality is there. The only difference is that if they are using the clock container registry, the actual deletion will be fast, but people that don't use the control agency should also be able to use it because the underlying functionality exists, so they are still able to delete tags using the nodes work around it, which is slower. Okay, we can possibly just add a note saying if you're, not using the deadlock, container registry or deletion process will be around 80 80 percent slower.

D

So maybe that will also commits people to move to work on the registry at some point, but I think just blocking that and not making it available to self-manage these instances or those not using our container registry is maybe a bit excessive. I would prefer limiting the throttling then actually limiting the functionality.

A

Okay, so we could we could enable that well once we, the other change we were talking about is the asynchronous updates, and that requires just changing something on that we've already done and applying those changes once we do that, can we then focus on enabling this for self-manage instances is. Anything else have to happen on our end to make that work or there any other concerns preventing us from doing that.

D

We can apply the the first technique to the synchronous jobs whenever we want a lonely work project, for instance, is part of why the hotel registry- and that's the only that's the only limitation I've- refrained from doing that, because the problem was on the UI. It's not an issue if I give it takes half a second more if it isn't a synchronous job. But now we are talking about a lot of load and a lot of the elites happening. Asynchronously, so yeah performance will matter as well, so yeah we should.

D

We should apply that as well. I would prefer to do that once we have a decision around the long term compatibility plan with third-party registries so that we don't end up having more and more workarounds on top of workarounds to maintain compatibility with with our the registries. But if this is a concern now we have to do it.

A

Okay, I, don't want to talk so much, but I do one. We have eight minutes left so I'm wondering it. Are we zeroing in on a solution or idea, or is there more work that we have to do to think like? What are there questions that we haven't asked yet.

B

It sounds to me like there's, there's sort of like two aspects in which we kind of hit on which is one. Turning on the change that's already made for the async version, which solves Gilad, calm and, and anyone using or container registry and then implementing some sort of throttling to handle all other instances of this. But it sounds like for safety. We need the throttling before we turn on the other version, because if we turn it on, then anyone that's not using our container registry might have problems.

B

So it sounds like there's like the two steps are implement some sort of a throttling and then update the async to use the fix, that's already in place, so the real, the the next step would be, like you know, figuring out how the throttling would work, which is a little bit. We can probably continue that discussion and an issue on a more technical level, but I'm guessing that. That sounds like the path, unless anyone has any other thoughts, yeah.

D

Throttling is mandatory. The performance improvement is, is uh nice to every basically I think it's worth to look at the modulus tattling thumb on the documents about the throttling for sake crops. Maybe we can leverage on that to throttling without changing the code, which would be nice, but.

F

There is something that we need to do short term and we were discussing this with Steve today, so the expedition policy Stern it's available for project for new projects right, but is not turned on by default. So if a new project is born into 0.8, they go for three months: they pile up 1 million tagged. It becomes the same problem as an old project which may be a million tagged accumulated over three years.

F

So we should start to think that maybe we should have the exploration policy turned on by default, with some sensible default and then into updates on the short-term like. This is something that we can do rather quickly and it will patch up future problems.

E

Perspective I think it would be sorry, I don't mean to jump in, but I think it would be really great if we did have that default on situation for all new projects just because that would guarantee we don't end up in a situation where there's already so many images and tags that it's difficult to deal with and also follow us along with get labs overall products theme of like on by default.

E

So if we can move forward to that direction, and that would be from a UX perspective, it would be the preference because that's kind of the ideal scenario we want our users to go down. I.

D

Think it would be good to give it up to to the infrastructure team saying we have a feature here. We are going to turn it on. Can everyone please keep an eye on statistics and metrics to make sure that we are not overloading the system and in case something what is what we can act and either disable the feature or spin more resources to handle yeah.

B

Wondering if we have, if we do turn it on I, feel like we'd want to like. Let users know that you know suddenly I mean any project that has a container registry or container expiration policy is going to suddenly start having tags deleted like we won't turn it on and I suppose anything that up to this point, that already is like currently turned off. We shouldn't turn on. We shouldn't just like turn it on on users that are unsuspecting, but if a user creates a new project and- and you.

D

B

Up tags, they should at some point have some sort of message on the front end or somewhere. That says you know you are using the container registry by default. Things are going to be deleted regularly.

B

Just so people know I think.

D

It is already yep yeah that is already a warning for auto die. Folks. Whenever you create a project, you see a warning saying a lot of their jobs has been enabled in because you don't want it just configure it by yourself by creating a github CI file. Maybe we can have something like that saying automatic special policy is being set up. If you don't want it, go ahead and turn it off or just settings.

D

F

Thing that we could do immediately is to display on the container registry that says the next policy is going to run at this time. At the moment we already have to seem for ready in the API, so you can just put on the front end, and then we could have some nice to have and saying like X amount of tags are going to be expired, or this tag is going to be expired.

F

Maybe it's not, but we can't really say expression, policies on and it's gonna run in a week and go check your settings and then I see link to the settings. I think.

E

It'll be good as long as we get started, where we kind of make it obvious that the feature is now enabled and then, as we train our users, that this is like the default of what we expect our users to be. We can start winding up afterwards, and we can do that during the rollout of that expiration policy as well and there's quite a few areas and get lab where we kind of display like hey. This features on just so.

E

You know, and our users are pretty used to seeing that and making adjustments there in where like. If they didn't intend for their retention policies to be enabled, they can go and fix it. Then we can make it pretty obvious.

A

What a show of hands are one being the as little confidence as possible and five being maximally confident. How confident are you all that we could solve this problem but bite and turn it on for all new projects? With with the plan that we have that we've been talking about.

B

You mean in a certain time span are just in general in.

A

General and they'll say maybe wait and then that yeah that's a great follow-up question I would be curious. The level of effort to, but you know, I see a lot of force. I.

B

Think confidence in doing this is high time.

B

That's a different story, but.

F

The good thing is that we can divide this in step right. We can enable the feature of like the expression policy for everybody with sensible default and a lot alert. Then we can obstruct link and then we can have like performance update on dancing, possibly ocean parallel. So there's our platform.

G

A

It sounds like there's four issues to open. One there's two front end is shoes. One is to display when the next policy is going to run another one is saying that the this power enabling it by default and saying that you know this is enabled by default, then there's the throttling issue and which maybe we'll luck out, and we already can control that with that existing my Rumson and then there is the issue of turning it on for all existing projects, first for self-managed and then for get lab calm.

D

Okay, the performance improvements for this includes programs, okay, so yeah it looks like we can work on some of these in parallel like the first one should definitely be the warning, but on the container registry page showing when the next one is going to run and also the warning for new projects that enable it and work. But before that we need to work on on the throttling, that's mandatory, then we can level it and optionally improve the performance of the en suite.

C

Yeah I mean I, think I think it's me, I mean really did if we came to the track like outside of the registry just cuz. This enables this fix to work with non get web container registry container registries. Otherwise you know difficult.

A

So we drawn and Haley, can you open the throttling issue and the change for rolling out the asynchronous deletes and then Nico and Ian? Can you open the issues for the the front-end and then Steve? Do you have any issues to you could use the existing issue that we have that's rollout support for all projects, yeah.

B

We can use that issue and I'm sure I'll be really involved in the throttling issue to you. Yeah.

D

The throw thing that complete meets to a panel home on the real site, yeah.

B

Yeah, how about I can open the throttle in one since it's the rails side of things, okay,.

A

Cool thanks everyone, it's hard to see all that feedback in the issue of people, saying that they were mad, that this didn't cover, but at the same time, it's nice to see that people really want this feature and that you know they're anxious to have it in their hand. So it's it's cool to see, and hopefully we can get this out and in their hands soon and I'm. Gonna stop recording.