GitLab Developer Evangelism Team, 12 Aug 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Developer Evangelism demo: GitLab Storage Analyzer v0.1.1, 2022-08-12

Description

Michael Friedrich walks through the first v0.1.1 release, the job artifacts report summary as markdown table file for projects and group with project, and shows the deletion of job artifacts using the CLI script with Docker and from source.

GitLab Storage Analyzer is a new Python project to help with analysis and cleanup, developed by the Developer Evangelism team.

Documentation and usage in https://gitlab.com/gitlab-de/gitlab-storage-analyzer

A

Hi, my name is Michael Friedrich I'm, a senior developer, evangelists gitlab and today I want to talk a little bit about storage usage and how to manage it.

A

um I came across the usage quota uh within the storage tab in projects, so basically this documentation and how to view it um and was wondering whether I can automate things, because this view provides me with some insights and I know I'm, currently using 36 megabytes, but like how can I automate this and um I've then looked into grinding as a script, using the gitlab API using python bindings and um have now a current state of the project which, which I created, which allows me to analyze Dropout effects in order to generate a markdown table an overview to immediately see what's going on and in addition to everything like show something uh the project.

A

Also, the script also allows you to Define thresholds, which means you can either filter by age or by size and then get, for example, an overview Mark to highlight that in that way, um by default, the script doesn't do anything rather than generating a markdown report into a file which I will show you in a bit and the other thing it does, or it provides the functionality to actually delete something either based on the threshold filters or delete everything so deleting all artifacts and you can either pass along a project IDE or a group ID and the the script will then try to collect all the projects and all the drawback effects.

A

Underneath now um everything is documented. You can either install it from Source or use the provided container image. um The only real requirement is providing a personal project or group access token, with either read API or API permissions, depending on what you want to do now. um Yeah and there are. There- are some other configuration options as well, but in order to look into that, let's quickly switch into um terminal and let's see whether we can generate something.

A

So this is basically running Docker Expo exporting the gitlab token, the project ID, which has been prepared- which is actually this project over here, so um just to ensure that we are looking at the same numbers. It's 38, 4 ish, and it's over here 30th voice. The idea is to generate this into a specific output directory, which gets mapped um essentially, and the script also dumps everything which it generates onto the terminal.

A

So we can immediately see it over here as well, but in the end we can see the job at effect, size is 36 megabytes and if I want to open this up, this was generated over here into this directory and we can see the markdown table and the nice thing in uh Visual Studio code is command shift, p, markdown, open preview, and we can see this rendered probably there's an error.

A

Probably we should maybe increase the size a little so that the view is is even better and the good thing is, or the thing I've added today, actually because I started on Wednesday, adding all the URLs for the project. So you can immediately jump into the user's usage quotas.

A

For example, or investigate the job IDE, um which happened for generating these artifacts I've, actually created a small generator script, which acts Standalone in a gitlab CI file, which is great, so we don't need any extra scripts or so um the generator's job is actually using the the parallel Matrix builds instrument into different count, sizes, which you can then just like include as a remote file to test and just generate some artifacts in the background it's using DDE and and you random, to generate the different block sizes but yeah in the end, everything works fine, and this is amazing to test now um that we have generated this output.

A

um We didn't do anything yet so in order to really do something- um and this will be something I will be doing now for this on this side in the terminal, let's see yeah we can.

A

um Oh I was actually looking at a different thing.

A

um This is the group ID which I was looking at, so you can also export the group ID into the terminal and then run the analyzer I'm just doing it again, and um it should be generating uh the gitlab analyzer. No, the gitlab analysis report. That's the name and we got a C um again a little longer markdown, which oh, it's not a little complicated to to show everything Maybe.

A

Let's move this one over here and close this one um but yeah in the end, it's a longer listing it's not beautiful yet, but still it provides some output and we do have a Dropout effect size in summary of 360.. Now um the thing I want to show you actually is. We can also say hey. um We want to um add the delete method. Let me quickly check the documentation.

A

um We can Define a threshold, that's the first thing we want to do potentially just using an age threshold of, for example, let's use one hour which is like 60 multiplied by 60 and run the script again, and we can see we get an additional information about the output, saying that the age threshold over here would potentially allow us well to to delete all the artifacts, which is great we can. We can also see it, gets highlighted in this direction now.

A

The other thing I can do is in addition to this threshold and it's great writing documentation to being able to copy paste. We can say we want to actually export the file size and say this should be matching more than 10 megabytes and I'm, setting the H th threshold again.

A

Which then, should bring a different uh report summary yeah here it is the size threshold is actually more than 10 megabytes, which would allow us to reduce the size by 280 megabytes. um This is a different pattern. I I try to implement I'm, not sure if it makes sense to delete by size, um but probably it's it's also helpful. Now the thing is, nothing has been deleted yet so everything is like generated. You can analyze this.

A

You can upload the markdown file uh into a git repository, for example, to render it you can upload it into snippet into a Wiki. Whatever is uh click quiet?

A

um The script connect actually like if you want to use the docker container or um the script itself, just run it in CI CD and generate the reports just be aware that um it requires API calls in the background, so you might be flooding the API um a little bit and might be great limited because of that so I would recommend maybe running the script once or twice a day or something like that, um but in order to really actually delete something, the threshold doesn't do anything.

A

The delete mode needs to be specified, which is none by default and in our case we want to delete everything which has been matched by the threshold.

A

So we are setting this and now we can actually see that um the analysis happens again and the script starts to delete the chops with the add effect and the sizes and prints that so it will take a while, because this is like an API call for for everything and after a while um it should be finished and when we then run the report again, uh we should be seeing that there is not so much um data left, so we can see uh just run it again.

A

That we have deleted all the ad effects which have been matched not sure if whether discrepancy is coming from, probably that's um I I think I found the buckets. It's also calculating the the chop. Traces and everything else um probably gonna fix that, um because currently, um when we are looking at I, can actually look a little bit into the source code. Right now, when we are looking at the uh uh calculation for everything we should be seeing.

A

um I said calculate the size.

A

The size summary is over here and.

A

Yeah, probably I will check this out later on, but in the end, it's like it's collecting the projects, the jobs, the artifacts and then calculating everything and also generating uh the markdown output, so in in the end size sum is for some reason um a little off, but I need to investigate on this. As said, it's, the first version, so there are, there are probably some bugs inside.

A

um The most important thing is that you get to see how much or you get to see a detailed overview of what is being used and how much you can delete. The other thing you can do is you can also use the delete mode or which doesn't use any threshold, but instead we'll try to delete everything which um is with a warning So. Currently it waits 10 seconds to give you some time to abort and then tries to fire the delete operations and, in the end, um yeah now everything is gone.

A

Maybe yeah, maybe the calculation was is actually correct um in a sense of um the deletion is not immediately on gitlab.com in the background, so um this is actually like. I think it's something in the background type running and it's not like live deletion, but it takes some seconds or some minutes. So probably like me doing uh doing this demo now um have hit the I. Don't want to wait uh problem now.

A

um If we look into the set again, we can also do that. Potentially I just need to cheat so that we look into um what I wanted to do as a demo. The demo case is also documented, so you can just like copy paste this and modify the project ID, for example, and um let's maybe try this one as a last demo example and minus copy paste takes so long. I think my computer is a little overloaded and.

A

Yeah over here, so we want to delete everything.

A

Yeah the delete mode is threshold. Let's see what happens here,.

A

A

Maybe I'm getting red limited now.

A

Yeah, so um in this case we should have deleted everything, because the H threshold actually matched for everything, and you can see that the size threshold is marked with this emoji and the H2 actually with threshold with this one, and if we run it again, um we should be able to see nothing, hopefully just takes about yeah. We are not.

A

We are seeing nothing and everything has been deleted now um in order to verify that we need to look into the um Canada look into the artifact generator group into the uh we want to see the use. Where is it?

A

um No I want to look into actually I want to look into the project this one- and here are the usage quotas for the project, and we shouldn't be using that much okay, it's not updated yet, um but maybe we're clicking over here.

A

And I don't want to wait so in the end, um I'm- probably faster, with automation in this regard, but keep in mind, don't stress the apis too much um yeah and the source code is MIT licensed it's a white labor. Currently, as version serial one one yeah, the release is tagged. The release is over here, so you can either download it. I would probably recommend just cloning, the repository and um or the project and running the source or just using the the darker container.

A

So you don't need to install anything and if you're interested in contributing or I'm trying to like understand how it works. So everything is in this file at the moment. It's not it's, not the perfect python code, but it works it's using the python gitlab Library binding. So thanks for to the authors for this amazing Library, um which allows to abstract many things, yeah and everything is like I, don't I haven't found the current line, count I, think it's 300 or 400 or something, but you can see.

A

A lot of things are like reading um input and other things I would recommend either using visual studio code um to edit the code. uh The the python code with intellisense or using your own preferred IDE yeah, and maybe you want to dive into adding more storage types. Currently supported is robot effects. This is something which I think is probably the the size type which generates the most storage next to container images. Obviously, um but it's yeah I I needed to start somewhere in in the end um oops.

A

This is not the correct one in the end I wanted to and my the recording is making my slow computer, even slower, I have too many tabs opened, cannot see that um I wanted to show you. There are some issues open, so I I'm thinking of adding container images.

A

um This is really slow um container images packages, the repository usage and so on, and all the uh the issues have documentation and ideas on them. So you can immediately see what what to do now. In the end, um we have deleted everything successfully and I hope you liked the project in the script and it's helpful for you. Please use it provide feedback um or send a merch request and contribute um really appreciate it yeah and thanks for listening thanks for your attention and see you next time.