GitLab Feature Walkthroughs, 24 Mar 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Instrument GitLab with Prometheus for Self Monitoring

Description

In this video Reuben, a Backend Developer at Monitor, is walking us through the process of instrumenting GitLab application using Prometheus and displat those metrics in the GitLab Self-Monitoring project

A

B

A

Sorry good, no, no still shocking in this demo of they'll be showing how to add matrix to the gitlab product, and then we add it to the matrix dashboard of the self monitoring project. Okay, so we'll start with sharing my screen.

A

Can everyone see this yes,.

B

A

So so normally, when you want to instrument something, you will start by looking at the client libraries so get lab uses Ruby on Rails. So normally you would look at the Ruby client library.

A

Which is this one but get lab uses or Fork of this? So you can see that in the gram file.

A

Here this one from Eclipse client M map, so instead of looking at this, the github page for the client library will look at the one that get lab users.

A

I have the link in the doc.

A

Okay, so so this is the library that github uses here. You can see examples of how to use this library. Your counters are created gauge, histogram, summary, okay, so so this Fork was originally created, I think because we needed a library that could work with multi-process application servers like unicorn, so at the top. If you see it says this funkiest library is a fork of the Prometheus ruby client that uses memory mapped files to share metrics from multiple processes.

A

Okay, so that's a brief intro to the funkiest client library that you use. So what matrix should be be adding to the gala product that will actually be useful, so one one thing that I could think of was measuring the amount of time it takes to process custom dashboards in our back-end. So.

A

The processing happens in.

A

Yeah, so this is, the processing happens.

A

As you can see, it calls this and then it loops through all the stages. Okay, there are multiple stages and it looks through all of those. So we want to measure this.

B

A

Basically, we are measuring.

B

We we are measuring an action in the UI in github, the app itself and user can select more or less any action that you want measure any metric that you want and to instrument it in form or less the same way that you are doing this right now.

A

Right so I'm basically skipping a few steps here. Normally you would look at you know what you want to measure and then you would hunt down the code that you know that actually does it. So, for example, here what I thought, what I had done was I looked at the matrix dashboard and then followed the path from there look at the API, which serves that then find the code which actually does what you want to measure. So, for example, this is the code that solves the matrix dashboard page that we almost become level so.

A

Okay, so I'll just show y'all the library that we use to make it easier to add matrix and measure stuff.

A

So this is sort of like a helper. This is the one that we want.

A

So this is the Prometheus helper and you can see there are methods provided to create counters summary gauge histograms. The rest are not important for now.

A

Yes, so let's go back to dashboard base.

A

So now we have a choice of the four metric types. A gauge met, counter summary and histogram, so we want to measure processing time which can go up or down. So we can't use a counter because counters are supposed to be only something that grows. It cannot be a value that also decreases so a gauge is possible, but the way you measure it makes it difficult to use a gauge, because the way pro meters works is that an application measures something and serves up a matrix page and Prometheus periodically scripts it.

A

So if you use a gauge, you will be measuring values. Each time a user hits the dashboard endpoint, which is you know it can be hundreds of times a second. Even so, what we want is an average over time. We don't want from Ethier's to pick up one value in a minute. We want promises to pick up all the values in that minute. So what we use is summary which, which automatically gives you the count of the number of times that you've measured, something as well as the sum.

A

So we can use that to get an average sum by count. Okay, so.

A

A

So this block of code is what we want to measure. Let's use the benchmark library.

A

So this should give us the time and seconds that is cooked to complete this.

A

So this processor has one characteristic that if you can see this here so if dash board does not exist, it can return nil and when it does that it does not actually do any processing. So the amount of time taken will be very low, so just so that we don't skew the average. Let's exclude that case if.

A

Dashboard is maybe boy if dashboard exit then.

A

Then we need to observe that value the processing time value so before we can do that, we need to add the metric.

A

So this method takes two parameters: one is the name of the metric.

A

According to best practices for naming metrics, you should start with the name of the app in which the metric has been added. So that's why I have added gate lab at the start and the unit is usually at the end. So in this case, let's use milliseconds and the second parameter is the help string. So it should describe what the metric is measuring.

A

A

The observed method again takes two parameters. First, is the labels I.

A

Have not defined that as yet, but we'll do that soon.

A

So the the value returned by benchmarked or real time is in seconds, so we'll multiply by a thousand to get to get the value in milliseconds and now we will define the label.

A

So, let's add two levels: one else for service that is which service is currently processing. This dashboard.

A

And we'll add a label for stables.

A

So this calls the sequence method, which gives an array of stages.

A

Each element in the array is a: it is a class, so we take the name of that class.

A

Let it take the last bit so.

A

Just to give you an example, one of the classes can be, for example,.

A

This can be one of the stages.

A

So you can say, get lab metrics both stages. So this is a very long name and all the stages are under this namespace, so they will all have the same names so be only interested in the last one, which is why we are splitting by double colon and then joining it. So when we see the values of the matrix, it will be clearer.

A

Okay, so we have those interim levels.

A

Okay, so this should give us the matrix.

A

I'll start, my DD here.

A

So while JDK starts is anything unclear about this any questions I.

B

Think it's maybe while it starts we'll take the time to, can you pull up the list of the available metrics because obviously we get lop itself is already instrumented. So.

A

B

Is an out-of-the-box metrics, the metrics that we've instrumented already in the code and we shipped it to our users, so I guess the first action that the user that would like to instrument gitlab. The first thing is: first, you should go to the to the documentation. You see if this metrics is exposed, I'm, ready and available yeah.

B

A

So yeah so, while you're deciding what you want to measure, you would check whether it's already measured. So this is the doc which lists all the metrics that we've added you can find it online.

A

It's quite a big list.

A

Yeah so I'll add it to the dock.

A

And ideally, after adding a new metric, you would add added to this list resolving.

B

Yeah, if you do it like in the product itself before.

A

Even if, if a user adds it to their own gate lab, if they feel like you, know pushing it upstream as well, they can do them.

B

Right, we do accept contribution, actually we recommend user to send it upstream and we will gladly accept any contribution of new metrics.

A

A

So I already have the self monitoring was decorated. If you don't have it, you can go to the admin area under monitoring, settings monitoring and create the self mountain project.

A

So before we check this, let's look at the metrics yeah.

B

Maybe maybe say a few words about the gate: flop, self monitoring, because this is a new feature. Okay,.

A

So recently we added the killer self monitoring project whose goal is to allow users to be able to monitor the gitlab instance itself right out the right out of the box. So they don't have to do anything they'll. They can just go to the admin area. Settings matrix, enable the self monitoring project- and you know eventually, we hope- to have multiple dashboards in the self portrait project which show you matrix of get lab itself.

A

Sorry- and you can create it here,.

A

B

It acts like any other project project that you have this. This is, it is built for monitoring lab instance right.

A

So we would go to operations matrix to look at the matrix dashboard, but before that, let's look at the matrix.

A

So if you go to the admin area under monitoring and health check, you can get the link to the gate lab matrix page, which Prometheus scripts.

A

So right now that metric does not exist because it hasn't been created. Is it so, let's open the matrix dashboard so that it gets created.

B

So it is not created when you define it it created only when you invoke an action that will trigger some sort of the response time. This is this only then the metric will get exposed to meters all right, yeah.

A

A

Looks like those amazing.

A

Okay, so right now we are only interested in the API requests to get the actual dashboard which has completed successfully. The other things are not working, because Prometheus is not running right now, we'll draw on that later. So now there should be two metrics here. It should be, should have run twice.

A

Yeah, so if you can see the count here is two because I opened the dashboard twice and we also have the sum so to get the average. You do some by count simple method. Okay, so so now, I'll just show you all the docs which describe how to monitor your GDK instance, where Prometheus.

A

Okay, so you can download the Prometheus binary then.

A

Then you need to add this config into the Prometheus config file. I've already done that, but I'll just show you.

A

This is visible right. The terminal.

B

A

So there's a dot EML file here, Emma and if you can see here at the bottom, yeah I've added the configuration for Prometheus to scrape gitlab. So you have the matrix path. They have a token and you have the target so.

B

Just explain them: do you need to do it also in the GDK like.

A

Know if you're using omnibus, then everything comes automatically set up, there's even a Prometheus system. But if you are running get love from source, then yes, you'll have to set up your own Prometheus and.

B

To emphasize use the official gate luck, you don't need to do that. Prometheus is already installed and set up. This is just for the game right.

A

Gvk and installation from so Omnibus, then you don't need to do anything just like in a bit about this, so the job name. This is the matrix path which you can see in.

A

C

A

Check page so, for example, here you can see the matrix part is slash matrix. So that's what we've added here and this token again, you can obtain from the health check page and the target is where your get lab in students runs okay. So we have this one big setup, let's run from here.

A

Selamat is running.

A

Let's check the targets.

A

It hasn't scraped gate lavas, yet.

A

So by default, Prometheus scrapes the targets once every minute you can change it to whatever value you want. Okay, then it has crepe target lab. So now we should be able to see the metric that we want yeah. So here you can see the count and the sum. So let's look at some by account.

A

Okay, so it's 140 milliseconds, currently, okay! So let's add this to the self-monitoring project.

A

I think I have I've already added it yeah, so I'll just show you the manual configuration. So this is the URL of the Prometheus instance that I have running on my local system, and this is the custom metric.

A

Okay, it's the same thing that I just find it's the sum by the count.

B

A

I've added this so that we see each individual service that processes both.

A

Okay, so now this should work.

A

A

This is the current processing time value 140 milliseconds. This.

B

Is the new metric, so by default we have two: two charts web service, an API service and you've added the customer to that. We see here that for me to cease picking it up and.

A

So um this value that you can see here is from the service label and currently it's showing the self monitoring both of this. So, for example, if you, if you open the system dashboard or if you open some custom dashboard, you will see those labels as vary here,.

A

Let's open the dashboard again just to.

A

Get some different values.

A

Okay, so the amount of time taken reduced a bit so that that's how you add a simple metric to measure, something that, like you, want to measure and.

B

If you want to add more, you just click on the add metric button on the upper right side, but.

A

B

A

So this is the same UI, which I've shown just now.

B

A

Custom metric form, so you can add your metrics. You can also add metrics by creating custom dashboards. So that's also an option.

A

Any questions about this I.

B

Think it's it's clear to me.

C

C

A

C

Notes but Ruben, if you have some time just like last time, I'd really appreciate if you could just go through and see if I missed anything important, if you can add anything to provide more context into those steps. That would be super helpful, yeah.

A

Sure I'll do that.

C

B

Going to upload this video and also I'm, going to update the direction page, we have it an actual page for the self-monitoring we're going to put this video there as well. Hopefully, this will help our users.

B

Fine escape any metrics that they like.

A

Yeah and I'll be opening a MRR for this, so the metric will actually get added to add it to our code base. So we can start using the values that it shows us to improve performance.

B

A

Problem, okay, so you every don't, don't have any more questions. I guess we can end this. Yes,.

B