GitLab Feature Walkthroughs, 12 Mar 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: How to instrument Prometheus metrics in GitLab

Description

In this video Reuben, a Backend Developer at Monitor, is walking us through the process of instrumenting an application to track Prometheus metrics in GitLab.

A

About the recording who's.

B

Okay, so today we'll be walking through the process of instrumenting, an application which is deployed in grid lab, so for this demo we'll use the Tanooki ink project. I'll share my screen.

B

So can everyone see this.

B

Okay, so I the tanuki in project into my under my username. So if anyone is not familiar with this, this is a simple application which allows you to generate sentry errors. So let me let me run it on my local system.

B

I've already it on my I've, already cloned it on my local system, so yeah, it's updated so on your local. You can Don it like this.

B

Now it should be available on local oz mm.

B

So here you can see it's a very simple demo project, so what we'll be doing is we'll be adding metrics to track how long this server takes to the spawned for this page, since there is only one page in this project.

B

Okay, so first step is to.

B

First, step will be to add metrics to our project and the project has to export metrics in the format that Prometheus accepts. So we look at Prometheus, client libraries, since this is a node.js project, we go to node.js.

B

And this is NPM library.

B

So, let's, let's add that Larry.

B

So the command to add.

B

Okay, so while this downloads the library and saves it into the filing we'll move to the next step, so the response time is the other library that we also need here, because it's just it's just to make a evoke easier. It makes it easy to check the how long the server took to respond on a particular path.

B

Let's install response time.

B

So once we have those libraries installed, we need to tell application to export metrics I've already added the code. I had added the code and then I reversed it so I'll just copy it back.

B

In hold on for two minutes.

C

B

Okay, so now we add I'll copy this code and Analects planet.

D

Where did you copy it from? Is there a place where the user would be able to find an example? No.

B

No, so this is a commit on this project that I made a few days back, while I was testing it out so just to save time, instead of typing out all this I'll copy it from my previous comment.

B

Okay, so this is the Prometheus client that we just downloaded and saved. Then Prometheus has these data structures which you can use for matrix gathering, so they have the counter the gauge, the summary histogram. So here we'll use the summary- and this is the register value register a metric, so I'll show how it's used.

B

This is our new metric, so the name is the new key responses m/s. So the recommendation is that you prefix your metrics with the name of the application that it refers to, so that it's easy to identify metrics later, then, you add some help text and you add labels.

B

This is a method to tell the Prometheus library to start collecting default matrix so default. Metrics are stuff that every node.js application has.

B

Now this is the place where we collect how long a server took to respond to any pay requests.

B

So here you're adding values into the summary. The summary that we created here.

B

As you can see, the labels also added, so you have the method, path and status which are taken from the request method, URL, which is the path and the status code and the time which is provided by the response time. Library is an observation which is added to that summary. I hope, I, hope, I'm, making sense here.

B

B

This is this is the place where the matrix path is added to this over. So as I was saying, we need our application to export matrix and a format that Prometheus accepts, and it's usually on this path, slash matrix, so we'll be using the same default path.

B

Is there anything not clear here.

B

A

I think the question is: if I mean you use, how do I know that those are the things that I need to add? Is that is there documentation some well.

B

Study the last bit I didn't catch I'm.

A

Saying in documentation, but just the cottages have just added: how did you know.

A

That's the question: where can I go as a user? Okay,.

B

So usually the best place is to look at the description of the library that you think so. For example, I used the node.js library, usually in the readme. There is examples for how we can use that library, so here here, for example, it's showing you how to use it so we've done something similar require a prom client require from client collect default, metrics.

B

We've created the registry.

D

B

B

Here shows you how to create each data structure, counters gauges, histograms and.

A

B

One we are using summary, so what you want to measure? That is something that you'll have to decide based on your application, but how to do it? Usually you can find it find information about that in the description of the library that is using and.

A

B

Libraries are documented here in the Prometheus Doc's, so this is a good starting place.

A

B

Any other questions.

B

Okay, so now that we've added this, this counts as a separate module in no Jess. Just adding this doesn't really do anything. We can see that here we have the server running.

B

Now, if we add metrics, there will be nothing because we're if you are familiar with nodejs the you still have to tell the server to serve up metrics on this route. So we'll do that now.

B

This is the existing server code, so.

B

This is what we need to add.

B

B

We are loading the Prometheus module that we just created. Okay, then we tell the application to use the response counters.

B

So we have that here. If you remember response counters was defined here. So this is where the metric actually gets measured. You could say all observed.

B

Then we tell the application to serve up the metrics endpoint.

B

And we tell from it is to start collecting metrics.

B

Okay, so we've loaded Prometheus, we've added the response counters method into the middle way. We've added the metrics route and we've told promises to start collecting metrics. So now there should be a matrix, endpoint.

C

B

So all these here are the default metrics that if you remember, we told Prometheus to collect, collect default metrics.

B

The only metric that we've added is respond. Okay responses, m/s that you can see right at the top canopy responses m/s, so right now there is nothing because we've not made any requests to the base route, which is this after adding the metrics. So if you refresh also Italy okay, now we have metrics here, so you can add more by simply refreshing. This.

B

So now you have multiple requests.

B

Okay, so let me just check if we need to add any more code.

B

No, that should be it okay, so now that we have our application actually exporting metrics in the format that Prometheus likes, we need to tell Prometheus to collect these metrics. So for that there are two methods.

B

One method is documented in our documentation. Let me just show you that.

B

All right, so one way is to add this file into your repository with whatever values you want and when deploying gitlab will pick up the values from here and use them so we'll try we'll try this first or I think it'll be better to try this one okay. So the other method is to add this variable into your CI CD variables and add the appropriate value. So the value I think is not well documented. The kind of value that you need to add in order to have Prometheus collect metrics from your application.

B

So we need to add this key with this value.

B

So just to mention this is under your project settings MC ICD, and then you expand the variables section.

B

So we have this variable set and now we do a deployment.

B

Before doing the deployment, I'll just show you how to how to open the manage Prometheus, apps, UI, so I'm, currently using kubernetes cluster from monitoring project.

B

So we'll connect to this first.

B

Then they'll list out all the pods.

B

One of the pods should be the Prometheus server yeah this. So if you take this pod named prometheus Prometheus server and.

B

You put it in a port forward, come on so here you have cube serial port forward, then the pod name and the port that you want to access on your local system :, the port on which Prometheus is served on the remote system, ankit lab, managed apps is simply the namespace. So here you can see that from here, server is available under the get lab manage apps namespace. So that's why the iron flag is required.

B

So now you run this and now you should be able to access the Prometheus server from your local machine.

B

Yeah, so this is now the Prometheus UI given by the Prometheus manager app, which is installed on that kubernetes cluster. So let's check the targets.

B

These are all default targets: kubernetes, nodes, kubernetes, pods, kubernetes service endpoints, so here there's only one, the Prometheus endpoint. Once we run the pipeline, there should be a second one here, which is the application matrix that we just added.

B

This will take a bit of time, so I hope everyone is clear about what I just showed the community is poured forward and prometheus targets. Any questions about that.

A

Drink it clear yeah, maybe it's one is the edge of bronze. Maybe you can explain a little bit about the environment that you have disappeared, installation of primitives that you've done so make sure it's clear to everyone.

B

Sorry, can you repeat that environment and yeah.

A

I mean so, you have.

A

This cluster, that is deployed on Google, Cloud, right yeah and you install primitives on that cluster, using the geek lab manager app today.

E

A

Just want to make sure that we have the right context for anyone else that is watching this video okay.

B

E

B

I'll show the kubernetes cluster.

B

This cluster is created in G K and we've installed hmmm too low, because you require helm before you can install any other. App ingress is required to deploy the application sort manager so that we can use HTTPS and, of course, prometheus.

B

Yeah, so out of these, only Prometheus is really relevant to this demo. I.

B

Can also show you, the environment logs, just something interesting.

B

So this is a simple application which simply starts listening on port 5,000. Here doesn't have much loss.

B

That's checking Saddam.

B

So, though, you had asked about how how someone knows the what code to use to collect these metrics, so it can get complicated, because, even though you can find out the basics of how to collect the metrics from the library that you are using, sometimes it can get complicated to actually collect the value that you are interested in. So, for example, here we are interested in a simple metric which shows how long a particular path the server takes to respond right.

B

So, for example, this was pretty simple, but depending on what value you want to collect, it can sometimes get complicated to actually get that value and that I suppose there's not really any documentation anywhere to help you with I.

A

Guess all that you need to know the back and front end of your application ya.

B

A

Exactly what you want to collect, yeah.

B

B

Okay, so this has completely yeah so now there should be a new target here. So, as you can see right now, there's one if they refresh this page, there should be two yeah, so it says down because it has simply not. It has not scraped it as yet. Oh so Bertone.

B

Let's hope it succeeds the next time.

B

The default is that it scrapes every 60 seconds.

B

Okay, I guess something's wrong.

B

I'm sorry, we forgot to push the chord change that we made.

B

So these are the libraries that we added- and these are the code changes that we made.

B

Do you want me to create an amar for this, just so that it's easier to see all the changes together.

B

B

So these are all the changes that we made.

B

D

Can you please paste a link to this, mr into the chat? I will add its.

E

B

Okay, I'm not sure how to access the chat or gonna.

D

Okay, you can just send it to me later on slack, that's, what's fine I think.

B

B

Okay, well thanks now, they're thinking of this done.

D

B

D

You for walking us through the process to me. It all seems very complicated.

B

D

It actually yeah.

B

It took my car a bit of time to forget it's not I.

E

I also find it a bit complicated, but I'm gonna review the merge request and try to work a bit on it. Is there a simpler scenario in ditch in which this could be demoed? Maybe you.

B

Mean adding a simple metric.

E

Know maybe getting a feel for the whole application back end and the front end?

E

Okay, just why I feel that maybe the business logic to the bit in the way of understanding it for me but I'm sure if I, if I review your Ma's request, I will understand it better afterwards,.

B

B

Right legs and now.

E

Could you please.

D

B

I'm, this start paying.

D

You're cutting out now.

D

C

Worked perfectly fine for the walkthrough, though so grateful for that.

B

Yeah, let me check if my back.

A

A

You we cannot tell you.

D

Don't what do you think I? Don't have any any outstanding questions around the experience.

A

Now but we're not done yet.

A

Let's wait for them to be back.

B

Sorry again, yeah.

A

B

Okay, okay, so I'm three I was just saying why we wait for this pipeline to finish. I can just maybe explain a bit about the Tanooki project. Would that help.

E

Yeah I think so, if everyone would agree and if you all would find this helpful yeah.

D

B

Okay, so I think the Tanooki project was first created as a easy way to generate sentry errors. So if you see the UI.

B

It has a button for generating errors and it has a link. You, you wanna, show you.

A

Not seen, oh sorry,.

B

Let me share my screen.

B

Like can everyone see this.

C

B

Okay, so if you look at the the UI, it has a button for generating errors which generates entry errors, and these are just links, static links. So even this button does not hit the back end. It does not hit the server. It generates errors right on the front end and sends it to century. So the only route that we have in this application is this: the base route, the route route. So what I've done here is I've added a metric to calculate the response time for that route, so this will probably be familiar to you.

B

B

This is simply to calculate the response time for the route route and we told the server to use this response counters this one counters. Okay, so that's it. The application is a simple application to generates entry errors with only one route, and we are in this demo, we've added a metric to track the response time for that one Rock I hope this made things clearer, yeah.

A

And if you had like multiple routes and like multiple buttons that would actually hit the back end, then you would require to add additional additional changes to your application to capture that specific action. No.

B

Actually, no because this should capture response time for any route that exists in the application any except slash, metrics, that's because we have explicitly excluded it.

B

So basically, this has been added as a middle way, so any requests any route will go through it. So, even if we add a new route tomorrow, that should also be direct.

A

And we will see this in the slash, metrics end kind, yeah.

B

Yeah, so here you can see the Nokia responses ml. This is the metric that we added. So since it's our summary summary data type provides you with all these, so it provides you with quantiles. So this tells you, for example, this 6.9 milliseconds is the 50th percentile. So that's 50% of request.

A

We lost him again.

A

A

B

A

Out on the 15 percentile.

B

Okay, so these are all percentiles so, for example,.

A

D

B

That show you show.

A

Dui, because when you get there sure.

B

Yeah, so everyone can see this okay, yes, so the 50th percentile means that 50% of requests get get a response within six point: nine milliseconds.

B

So another common thing that is done when naming matrix is that you add the unit as a suffix to the metric name, so that it's easy to see what the unit of the value is, because otherwise, when you are looking at something like this, you don't really know what unit a value has. So here it has a milli second unit and as I was saying, you have percentiles here, then this is the sum of all these values, and this is a count of the number of observations that have been made.

B

So there have been two observations and this is the sum of all these values. So similarly, you have all these for each different route.

B

Actually, technically it's for each different combination of labels, so, for example, this is get the slash path and status 200. Then here you see, method, cap path, slash and status 3 through 4. Then this is a different path again, a different path.

B

So this is what the summary data type gives you, if you use something like gauge and counter it's much simpler.

B

Okay, so the pipeline is finished. Let's check from yes.

B

Yeah, so the new target is now off, which means Prometheus has collected metrics from it, at least once so. Let's look at some values since we started a metric with a new key, we can simply type into no key and see all the metrics which are available.

B

So, for example, this is the new key responses m/s metric. If we want to see just a slash path, we can do this.

B

So you've got all the quantiles and different status right now. There's only 200 stickers available.

B

Okay, so let's add this to the project chart.

B

Should we add it as a custom metric.

B

A

Yeah, whatever, whatever you think, is easier.

B

Okay, so this is the default dashboard, we'll add a custom matrix that it appears here.

B

E

Would be a goodbye.

B

B

Okay, so it's showing the response count for every single one.

B

If I remember correctly, you can use them, you can put a label here and it should print the value of the label in the chart. So, let's put in pop I.

E

Think it's in the legend label in the variable it's.

B

E

B

E

And you have to put curly braces around if I. Remember correctly, the braces are double. Oh.

B

Let's just put a lock.

A

E

A

Doesn't easily the why label.

E

B

Yeah yeah so I think that's fine for now. So here you can see all the different paths for which dramatizes collected metrics- and this is avocation- is now serving up its own metrics Prometheus is collecting it and get lab is displaying a chart based on those metrics.

B

So before I get to any questions, I'll show you the other method for telling get lab to telling the managed Prometheus to collect metrics from your application.

B

So the other method is to add this file. In your repository.

B

We already have the dot cat lab folder because it's used for dashboards and Yushu templates. So, let's add a file here.

B

And here we have to add the same value that we added into the variables in in the ML format. Instead,.

B

So we'll remove this Elm upgrade extras and we'll put that value here. So since this is the ML format, it will have to be like this.

B

There'll be a new deployment and the value should the target and prometheus should remain so. This target should still be there after the pipeline finishes.

B

So those are the two methods that I know of to tell me the manage Prometheus to scrape your application.

B

A

So primitive support like counter gouge and histogram right. So, if I wanted to collect the metric in a different way, well should I do this change.

B

Meaning you want to try using one of the other data so.

A

Just just where should we change.

B

So, for example, if you, if you want to use a counter say so, we would have to change yeah so right now, response is a summary. So instead you would have.

B

Something like this.

B

This would be a counter, so there'll be no unit.

B

Then similarly, we'll have a help and if you have any labels that you want to define right at the start, you can define them here.

B

That should be it so, similarly for a gauge histogram.

B

And, of course, you'll have to change how you collect it so, for example, for this counter, that's rename this so for this counter, you would.

B

Add your labels.

B

Assuming you have no labels, you can just observe.

B

Okay, so this is our counter, so there's no observation, I think the method is called increment.

B

B

Yeah, so I am saying that stands for increment.

B

So this should increment the response counter every time, there's a request to any of the routes. Ideally, you would add a label of at least the URL so that you can differentiate, otherwise it will increment the same counter for any request. So, for example, if you have two API API a and API B, it will increment the same counter. Each time you get either API, while, if you add a label with the URL, it will create a different counter for each different possible label value, which is what we usually want so yeah.

B

This would be how you would add another data structure.

B

Anything any other question.

A

It was very helpful and I'm sure that other people that watch this video will found it helpful as well. So thank you for doing that.

B

Sure no problem.

D

Yeah, that's also all in my part, I made lots of good notes, so I think it will be a good starting ground for us to start looking into into it. Looking for ways to improve the process.

B

Yeah I think for now just a quick improvement that I can think of is to add some documentation which sales, for example, what kind of value you need to add into the auto deploy values file, because the current documentation only tells you that you need to add this file, but it does not give me both of what you can add into the side or even what value you should give for this variable, so I think that's a low-hanging fruit that we can quickly add so I'll, open and I'm are for that.

A

Weeks ago, I can.

B

A

You to get them all I.

B

Sure that'll be.

D

Awesome well thanks. Everyone I think it's a wrap thanks. So much really was super helpful.

B

D

B

Boring thanks. Everyone thanks. Everyone.