Kubernetes KubeBuilder Controller Runtime and Controller Tools, 9 Sep 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Demo Grafana Plugin New feature for custom metrics

Description

#kubebuilder #grafana #demo

Demonstration: new feature to generate graphs for custom metrics with the new Grafana/v1-alpha plugin by Tony (https://github.com/Kavinjsir) (Google Summer Code 2022) - More info: https://master.book.kubebuilder.io/

A

Hi everyone, I'm tony and I'm a mentee under the guidance from camilla russia and rashmi for the jsoc project this summer, so we've been working on creating a new grafana plugin that has helped to simplify visualization the status of the operator through the grafana web ui.

A

uh So if you're not very familiar with profana plugin, it is it's just a newly created one, as I just mentioned, and basically it will help to uh simplify the generation of the grafana magnify fast based on the default metrics generated by controller runtime. So that's the user can directly copy the content of the manifest to their grafana web ui to visualize to observe their operator status.

A

So I recommend, if you have interest, I recommend you to go through the documentation and and I'll be glad if you can uh have a try on that, and let me know if anything uh you have questions or issues or any idea. You want to bring to this plugin very I'll, be very glad so uh currently for the current release of the grafana plugin we have, uh it will bring you two dashboards that by default will give you the metrics based on uh the control runtime.

A

So one dashboard is the controller resource matrix that will show you the basic cpu and memory usage of your operator. So in the future we probably will bring new uh resources based on different perspective if it is necessary and another panel. Another dashboard is the runtime matrix. So this basically will show you the relative uh status of the reconstitution and the word queue.

A

So it will give you the performance of your rack, conjugation and the status of the work queue, uh so that's like, or what previously the ground up plugin can bring you so upon that uh russia, and we have been thinking. If ever we can try to visualize the custom metrics. So that's what I'm gonna bring for today so for custom. Metrics.

A

We're thinking that, like things operator, can be uh various kind because, like like different users or different companies, may have different kinds of operators, and it can be a common case that uh the operator author will create or compose some custom metrics based on different perspectives.

A

And so it can be interesting if we can help them to visualize uh their custom metrics through the graffana web ui.

A

uh So actually it feels like a challenging because I just mentioned like since it is customized so like uh what the meaning of the metrics and like what kind of the panels they want to visualize can be vary, and it is like a little bit hard like for uh the plugin to judge like what kind of the panel and what kind of the expressions uh the one you see so we're, starting like the very basic things for like experimental.

A

So, and uh so I can show like some latest uh some like uh specification in the latest documents uh of the master branch so uh again like the grafana plugin can be triggered by the coupe uh sub command init or like the subcommands added, and then this time it will also generate a config email file. uh That is an entry for the users to define the metrics and uh they want to view so inside the config demo.

A

It will give an object, called custom metrics that is basically an array and for each element it will help the user to define the metric, uh the type and the expr. So the metric is just the raw custom metric, the user defined. The type is the type of the custom metric. That is the prometheus data type, so you say it can be counter gouge and histogram, and the expr is the composed prompt ql based on the metric circuits like custom magic is like uh this. This is just the raw metric and more in general cases.

A

We try to build some like complicated, prompt based on that. So this is the field for exp, uh so for them. So for the district field, the metric and the type is required so that the plugin will know like uh what exactly the metric it is and the expr is optional, so that uh if the user would be glad to bring the xpr, then the grafana plugin will just directly use that.

A

If the user not sure like what the xpl would be look like, then the plugin will generate automatically create one based on the metric and the type.

A

So I'm gonna show a very briefcase how we can go over that.

A

So here is the uh sorry.

A

So here I will be using the memcache operator, so this is like a very basic operator and the operator sdk test data, uh but it's still like it can be driven by code builder, so uh I'm gonna use the computer edit sub command with the plugin.

A

So after that there will be a newly created, we found a directory under the project and there. So this is uh what this new feature bring is I just mentioned it give us a directory with a config email file.

A

So uh inside it uh I I was going to create some configs for the custom metric. So before that I have added two custom metrics based on the memcache operator. One is uh it has a histogram which is the memcache operator reconcile time seconds. One is the reconcile total. uh It is not necessary to be very uh correct, but just is a trial for like the custom metric.

A

I just copy and paste to the config demo.

A

Oh so for like for histogram it's usually we would use the buckets of the this metric and the next one is the reconcile total.

A

And again I will define it as the counter and I'm not necessary to use the expr. uh So let's get started so now. uh I would also then I will again trigger with the coupe added. So this time since I have defined the config gmo, the plugin will read that config and generate the json file.

A

So if we open the directory, we will find there's a newly created json file under the custom matrix and.

A

So search for the keyword expr, we will see that it will automatically generate a new prom qr expressions based on the given metric and expression. So this is the histogram. It will generate like a histogram quantum, to show like some statistical results based on the custom metrics, and so here it will do some summary rates based on the reconcile uh based on the counter metric, so basically like what the plugin do is.

A

It will try to compose a very basic uh promoqueer expression based on the type of the metric and will insert the custom metric into the uh into the expressions.

A

So now we can copy the contents of the the manifest and we go to. We can go to uh the grafana web ui. So let's try to create one.

A

So we just paste it in uh the input page and let's say we chose a folder playground and let's select the relative data source, which is prometheus and click import. So now we can see, there's the uh so there's the uh the panel. So if we choose so let me, let me choose different time range uh say like 10 hours, so we'll guess give this one. So I probably have a wrong expressions on the default metric, so let me try to fix that so uh so that's the reason is because I guess I didn't.

A

uh I didn't correctly spell the custom metrics as I defined here. It should be buckets. I guess so. Let me see if I can fix that.

A

Oh, it seems it's correct because there's the histogram I'm not sure why it's not showing outside, but it looks uh we can view what's happening over here. uh So that's! Basically what currently uh the custom, how like the profound plugin, will bring you uh the panels for uh the user-defined custom metrics.

A

So it's very like it is still like an experimental feature and in a very early stage because, like you can see that uh which is trying to like build upon uh the very basic problem, q expressions to help define the panel uh based on a waste of custom matrix- and we just deal with uh the the timeline drawing panel uh as uh as what is displayed over here and so be very glad. If, like you, would be, uh have interest on trying this and I'd be very glad.

A

If anyone would have some ideas, uh how it could be better. uh So it can be like challenging because, like at one point, we don't want to leave so many config field for the user, because it be brings some burden of learning curve for the operator author, but still want to make it functional because, like since it is custom matrix, it seems like uh there's like a couple of things: that's really defined by the user side, not the plugin itself, so yeah, that's what we have for the latest feature of the grafana plugin. Thank you.