Cloud Native Computing Foundation PromCon North America 2021, 1 Nov 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Practical Kubernetes Monitoring with Prometheus - Michael Friedrich, GitLab

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Practical Kubernetes Monitoring with Prometheus - Michael Friedrich, GitLab

Monitoring for microservices and distributed workloads isn’t an easy task. Imagine that you become responsible after years of traditional service monitoring. You start your research - Kubernetes as a container orchestrator already is a complex ecosystem to understand. Join this talk for a journey on the first steps, best practices with Prometheus, Grafana and the Prometheus Operator. The adventure does not stop here: Use client libraries to instrument your own application with metrics, deploy it to Kubernetes and what now? Explore how service discovery, long term storage with Thanos/Cortex and alerting help complete cloud native monitoring. It’s also an iterative process, with new workloads, changes, and a constant work in progress. This talk will help prepare you for cold winters and hot summers, silencing alerts before they exist. Spoiler: The end will peek into observability with logs, traces and SLOs, and show monitoring use cases on GitLab.com SaaS.

A

Hello, everyone good morning, good afternoon, good evening, wherever you are I'm michael waving over from germany um this time virtually um and today I want to dive into practical, kubernetes monitoring with prometheus with you in the next few minutes. um Before we continue before we start um you can reach out to me as dns michi on on social media. This is dns m-I-c-h-I and without further ado, let's jump into action when it comes to monitoring. We probably need to first start understand.

A

While we need to change from the traditional host and service objects, monitoring, we had state-based monitoring with narios back then, with more like metric names, tags, volatile container names coming up and other similarities and needed to understand. Well, microservices and distributed systems need a new approach in monitoring, and this is where we want to dive into today. Like say: okay, we have kubernetes, we have containers microservices and how we can we monitor that changing the ideas changing the knowledge is oftentimes hard, but here are a few tips of saying.

A

Well, we have microservices in containers which can run run anywhere in the cluster, so you actually don't know the ip address. You cannot really say well, it has a host name, which I can ping and say it's.

A

Okay, um the other thing is a business process model doesn't work with saying: hey two out of three replica sets means that the state is still okay, because this is different, similar to how kubernetes detects the health state and might be repairing the health state or even add more resources to the pools in order to ensure that the application is running because sometimes the service might remain broken from the outside and in the next second, it it's already fixed again and in a traditional monitoring system.

A

This could be meant flapping between states and generate alerts, which is not something which is applicable to kubernetes and microservices similar to mapping. States like like okay, warning critical, they don't necessarily fit kubernetes objects and inside the cluster. So forget, forget everything you know about this when approaching kubernetes monitoring and focus on metrics, logs and and everything else, which is highly dynamic.

A

Now, where should we be starting our monitoring journey? Should we be running promises inside the cluster or outside the cluster? What what is needed, because, well, if the kubernetes cluster goes down and kubernetes is running inside, we don't have any monitoring.

A

The thing is when we need to do external monitoring, we might be to need, we might need to open up the apis.

A

If we don't have any side cars and there is a potential for security problems, because we also expose certain route access and other things which are not really made by design the design is to say hey, I want to run prometheus in the kubernetes cluster use security policies which are in there communicate with specific api servers and other things and follow the best practices. With regards to permissions.

A

If I need external monitoring, I can install ping probes to monitor the monitor, which I need to do with a traditional monitoring system already now, if we dive into promisius and say hey, what is that? What you probably know it already? But how can we approach this or combine this with kubernetes monitoring and getting the big bigger picture is like prometheus, crepes um endpoints so have the http endpoints and collects the metrics from there.

A

So at a certain point, we need to define the promises targets and one thing uh which comes into play with kubernetes is to have service discovery which promises um discovers the targets in promisius uh in in kubernetes.

A

Sorry, um using the file service discovery, for example, and then automatically adds the script targets to the monitoring post them in a in in an interval, and everything is fine with all the collected data, we of course need to define alert rules for um sending notifications and alerts and also query the data with promql from the promises web ui from grafana from specific other api clients in that regard.

A

Now, when it comes to querying data, keep in mind prom ql keep practicing that this, because this really comes in handy with um learning kubernetes monitoring, um specifically knowing the basics and also using, for example, prom lens as a as a tool to test the quiz now. The other thing I mentioned is service discovery, and this is like really using a dynamic list of targets which prometheus can then add to the monitoring.

A

This is something to keep in mind um to understand how this works with kubernetes and prometheus, but let's get into action and say: well, we want to monitor kubernetes and what is needed to actually do that, and this is where the promises operator and specifically cube promises comes into play which allows to install um an entire package or the components in a in a package which you would otherwise install manually by using the operator by using help, charts and so on so cooper mises provides the promises or uses the premises operator to install promises in a higher availability setup next to the alert manager next to node exporters.

A

It also adds a promises adapter for the kubernetes metrics api, so you don't need the metric server anymore. You also get cube state metrics pre-installed. We will see in a bit what is what it does and grafana to have uh dashboards and um fuse available in.

A

In addition to that- and this is the icing on the cake- we get pre-configured cluster monitoring for for kubernetes, meaning to say we have um grafana dashboards and also alerting mix-ins pre-installed and can actually make use of that and have the like the five-minute success of I'm deploying the promises operator or cube promisius, and I can start playing. I can start investigating, what's going on in my kubernetes cluster.

A

In addition to that, um the uh q prometheus allows us to dynamically, create service discovery objects which is helpful for deploying own applications and adding dynamically the application endpoints or target endpoints for application metrics. For example.

A

Now the installation is pretty straightforward, you just follow the documentation, apply um the manifest and wait a bit and then, basically, everything is fine um in order to familiarize yourself with the deployment. The first step is to add port forwarding um to have the ports available um for production, environment, use an ingress controller like nginx or something else in order to expose the http front-ends.

A

One of the frontends is the promises ui itself, which you can use for basic queries, visualizing the charts and so on, so basically testing and inspecting what promises is doing for more advanced uh user interfaces and dashboards and panels, and so on. um You get grafana pre-installed, which, which comes with the default password and so on, and last but not least, we also get the alert manager ui for that very reason and can start or when, when we trigger an alert, we can investigate the alert manager ui.

A

Now when it comes to specifically monitoring kubernetes with prometheus, we want to have something available and the great thing is: q. Prometheus provides a long list of grafana dashboards, which we can inspect and and and see what is going on, for example, the kubernetes api server, um the compute resources in the cluster. We can access the cubelet.

A

We can get the promises overview so seeing the health of the monitor actually and many many more things which come by default, which come out of the box like this is actually like a production view already on the kubernetes cluster itself, so oftentimes, it's wise to just use that as a production view and then learn from the queries and from the panels to create your own dashboards and panels in grafana to monitor the kubernetes cluster even even further.

A

Now. The other thing which comes to mind is well. If I deploy my own application into the cluster, I also want to monitor it and this example is um a fork from the prometheus demo service from julius waltz, which basically exposes some synthetic metrics and in order to deploy it by uh using the deployment and service resources, we can actually like run it into the kubernetes cluster and later on, start monitoring it.

A

So I would totally encourage you to try it out later on, deploy it into the kubernetes cluster and then monitor the demo service, which basically means we need to create a new custom resource definition, called service monitor and make promises aware that there's something new, a new service discovered to in in that regard and ensure that it takes up the metrics, endpoints metrics endpoint and starts monitoring it.

A

Since there were three replica sets defined, um we can see two of two of them already up as a scrape target, and one of them is still unknown, um which means that at this at a later point we have three targets available and based on that, you can go ahead and create a new grafana dashboard, a panel selected, a promisius data source search for a specific cpu metric, for example, and then use a prom curl query in order to fetch the data and visualize the graph.

A

The other thing which you get out of the box with q promises is container metrics, so the cubelets provide an embedded c advisor exporter which comes at the endpoint, slash metrics c advisor and allows you to monitor the resource usage. Specific out of memory kills, and so on an example is it looks like this. um So building on that and using that gives you an insight of saying hey, there is a spike. There is something which is continuously going wrong in order to really get an insight into, for example, specific namespaces using too many resources.

A

The other thing which we also get out of the box with q permissions is cube, state matrix, which queries the kubernetes api server for specific health states like the deployments, the nodes, the ports, we can query the replicas available and their status and everything is installed out of the box, which is super helpful to get an insight without any further manual actions.

A

Now this is what we get as the basics, so we didn't do anything yet or we didn't modify anything yet. This is just out of the box monitoring or out of the box. It's you install the prometheus operator and q promises and you have everything available, but what, if we want to customize it, so q promises uses jsonnet and you can develop your own rules and dashboards. So you can add them.

A

You can also monitor other namespaces, and I tried this recently to monitor the ecc. The other one can contribute namespace in our kubernetes cluster, and this is how it looks like to create your own jsonnet file, modify it and then generate the yaml configuration and deploy that into your kubernetes cluster again- and this is like an ongoing process, you you can modify it, you can add even more things to it and use cube premises really really as a basis and add your customizations on top. So you don't need to reinvent the wheel all the time.

A

The other thing around more advanced monitoring or what I want to bring to your attention is: what else can we do with the monitoring data with the metrics collected and one of the things in the sre handbook, from or in the sre book from google? Is the foregone signals of monitoring like what is needed to immediately see what it that something is going wrong with your application, and this could be latency? This means traffic. um This is like errors and also saturation of the service and oftentimes.

A

You need to think about code instrumentation, so actually modifying your code and exposing the metrics for promisius to see the http requests to see anything which has been failing and so on, and you need to correlate correlate that in order to immediately see that something is really going off now, if you, for example, want to dive into instrumenting your own applications.

A

This is an example for python, which I created a while ago for a workshop um on the other side, I would encourage you to check out all the promises, client libraries for ruby for rust, for go for anything which comes to mind and if it's not there, look into the existing examples how to really instrument your application and deploy it and have it monitored.

A

Now, monitoring and metrics is fine, but what about alerts and escalations? As mentioned before, the promises alert manager comes pre-installed with cube, comm cube promisius.

A

We can use the promises, alert rules like prom, kill, queries and applying a certain threshold and times and so on, and the alert manager ensures that there are no duplicated alerts that you uh prevent um alerts which have been fired already. You can acknowledge and silence specific alerts for a given time. You have high availability available.

A

The transports are email or chat systems. Everything which, which is basically known, already, is similar um here already. The main difference with defining the alert rules is, we need a promises, rule custom, resource definition which wraps the promises, rules format and ensures that the alerts, for example, are being deployed into the kubernetes cluster into the promises configuration.

A

But this seems rather strength, straightforward and the link documentation for the promises operator is pretty awesome.

A

Now, jumping into a related topic to alerts with sla, slo and sli, we define a specific service level agreement, which means potentially with customers, 99.5 percent availability. Our objective is even higher, saying, hey. I want it to be a 99.9 percent well available and for achieving that we need certain indicators. So what is hurting our um our production system? What is the error budget? What indicates that is it errors? Is it latency, saturation and so on?

A

So when we start thinking about this, we kind of need to also think about slos and build our alerting based on slos. For example. Now slos can be difficult or can be. Like you write a lot of code, you need to define the prom cal queries um this and the alerting rules for the alert manager and so on.

A

So it needs a common specification and, most recently, openslo was formed to unify that or to define the specification for slos so that slo generators like sloth, for example, always generate the requested format or the defined format, and you can also use slo management. Pura was announced most recently, which allows you to grab using a ui to manage slos for promises, for example.

A

So you don't need to go into manually copy pasting stuff in order to manage slos for prometheus and and also for kubernetes, then now um slos with promises can also be used for for even more things like not only for metrics and alert calculation, but also for quality gates.

A

When something is deployed into staging- and you say hey if the slo is failing in my staging environment, I want to have a quality gate, for example, with captain and ci cd is red and it fails just because the slo is has failed, and I don't want to deploy any performance regressions, for example, into production and then debug it at 3 am in the morning and burn out from that. So this is one thing to keep in keep in mind. The other thing is I I can can really touch it.

A

um I would want to like talk talk about it for hours for gitlab.com the sas platform. Our sres and infrastructure teams have also defined um their own metrics catalog, which is something similar to jsonnet, and this also gets generated and the great thing about generating all these things is. We generate the alerts, the dashboards, which allows us to define the key metrics the thresholds and also add description and observability tools.

A

So when someone is on call and dslo alert is being fired, they can immediately see what is going on and what is needed to fix the problem, hopefully so that um a production incident can get resolved rather quickly.

A

Now, last but not least, we need to shift from just monitoring to observability just for the reason we have metrics with promisius the exporters, the side cars we have the applications with metrics, instrumentation and also integration outside of kubernetes, but this is probably not enough for running operations in the sre, and so in that regard we shift into well, let's keep an eye out for logs.

A

Maybe we have a central, lock management available, probably have either elasticsearch with beats. Sidecars and demon sets with auto discovery and keep sorry kibana as a front end or um using a more lightweight approach with loki and and promptil and fluency, and then using grafana as a front-end. This is something you need to keep an eye out for um and keep and make your mind about it next to metrics inside your kubernetes cluster.

A

The other thing I want to bring to your attention is traces which work a little different to logs and allow you to really get distributed. Insights into application performance as well um and, for example, investigates low queries. The tools in that regard are jaeger, tracing grafana, tempo and and open telemetry.

A

This is something to check out and also pro profiling, continuous profiling, providing application performance, insights for developers and defining the four pillars of observability in the future.

A

Last but not least, secure monitoring um is a hot topic. It's needed, so you pro potentially have thought around automation and for infrastructures, code and githubs like using the promises operator and cube promisius, extending it customizing it with jsonnet, using um infrastructure as code for inventory, for service discovery with terraform and several and and the like and deployed continuously practice git ups with monitoring in your kubernetes cluster.

A

The other thing is like security monitoring and security means you want to detect vulnerabilities before they hurt production. This not only covers like code vulnerabilities but also container images, security scanning with dependencies and so on and kubernetes security in a way of cluster image scanning, but also ensuring that policies are being applied like the open policy, agent, kiverner policies and speaking of kirvana policies. I want to bring the promises exporter and the grafana dashboards to your attention, which allows you to immediately see what policy has been triggered when it failed.

A

When someone tried to circumvent it and so on, which is super helpful for production environments now, last but not least getting through ideas and some thoughts, maybe think about writing the data from promises outside to kubernetes like using re remote rights with ops trace and the grafana agent. As an example, look into the gitlab sov runbooks, um how it's being done check out opentelemetry to unify metrics logs and traces and continue looking into continuous profiling.

A

Now the resources which I've, which helped me over the past years to change from traditional monitoring to cloud native monitoring with kubernetes with promisius, are all linked here: um special thanks to julius for promlabs for the trainings. They are totally awesome.

A

I would encourage you to use the exercises in the slide deck, to learn even more async and to think about what to prepare, what to instrument, what to observe and how to iterate on the specific things so like throw in chaos, engineering think about quality gates and optimize everything which is there. So you you sleep well at night, and everything is hopefully in good shape and your monitoring is not red.

A

Thanks for attention. I'm happy to answer any questions and look forward to seeing you online and hopefully soon in person. Again, you.