GitLab Developer Evangelism Team, 19 Sep 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes Observability Workshop

Description

In this workshop, you will learn the basics of Monitoring and Observability with microservices and container clusters. The hands-on session includes Prometheus and Grafana, kube-prometheus, alerts/incidents, and customizations/scaling. We will discuss best practices with logs, traces, and how innovation with OpenTelementry and eBPF can help with advanced Observability workflows.

Speaker - Michael Friedrich (@dnsmichi)

►►► Pre-requisites ►►►
1. Installed & configured a minikube cluster (https://minikube.sigs.k8s.io/docs/start/)
2. Make sure to go through the previous workshops:
- Linux & Docker Fundamentals: https://youtu.be/EUu1E_YKGTw
- Kubernetes 101: https://youtu.be/PN3VqbZqmD8

►►►Resources ►►►
► Workshop Slide Deck - https://docs.google.com/presentation/d/1uuYIPwQjckNiPPJQcN8tauZ8KSIdHASbyoAxJ0fc6uQ/edit?usp=sharing

Chapters
00:00 Intro music
01:38 Introduction with Saiyam and Michael
08:10 Workshop Start
08:58 What to expect
10:05 Workshop requirements
12:22 Tips
15:06 Monitoring, quo vadis - in a nutshell, black-box, metrics/trends, microservices, whitebox
23:00 Kubernetes - learn what to monitor
29:45 Metrics with Prometheus - Architecture, PromQL, UI,
39:15 Prometheus Operator - install Prometheus in Kubernetes
52:22 Kubernetes Metrics
54:45 Prometheus Metrics in Grafana - Dashboards inspection, deployed by Prometheus Operator
01:02:56 Workshop dashboards in Grafana - first panel and Kubernetes dashboard, container metrics, kube-state-metrics,
01:21:34 ServiceMonitor CRD for /metrics endpoints auto discovery with Prometheus
01:37:56 Monitoring 2.0 - Prometheus client libraries & instrumentation,
01:45:43 Alerts and SLOs - Prometheus Operator CRDs, Alert Manager, podtato-head
01:55:53 Trigger alerts for podtato-head deployment - blackbox probe, dashboards, alert rules,
02:14:45 Service Level Objectives & Ops Confidence - Golden Signals, SLOs as code,
02:19:15 Customize kube-prometheus - custom dashboards, reduce visible data, see what is important
02:22:50 Beyond Metrics - Logs, Tracing, OpenTelemetry
02:31:28 OpenTelemetry demo deployment - shop, website, Jaeger tracing,
02:39:05 Discussion: Why traces? What's next
02:45:35 Performance and Scaling - data retention, long-term storage, distributed scaling, GitLab.com SaaS production insights,
02:53:50 Observability and Chaos Engineering
02:56:06 Chaos Mesh
02:58:10 DNS Chaos - demo exercise from KubeCon EU, leak app memory, verify alerts and SLOs
03:12:45 Take action from chaos experiments
03:15:00 Future Observability - eBPF, auto-instrumentation with Cilium Tetragon, etc.
03:17:30 SLOs and quality gates with Keptn
03:17:48 Security - Policies with Kyverno, Hardening (book recommendation: Hacking Kubernetes)
03:18:54 Your adventure -
03:19:20 Q&A - https://o11y.love and outro with Saiyam and Michael

►►►Connect with Kubesimplify ►►►
► Twitter - https://twitter.com/kubesimplify
► Discord - https://kubesimplify.com/discord
► Website - https://kubesimplify.com/
► GitHub - https://github.com/kubesimplify

#kubesimplify #kubernetes #observability

A

A

A

A

Okay, so hello, everyone and welcome to another amazing Cube, simplifier, Workshop I hope you all are doing good, uh please introduce yourself and the chats where you're joining in from uh and you're excited to learn. uh So my name is Sam paduk and I'm, a cncf Ambassador working as director of technical evangelism at sibo, uh and this this set of workshops. This particular Workshop, is part of the series that we have been doing called Cube, simplify workshops. uh So we have started from Linux and Docker from scratch.

A

We went to kubernetes, we went to a container and Community security, chaos, engineering, service, mesh um and even kubernetes troubleshooting, and now today, we'll be covering a different domain, which is uh kubernetes observability, uh which is again super important uh for for any application. Any infrastructure, even kubernetes, so monitoring observability, is something which, which is very critical. When you are learning something- uh and you should be knowing about that and we we got the best for you.

A

um So today, uh I'm joined by Michael Frederick uh from gitlab, uh who will be telling us about kubernetes observility. He has been preparing this Workshop uh tweeting about that all the preparations that have been going on and we have loved those tweets as well so make sure to share this particular stream in your network right away. Do it just right now so that you know maximum people can join in and they can take full benefit from it again.

A

The prerequisites are very simple: you need to go through the previous workshops uh or just you should have at least the understanding of kubernetes. So you should be knowing you know what a pod is deployment is what the keyword is. Architecture is and stuff like that, so you should be knowing about uh communities and you should be able to follow what Michael has in store for you today um and please be respectful uh to each other in the chat. That is very important. We are all here to learn.

A

uh So please make sure you are just asking the right questions. I mean you can ask questions related to kubernetes monitoring, observability, uh but just don't be disrespectful. There are people in the chat who are Cube, simplify ambassadors, who are monitoring the chat also there to help you out. uh Now, if you have a larger question like if you have something not working- and you want to post pictures and stuff like that, you won't be able to do that on YouTube chat.

A

So it's better if you can join the scrolling link, which is their cubesimplify.com Discord and there's a channel called workshops. uh So there also Cube, simple ambassadors.

A

Are there to help you out during the workshops and even after the workshop, so you should be able to like ask anything if you are stuck somewhere um and you should be able to follow along uh another prerequisite that was sent in the email who registered uh for this Workshop was given by Michael like you should be having a kubernetes cluster um so that you can you can you can work with it now you can use killer quota, but we won't be specifically telling you know how to do things with killer Coda.

A

So if you have attended the past workshops, you already know how to use it like exposing stuff. You know the node Port Services, how to access that and all that stuff. So you will be have to taking care of all those things, uh but all in all, uh Michael has done his level best to explain it from you know, step by step level by level, um and you will have a lot of fun learning this particular Workshop.

A

Just one request is stay till the end of the workshop, so that you can say that you have learned a good amount of kubernetes observability and you are ready to apply that knowledge in your current systems, because that is our goal. You should be able to learn something out uh from today's workshop and apply that knowledge in your systems, in your current production system or as a POC or as a personal project anywhere. So uh so, thank you so much for joining in uh Michael.

A

Can you please introduce yourself to the community and meanwhile I'll bring all of the hellos on the screen.

B

I will definitely do that with pressure, so my name is Michael I go as DNS Michi, which is DNS mici on on the Internet. It's like the lovely version of my key, but I figured it doesn't work in English that good um I've been with gitlab for two and a half years now, um as a senior developer evangelist and in my previous life I was an open source monitoring maintainer, so I I got to know quite a lot around.

B

What is monitoring the shift to obserability um in many many other things I'm currently living in Germany uh in the Nuremberg area. My origin is Austria, so um I moved there like 10 years ago for my passion for pencils and I'm super motivated to like share everything in for free, so that everyone can contribute. This is also a model of um not not only a model. It's like it's lived at gitlab as well, and I'm super happy to be joining you today with like a free Workshop. So it's not gated.

B

It's not like paid subscription thing, but I really want to make you provide you the best um way to get started. It's for free. You can re-watch everything later on if I'm talking too fast or you just tell tell the folks in the chat that I'm talking too fast and they can slow me down um yeah I'm looking forward to get started. Is there anything we need to like chat about before we Dive In.

A

uh No, uh so just we are getting started so make sure to subscribe. I always forget this particular portion, which is very important so make sure to subscribe to the channel uh like this video and press the Bell icon, so that you never miss a notification and always follow Cube, simplify for the awesome updates. uh You know all the awesome workshops that that we are conducting so that you can get all the benefit out there. Your learning resources should not stop you from getting jobs.

A

Learning new things and stuff like that and like Michael, said uh Cube simplify, aims to simplify Cloud native and we uh always love bringing free content uh to you all which is not gated at all. So just hang in there. It's all Michael's show now and he'll be teaching us kubernetes observability. Let's learn now.

B

Perfect, um so the slides um are now publicly available and I would encourage your ask Siam to share them in the YouTube chat, so you can like follow along um this slide. Deck has quite a few exercises and slides as we move on so don't feel stressed about tubu black, really understand everything or click on all the links which are required. It's just really. It should be a learning resource for your for you all, um and there is also like a repository, a project being linked.

B

We will dive into this in a bit um yeah and yeah. You can also follow me on Twitter and Linkedin and gitlab and polybourg and everywhere else. um This is just me but like to move on what to like expect in the next one or two hours, maybe a little more than that, um we will be doing a short introduction into like what is needed for this Workshop um then diving into monitoring where it is where it was um how it how we can like build something with Prometheus, grafana and kubernetes.

B

um Further down the road. There is always like how to deal with alerts, incidents, service level objectives and then going to the award question like what is observability and how does like tracing logs and even more add to that, um and since there is so much to learn, I added, even more ideas to the to the agenda for today, which is like scaling, observability scaling monitoring. What is eppf like a short introduction, some some a little bit about security as well, and then a lot of ideas, but don't feel overwhelmed.

B

um It's just like me, providing you or trying to provide you the best experience and the best learning resources you can. You can get out there, um the requirements for the workshops? Well, sorry, I'm already shared it, you should have. You should have a kubernetes cluster running. Um This can be like mini Cube on a locally on localhost. This can be some in in a cloud and it's helpful to have some sort of experience. um This is not required.

B

It's just like me sharing an idea of having, like you, you're, coming, maybe from the opposite admin side, and you know how to run uh and maintain services and applications. On the other side, it can be helpful to know you're a developer. You know how to write code, you're learning it, and you want to learn about like application limits or resource limits for your application, because, like resources are not endless- um and this is from a security perspective, it can also help to know like what is a potential vulnerability about exploits and other things.

B

So this everything plays into observability and and also kubernetes. We will be working with a project I prepared with the exercises and solutions.

B

um So if you like, if you're feeling stuck around the way- um or there is like it's, um if I'm like copy pasting too fast in in the terminal, it's always time to look into this project.

B

um There are the yaml files and the uh the solutions basically prepared, which we will be using, and there are also some other um protects which are which will be deployed. Everything is open source, so you can like inspect the source code and and uh the deployment and learn from that by yourself or maybe even like use the demos for your own blog post. You will be writing in the future or deployments and so on.

B

um Yeah. The kubernetes cluster is again described here so for everyone who might be watching this uh Workshop later on as recording. um There are many resources out there to get. You started with a fresh kubernetes cluster um I will be using one in civil Cloud, I told Samia. That's the easiest way for me to do it after trying uh killer code and figure bring out uh I really want to like use a production ready, MPT kubernetes cluster for this Workshop.

B

um When you start with the workshop, when we dive into the exercises now, um You can actively do the exercises as I go along um I I'm trying to wait a bit after the exercises to really verify that everyone is ready, but unfortunately, I cannot see your faces like in a traditional in-person Workshop. So if, if there is like the need to come back later on and re-practice the exercise or redo something, you can do that at your own pace.

B

So you can watch me, do the exercises or try to replicate everything as we go along. um Those slides which require action have like the exercise, um colon prefix or discussion, colon prefix, so that you know when action is actually needed.

B

I've also marked some exercises at Advanced because they require more knowledge and they might be moving too fast, so in in a specific sense, um just watch listen, learn iterate later on and try it yourself when you when you when we move along in the workshop, it can also be helpful, like taking notes in an online IDE in your in your local IDE um I, found it easy.

B

The easiest to edit just edit I read my MD with markdown um while doing the exercises, and you will also find many links in the slides, which you can also then um study later on as an async exercise regarding kubernetes monitoring observability. This is a huge topic, so kubernetes on its own is a huge topic to learn and um I. For myself felt very overwhelmed when, when I started and my advices don't feel that way, try to find like a smaller ability. Buildings like Brick by Brick building Lego, like in my background, do small iterations.

B

Also technology is moving so fast, really like focus on the basics. What is running in my cluster? How can I monitor it? um We might be gluing together in this session. I don't know yet, um but I'm. Also, not an expert, and sometimes my top history um makes the memory of my browser. Explode and I would also encourage you to ask questions and share thoughts in the chat um we will be trying to answer everything live here, I think at some point, um but also, like you know my social handles.

B

Please ask please tag me in public for your questions. Maybe we even meet at kubecon or community events in the future, uh to to do some Exchange in person as well would be lovely now. um This is more theoretical entry in the workshop, looking a little bit around like what is monitoring. uh Where is the current path towards monitoring like monitoring? In a nutshell- and probably it's it's not exactly a nutshell, but there are different types of monitoring, so you can monitor it.

B

um The heart rate, you can monitor different things um for um for it infrastructure we do have monitoring like server monitoring, Network, Hardware monitoring. um There are certain ways of application performance monitoring.

B

We got Cloud resources over the past 10 years, something like that. There are apis. There are serverless um workloads or functions, which can also be a way to monitor things, and we also got to use distributed microservices and like maintaining or orchestrating. Containers became a cluster thing, and this became, for example, kubernetes, but also other services which which need to be taken into account when thinking about how can I monitor this for what? What is the actual health status of this entire thing of a specific service? And so on? Now?

B

One of the things to keep in mind is um when, when talking about monitoring, you want it to be reachable over the network, so something needs to be pinged or cured um using a different protocol, be it TCP, HTTP SNMP, or something like that. So these are like um terms and and things you probably encounter in the wild when, when you look at monitoring now um in the beginning- and this is like turning back time 20 years ago like when arios was- was a thing- um we had so-called Black Box monitoring.

B

So it's like Service attached to a host. um There is a specific relation. You want to monitor a website and a specific external uh ping to see whether it works or not um and yeah.

B

The idea is really you want to catch it, for the customer calls you that they cannot shop like they're, not they cannot buy anything in your shop because everything is broken, um and um this could could be, for example, a competitive disadvantage because your shop doesn't work, but the competitors shop works and everyone buys there and says well not coming back to you.

B

um So the thing is monitoring things like you: have an executable check. Script deploy the services you do that in a specific interval and with like tools like nagias, we had the thing like calculating a state, so you had either okay or not. Okay, some text output telling you what is going on, um but how to like, interpret things like this. So what is like the value of telling me well, the response time of the website is five seconds. What what exactly is like? Is this a problem? Could this be like a trend?

B

So you see something go up um and that's also Black Box monitoring, so you don't really can look into the application itself. So what happens in start? Why is it slow um and so on?

B

um These are all things to keep in mind now, if we think about what we want to look inside the application, so maybe the application has logs a log could be a line which starts with a timestamp and then some context. What is going on select the application is starting. The request is starting and so on. This typically grows a lot. So at a certain point you might be thinking about Central, Lock management and then challenges begin. Like data retention search performance, you need a query language.

B

You need to think about many, many other things um and during an incident it would be awesome if the monitoring system with the state is directly linked somehow to the log. So you can see that the outage of the webshop is actually tied to some out of memory kills, which are only in the locks. Hopefully, there are locks, and the other thing which is like challenging is when you have added logs to the application or your develop. Open teams have have added them.

B

They also need to be passed by machines, to store them, and maybe structured logging is a better alternative, but still requires teams to really adopt and learn what is structured logging in the Json format, for example, when we move monitoring to like hey, we want to see something, but it's a trend of what is like a metric. When we have been polling services in Argus, it was called performance data which is like, then, a metric value which can provide insights when storing it over time.

B

So you get to see that the website responses one second and it's five seconds, and it goes back to two seconds or something like that, and this can be visualized which, with it within a graph, and it would be awesome to really calculate a trend for that in in the past years, there were like tools like rid tool, graphite influxdb, which which store these time series. So it's like a tsdb, a Time series database, but oftentimes. It's really like when the uh when the monitoring engine that collect something was blocked.

B

um You had gaps in the metric metric graphs and then also a scaling problem with like is it would I be needing the metrics or the values for five in five minutes um granularity or do I want to have real-time monitoring, which is like 10 seconds or one second or one nanosecond, or something like that. So there were certain challenges with metrics and Trends, um also with like classically monitoring, and the other thing which came around was um since kubernetes and and container clusters were a thing like.

B

The classically uh host service relationship doesn't really work anymore. um So there isn't like a state of okay or warning or critical for a container cluster for service running in there with metrics. uh We we have the advantage of like just having values, names and tags to being able to see something in a group um and the other thing which I've seen in the widest.

B

um If you have like a static configuration for host and service names, it doesn't really work with a volatile container names. You start a port, you potentially it fails. Then it restarts itself um and with like microservices and distributed systems, you really need a new approach for that um and last thing around, like monitoring shifting towards the future or making a change was from the state state, Black, Box, monitoring and also service level availability reporting like the state changes, overtimes and also the metric data points.

B

um We we're moving towards so-called white box monitoring or have been moving in the past, which potentially was the first Inception with Prometheus, adding um a flash metrics um HTTP endpoint into Docker as a container engine in 2016. So it's like six years ago already. This was one of the first visible things where you could look inside the application and developers created their own metrics available for the uh for users to really being useful um in deployments.

B

Now, if we think of kubernetes and container clusters, and how does this fit into everything like kubernetes up and running requires quite some black I would say knowledge. There are so many things to understand um and also like to understand when something is broken, how to like get to it. So I've, just added a few um thoughts and things like the architecture is important. The components might be important. You have nodes Parts containers, you have deployment services and many other things, and then it comes to apis.

B

Ports data sources and I know that um uh that the previous Workshop around kubernetes, Trump, tooling, was already awesome or is awesome um which, which is a great way to really troubleshoot things. But it's also like the same for monitoring, so someone says: well, we need monitoring yeah. What kind of monitoring for kubernetes, maybe availability, monitoring or is it about performance and resources?

B

um Maybe we want to identify slower blocking deployments somehow. So what is like the important thing we really want to know. um As said before, like how service state based with microservice doesn't work, so we want metrics, we want logs. Do we want something else? Is it necessary to understand all the components? Hopefully not? What are the best practices? So it's really like a lot to understand and we need to focus on certain things so like a microservice in the container can run anywhere in the cluster.

B

It's no like you, don't have a hostname or IP address, which you can immediately access at at a certain point.

B

If we're defining replica sets, you can Define the service being run on marketable pods, so there is no guarantee whether it says well, two out of three are working. The service is okay or not. Okay, so called classical business process models. Don't really work with that pattern.

B

um Kubernetes also detects health and repair stage. So if you kill a port, for example, it comes up again um and if there are certain like things broken in in the kubernetes cluster, um this might be considered flapping. So something goes up down up down in traditional monitoring systems. Declare that there's a flipping service and send you notifications for that, which is also noted, uh not the behavior.

B

We want to replicate with them and the other thing is like States um are different in in kubernetes, so something like okay, warning critically doesn't really apply, um and the other thing to note is, or to mention metrics and logs are like highly Dynamic. So there is a lot going on and you really need to be capable of seeing what is important to you um and manage all the data now.

B

The the first thing you can do and should be doing is like open the monitoring, logging and debugging docs for the application. So this is not really tied to kubernetes on its own, um but any application. You want to monitor so like what could be the entry point to see when something is broken so or how can I inspect like when a node is going down? What is uh how can I get an idea how to monitor, for example, Services which are being deployed?

B

um What is the description in the troubleshooting documentation, for example, when you go over resources, and then you get an idea about hey. This is something I can apply to my own monitoring, metrics and thresholds and other things so this um this goes even Beyond like we can correct a cluster event um and anything anything beyond that. So um I couldn't really like figure out my own use case for kubernetes many years ago, and so I was like hey.

B

Maybe I should just deploy an application and try to monitor it, and then I can learn more about kubernetes on its own. um So for just like looking into the section, um we can get quite an idea around um debugging ports, debugging services, for example. So what what are like the commands which are needed? This is helpful to really like dive into action when then, actually an incident or queues, but also like what is described um within this section. Regarding um things, we really want to monitor.

B

For example, the note health or something like that now in order to like play around with that and get an idea, is um to say, want to change quickly into the terminal and promise we will be using the terminal in this Workshop, mainly for installing things and just to get an idea. What is going on in the beginning so like I, don't have anything in this cluster.

B

Yet it was just previously deployed like two hours ago and for example, we can I, do have the K Alias also installed, but I really want to show you the the full version. I do have three notes deployed, and um but today we also want to see the the pots will be which are being deployed, for example.

B

Now this provides some some idea and in order to inspect the logs manually, which which can be helpful with troubleshooting and and getting to know things um we can, for example, also say cubicle logs now, I need to cheat a little um deployment.

B

For DNS um employment- and this is in the tube system- maybe space, so it has been running for so for a while and has some warnings being deployed. Okay, this is quite a lot, but yeah this. This can be like an idea to get to know what is actually currently like being deployed and going on in my um kubernetes cluster, since this is like a fresh q, a this class 0 and all deployments of my own. Nothing has been nothing in there, um but we really. We want to like we're getting to it.

B

um First things. First I've shared some ideas around like metrics collecting values over time and being able to calculate um or to aggregate them calculate certain things get on with Trends and one of the things is like Prometheus, so we have in kubernetes many different data sources. We might be using service discovery, which is, which is a great idea and within the cloud native computing, Foundation ecosystem or Community.

B

There's a great project called Prometheus, um which is we will dive into the architecture in a little bit, but the main idea is: it collects metrics from a slash, metrics, HTTP endpoint, provided by applications provided by kubernetes components and is able to store that in a Time series database on its own. It doesn't need by default any anything with external.

B

um We can look at Trends dashboards. We can Define alerts and service level objectives. We will be explaining or looking into that in a little bit um and have that available for incidents as well. Now um yeah. The idea is really like collecting everything, um but also provides client libraries which allow it to instrument your own code and provide the slash Matrix and point um in your application, so that you can monitor your own applications deployed in kubernetes with Prometheus. We will be looking into an example how this works. Also in this Workshop.

B

The other thing is like a kubernetes cluster. Has many many like volatile things going around so there is like container environment and we really need service Discovery, because if some something comes up like new, we really want to like add it automatically or have it available to being just approved or configured. And then it's visible in in the monitoring and observability dashboards.

B

um The other thing which I really like about Prometheus. Is it it's open source and it has an awesome Community, um but it also provides a query language which is called prompt ql, where you can um create your own queries calculations and provide that, like as a raw data or within the graph, and as mentioned before, it has its own time series database as a backend.

B

There are other things around scaling and long-term storage which which we will be taking in a little bit, but just like for the architecture how it Prometheus fits into the picture with kubernetes as well as you have the Prometheus server, where everything like runs together in in it.

B

We have service discovery which works with kubernetes with different other service. Discovery mechanisms like if you would be running a console or some other toolings, which provides Discovery capabilities. You can also use that with Prometheus. So Prometheus is works very good in my experience with kubernetes, but it's not exclusively tied to kubernetes. So you could also like install promises on your Raspberry Pi and monitor the temperature sensors um for humidity and something else for example, or do traditional infrastructure monitoring.

B

Now the idea with like Prometheus is you either get the uh the metrics from drops and exporters? There's also a push Gateway for short-lived jobs and when it comes to alerting there is a component or a service called alert manager, which we will also be trying out later on, so that we get alerted from matching a certain threshold or certain violation of our conditions of of being healthy um and we connect upon things and don't need to like look at the dashboard and say: oh, this could be broken.

B

We might be looking into fix now, um but instead getting actively alerted um yeah and for a form, curl, there's the premises, web UI and also later on, grafana and and other ways to visualize things now um prom Kell.

B

This is something which we will be using throughout the workshop a little bit.

B

um It's it makes sense to remember how to, for example, Define a range with uh the scrap brackets for the metric name to query or how to like apply labels with the with a different type of I think it's called Angel brackets, um but yeah in the in the end.

B

It's like it's the different one to the square brackets, um and this is like me Lost in Translation, but in the end it's really important to remember that, but also like have the cheat sheet for prom curl open, which are also linked at the bottom, and also how to define functions which allow to, for example, calculate a rate over time, calculated the summary or maybe even a Delta, and also do comparisons, which is later on important to say, hey if the metric.

B

If, for example, the memory U consumption goes over a defined value, this condition is rendered true and therefore I'm, sending an alert or I'm showing an annotation within the graph that something is broken.

B

um Yeah lots of theory. We will soon dive into action um before we do that. um There are different uis available for Prometheus, so the web UI itself can be used for testing the script targets so things we want to monitor in the future. We can create a metrics and we can also like create simple graphs for dashboarding. We will be using grafana in this Workshop, so you can Define Prometheus as a data source.

B

The time series database in grafana and also grafana understands prompt ql so provide some like additional order, completion and query helpers, which makes it more convenient to like really build something and later on automate, it and and other things, um and it also allows you to create panels and dashboards and save that and and share that.

B

um Another project which is currently in development is called prses, I'm, not sure if the name will will be like persistent when it gets released. But this is like the idea of providing dashboards as a code, um certain like gitups patterns inside um and it's currently in development.

B

So if you're interested in like following this, along or maybe even contributing I, would recommend checking out um the uh the linked GitHub repository in there um yeah last but not least, around terminology service, Discovery kind of works with using a script config for a dynamic list of targets, or it's built in so like kubernetes service Discovery is built in, um but there are like more ways of providing it.

B

We won't be like using uh many things um in like, as I said um in like in a raw like you will be seeing that it's using service Discovery, but there are customer resource definitions and other things which will abstract that knowledge later on now. The thing we will be doing now in the next exercises is we will be deploying Prometheus inside the kubernetes cluster and oftentimes when I share that like detail, uh training, um students and then folks ask like yeah.

B

But what happens when the kubernetes cluster goes down yeah, the thing is, probably you might be having some external monitoring or some additional monitoring which says well, the Q a this class dies down. You should be like doing something about it.

B

um If you, for example, add some like external monitoring and open up the kubernetes API with root access, this is this, isn't considered best practice so like something check, kubernetes and then having access to everything um could also like be a backfired to like you, shouldn't, be opening up your kubernetes cluster to the to external audiences, um especially because monitoring requires typically root access, um which, which means you might be exploiting sensitive data after all.

B

um So there are, there are other advantages like you can monitor the health of the cube of kubernetes itself. There are best practices regarding uh policy, security policies and permissions so that you so that the monitoring system or that Prometheus doesn't need Rolexes or something like that.

B

um Yeah and, like I said before, can use ping probes from external to monitor the monitor, for example. Now, thinking of how can we install Prometheus within uh kubernetes, there are different ways doing doing so. One of the things I found really reasonable is using the Prometheus operator uh project which is like in in uh I, won't say extra. It's it's an additional project maintained by awesome people um providing the way to install um Prometheus inside kubernetes and also automate the configuration of everything.

B

So you won't be needed to edit, like this configuration file and another one, because the idea is really to provide custom resource definitions, so you can deploy and manage everything from Prometheus to the alert manager to all the components which are which are then being deployed um and have um like the the best practices being available as well.

B

um The other thing which is helpful is like um generating things which you would be doing manually or would need to copy paste thousand lines of configuration um which is done using the premises operator already and to be fair. I haven't looked into everything of the premises operator. It just worked, and this was one of the reasons I said. Okay, this is something I want to use and also teach in the future, especially with Cube Prometheus.

B

This is like an opinionated deployment of the premises operator and provides an environment which is called Prometheus, Dash k8s, if you see it later on which also comes with alerts.

B

Predefined and specific rules, because, like one of the things which we will be seeing later on is metrics are nice, but if there are no alerts, the dashboards might not be as useful as a spawn intended them to be so that the idea is with Cube Prometheus to install all the components which would install install otherwise manually, which is like the premises operator premises in a high availability mode. Alert manager is also in high availability. So, like two instances there is a node exporter and the nice thing is.

B

There is like an adapter for the kubernetes Matrix API, which is called cubestat Matrix, which doesn't need to pass anything else and also like cubestat Matrix is a community project on its own, which is also awesome, providing all these abstractions and layers which I don't even understand, but I can really like consume it and the premises of writer deploys it.

B

um Grafana comes by default as well, and the great thing is there are certain dashboards and also alerting mixins they're called like mixins um being provided by default. So when you install it, you get like out of the box monitoring and can just um work with it. The last point on the slide is um dynamically: create service, Discovery objects, which we will be seeing in a bit.

B

We can just like Define a Target to monitor for Prometheus using a custom resource definition, and this is kind of the only thing we need to do, which, which is an awesome way of doing so now. The thing is um the cube. Prometheus project has certain compatibilities being listed, so if you're, not using one of the latest versions of um of kubernetes and I'm, currently using 1.22, we should be using or I should be using the release.

B

um 0.10 newer versions are then supported in in newer kubernetes versions. So the thing I did for this Workshop to really like work is I've prepared, a project which uses a sub module, which has a git sub module, which only uses the the 0.10 version, which is in this project. So, in order to really like do that, um we're gonna um clone this repository now and in order to initialize the sub modules use dash dash recursive, um and we want to do it just right in there you see what did I do.

B

B

Okay, for some reason, the Double Dash is not correct on the slides. This is the problem using different systems.

B

If it breaks for you, please correct the double dashes. I will look into a fix later on. um Now it tells us it has initialized the sub module and so on. So um looking into what is like inside it's a clone of the git repository of cube Prometheus and we can use cubectl.

B

Appropriate of cube, CTL create f, but for now just the Manifest.

B

um So there's like uh this, this order is required to create first the resources which are needed for the custom resource definition and also the monitoring namespace, and there is also like a check in there, which is this one. Until the service monitors have been initialized.

B

But should return no resources found, so this is kind of expected um that it really works and doesn't throw an error, and afterwards we can use curves up to create everything else in the manifests.

B

Actually, it throws an error for the setup again, but this is also expected and should be doing something. So this is like the step watching things doing something and we can have have something to drink or something else.

B

um And uh what we should be, having a little more um thoughts, for example, which and immediately see something that's going on. We also notice in additional namespace, and we will be looking into this shortly now. um These steps are um just copied from the um right task, buys always switching when I'm moving my mouse. These steps are like found in the in the official installation as well.

B

The only step I said is I'm using a different version for cube promises now for this Workshop, but you can like replicate this by yourself now um the thing we will be doing now in this Workshop is we um we, we will be using port forwarding uh with Cube CTL to access the HTTP resources, so we will be accessing Prometheus, grafana and also the alert manager and create port forwarding for that for a production deployment.

B

This isn't really what you want to do, but since I don't have any domain available and anything set up with an Ingress controller in that regard like nginx or traffic or something else. This is something which probably would make this Workshop a little more complicated and I want wanted it to like keep it as simple. As possible, so um the next slide has some ideas how to get started, but we won't be doing it now. The thing we will be doing is like open a new terminal and adding port forwarding for uh Prometheus now and.

B

B

Probably I should have opened more terminals um yeah, but in the end we will be adding this into the background. So this is like um the port forwarding command we're mapping it to 10 000 and everything is definitely which is in the output and putting it into the background. Sometimes it can happen that this is like. It kills itself or goes like to to death knowledge itself, and then you need to repeat the steps but I've added all the instructions which I'm like needed in future exercises as well, um so copy pasting.

B

The second one and.

B

Yeah- and we already saw something end of file- okay,.

B

This is interesting, not working in the deployment yet then it probably makes sense to have it run in the back in foreground already um and the next thing we need to do- and this is like- sometimes port forwarding works. Sometimes it does not. Maybe my internet connection is not good, because it's turning like um 5 p.m, in in Germany and and folks are getting home. They are like starting Netflix streaming or something like that.

B

um No um could also be oh, oh trusting dialing back at the end of fire. Maybe it's something else in the deployment.

B

B

Is there something going wrong? I think should be moving already pending the node Explorer spending right now.

B

um I would assume that everything works, as is expected now, at least um when we will be checking first off the locally host 10 000 Port, which is now mapped to the Prometheus UI.

B

We should be seeing it actually, yes, um and in order to test that, let me quickly see promises HTTP, request yeah we can. We can, for example- and this is a nice thing, with other completion in the in one of the newer newer versions of promises we can use HTTP requests. Total can execute this query and visualize that as a graph- and we can see that something is going on already in the past five minutes.

B

um So we get to see that promises is actively collecting metrics within the deployment, and this is this is kind of the HTTP requests um which have been made already to the promises. Api itself, which is currently me doing this request, um you should be seeing the same. You can totally query for something else which which comes up in the uh um in the autocomplete version, but this is like really getting to know that it's up there it works. You can also and I did that in the screenshot. You can also switch to black mode.

B

If you prefer that and do quite some more things which uh which we will be looking in a bit, but in the end it just promises, you don't need anything else now we will be getting certain uh kubernetes metrics, and this can be quite overwhelming over time.

B

B

um The idea is, and this something we will be seeing in in a bit- we will be um looking at cluster metrics, so getting to know the node status, the resource usage, the deployments, the number of ports. This is everything which uh Cube promises and the promises operator provides out of the box already. So we don't need to think about. How does the Chrome query look like how to like aggregate things and think about how it's actually like being calculated over time, for example, or create all the dashboards and panels ourselves? This will be provided.

B

The challenge for us is later on to figure out what is needed to figure out the health status of the Q, a disk cluster or what should be put on the dashboard to immediately see a failure of certain things um but yeah.

B

There are also like pod metrics for for kubernetes itself for the containers for the applications, but really like a lot, um but it's great to have something, because, if you don't set, if, if there is nothing, it's it's even harder to start with for container metrics, um the the cubelets provide an embedded C advisor exporter, which is also wrapped um by Cube Prometheus and can be used, for example, to do resource usage, monitoring out of memory kills and so on.

B

um So like accessing container underscore CPU underscore system and so on can be very helpful um to see things um working in the cluster. The other thing, as a reminder is Cube State metrics, which provides an insight into the health of deployments, nodes and ports and it curves the Q a this API server. So it's like wrapping things to provide Prometheus metrics just out of the box, which is super helpful to get an insight and not like building it by yourself.

B

um In order to like have that visualized, um the dashboards which come with the premises operator are built in grafana and grafana is an open source. This is open source, visual, visualization and analytics software that allows you to visualize and learn on and explore your metrics um I think I copied it from the website.

B

So um the the idea of grafana is to not only like use um Prometheus as a data source, but also attached to my SQL elasticsearch and other things, to provide a way to create dashboards and using panels being able to share that and many other things um in. In our example, we will be using it as a dashboard to query the Prometheus metrics Within kubernetes, but on its own there are like many more ways to do that.

B

um The thing which is great is like using the panels as a building block with the query Builder so like having um all the completion and suggestions um which are helpful. Do not um like I did 10 years ago, Googling for how this is called and how this is called and then copy pasting and hoping it works.

B

There are many like visual helpers, available, making your life easier when you're building dashboards, um so in order to access grafana, we also will be using port forwarding, and this is the command again if in order to uh verify it works and I'm hoping it works now um to accessing um the grafana UI, um it will prompt us to change our username I would recommend not doing that.

B

um So it's oops it's admin and admin for the purpose of this Workshop. We are not exposing the ports anywhere and just like a thousand pop-ups. That I should be changing my password now, because I yeah, no um I just want to really use that for this Workshop I think there are ways to kind of this it there for the cube for the Prometheus operator and the cube promises deployment.

B

There are ways to disable um the grafana um password reminder thing or to customize it in a way to, for example, attach to your SSO and an arbuck um integration and and other things. So this is configurable, but for this Workshop I just said: okay, let's dance uh the browser dance and have it available for now.

B

So um this is like the grafana interface and we can also see that um they are recently viewed dashboards, which have been pre-installed with Cube promises already, which we will be looking into a bit, or at least right now so um yeah. The next exercise, after like logging into grafana, is to figure out what we have there and as a reminder for myself I've put in there.

B

We will be looking at the kubernetes API server at compute resources at the cubelet, and we can also see um the health status of Prometheus itself in the deployment okay, so um from the menu over here we can use browse and default.

B

Okay. Let me actually navigate into this folder right now, and the kubernetes API server, for example, could be interesting.

B

um We don't have any activity that passed 30 days yet, okay, this is kind of expected, um but we can see that we have right, availability. Okay, we have something going on already. The CPU usage is currently at okay, it's not so much, but we can like deploy something and the CPU usage will be much higher.

B

Or deploy something over the kubernetes API server. We want to deploy something, for example, which queries the kubernetes API server, quite often, which we might be looking at the end with chaos, engineering and so on, um but yeah in the end, it's providing us with some resources and I think we can navigate back like this and it's again I, don't know everything so I also need to write it down and use the slides as a resource.

B

We have the compute resources for the cluster and since we don't have everything available yet this is like the cluster is using a little bit of CPU um and the good thing is, um for example, if you're, adding monitoring or like pure the premises operator into your monitoring.

B

um There is a certain cost of resources and infrastructure like which is assigned to the monitoring namespace. So if the monitoring namespace, for example, consumes all the resources in your kubernetes cluster, you need to think of like scaling the monitoring um and the deployments in there as well, but the cube system is consuming something and so on now this is um we get to see that we have the ability to get um dashboards like grouped by namespaces, for example.

B

This is using prompt curl in the in the back end or in the in the in the background, um and it's super helpful like to being able to see which namespace consumes, how much Memorial affects other things as well. The other thing we can inspect is um the cubelet and they are potentially like I didn't modify anything, but yes,.

B

Cubelet itself, there is like the container status.

B

um Hot start rate so like if parts are restarting a lot, this might also be something you want to alert on um or at a certain point, like a crash loopback off when Parts restart too often, then kubernetes kind of injects a delay um which is like the the back off um term in in the description um in order to like not to stress to restart, um but in the end you should investigate on these Eros um why they happen, and this is like a really helpful insight now, um okay, with the premises overview as the last one to look inside, and then we get to action ourselves, um there is the Prometheus overview, so we also get to see that Prometheus itself is healthy.

B

um It has a lot of targets. It wants to scrape in this deployment and I think the um recovery rate yeah. This is um This Is Us in that regard, yeah.

B

So in the end we get a lot of things already and um I think this is something I copied from a longer running system where we can see the availability and so on, but now we want to create our own um dashboard, a dashboard group. For now we navigate into dashboards, create a new folder call it workshop, and then we create a new dashboard. Okay.

B

um I'm not monitoring the questions in the chat, but um if there are some like um or if you want to like, if you have your dashboards available or want to share tips and tricks, um potentially you can just go ahead over there as well.

B

um What we will be doing now, let's navigate into the dashboards and for some reason this should this looks like an arrow or a buck, or maybe it's a feature. I, don't know I'm, not just kidding. um We want to create rows ah over here. We want to create a new folder, which we call Workshop and also call it. Cube simplify Workshop.

B

S simplify Workshop.

B

And we want to create a dashboard okay. Now it greets me with something, um but this is like the second part of the exercise when we have a dashboard, because the dashboard is not saved yet, but it asks us to create a new panel before saving.

B

uh We want to add a new panel and we want to monitor specifically the container processes.

B

So we get to see something so in order to um add a new panel, we'll click on this section over here and get greeted by a lot of data which I would recommend like focusing on okay. This is like the the preview of the panel.

B

um This is the query, so this is like collecting the data source and then executing prompt curl for our example and on the right hand, side we can modify the panel title, The Legend and many many more things um before we can like save and apply this now.

B

um This is the demo data provided by grafana, but we want to select Prometheus and the metrics browser should be in a processes.

B

I can just use enter.

B

And either enter or just moving the mouse Focus out there to apply this now we can also like modify and say we want to see the last 15 minutes. Maybe 30 minutes something like that. We could also like modify the uh the bullets um or make it a table instead of a graph or something like that. But this isn't something we really want to do now.

B

um We just close container processes and bumpy, don't be irritated. My browser knows that I've been giving this Workshop um in the past, so this is like stored in there. This is not a grafana autocomplete feature, but yeah. um This is something we want to apply now, so we have our container processes over here and um it should be working as expected.

B

um I think the screenshots should still be applicable. I just I was moving a little too fast in the exercise. We are now like editing the panel like container processes, applying it and um safe we're saving it as a first dashboard now. So the important part is this is still like a draft dashboard, and now we need to um save the dashboard. This is this button and we want to call it first dashboard in the cube, simplify folder, um and this should be good to go now.

B

The dashboard is saved and we could, for example, share it. I think, probably not, um or there are some dashboard settings yeah.

B

um The thing is, you can export a dashboard as a Json file and then import it in in a different installation or even like upload it to the marketplace or store it in your preferred kit, Repository and so on, but for us for now it's really like getting to see something.

B

um You can also set the refresh um intervals of the browser refresh and it should be updating automatically over time. The most insightful thing right now is can like hover over this. We can use, um drag and drop to resize everything.

B

um Just this works like a git history, so you can actually add notes when you're modifying a dashboard and so on, but this goes a little bit In Too Deep for grafana in that regard. So um now that we have created our first dashboard, we want to create a kubernetes dashboard for the following metrics and in order to practice that we click on the cube, simplified dashboard thing or we are just navigating back into dashboards, browse I.

B

Think this is better to um to highlight and when we hover with the mouse over the cube, simplified Workshop, there is go to folder, so we are going into the folder and um we are creating a kubernetes dashboard. What kubernetes observability, dashboard and monitoring dashboard, um which.

B

Okay, um which we sorry about that pop-up, um which we will be saving as um it is.

B

Yellow will be saving is just as an empty dashboard. Okay, um now one of the things we want to inspect is like the container metrics. So previously, I mentioned that the cubelets provide this, the C advisor Matrix, which we will be using, and it's at some point.

B

It's interesting like what are the resources being consumed by containers like CPU usage, which can be divided into system, user and and overall usage and, for example, also the the container memory uh so, for example, if if an application is leaking memory over time, you might be seeing a trend that memory is going up um which, which can be an indicator whatever you should be doing something um or like restarting the port, um to kill to um remove the leaks, could also be an option, but in the end, if there is a memory leak, it should be fixed in the application and not in the deployment.

B

Now. um The thing we want to do is um first inspect the compute resources for the cluster to see what is going on and then build it ourselves now, um moving back into grafana into dashboards. We do have the um no me and navigation Pros.

B

We have kubernetes and compute resources cluster this one, and this provides us an overview already with CPU utilization request and so on. So, for example, the thing would we can do you can edit a panel- and this goes a little beyond the workshop now to.

B

Edit this panel and that there you'll get to see um The Prompt query, so you could, for example, copy paste that into your own dashboard. This is like one way of figuring out how uh these things work, but then for our example. We won't be doing that.

B

We were just looking into um querying the container metrics ourselves, so we're going back into our own dashboards and in the kubernetes dashboard we will be adding a new panel.

B

So let's move into the kubernetes dashboard go to this folder. We have our dashboard over here and we want to add a panel which should be using the Prometheus data source, and then we want to look into the container CPU usage in seconds for a rate of one minute. Okay, let's try to build that.

B

We want to have the CPU usage.

B

In seconds total and in web brackets, one minute this doesn't work yeah, because I didn't specify any range um well, I need to specify the rate. I was kind of expecting that grafana would kill me now.

B

um Apparently, I need to know that by myself, okay,.

B

This is the first thing we want to be adding and we want to have like 30 minutes to see a little more than.

A

B

Or maybe in the future? Let's, let's just keep it one hour. You can also leave the the six hour default. If you want to.

B

um The other thing to add is um the summary by namespaces, so this is like the the same rate, but providing some as a function and then Group by namespace. So let's try to replicate that with the existing query we have been creating over here, so we can copy that and we want to add an additional query, which is the sum by name space, and this query this book, no, um oh yeah I made a mistake.

B

Actually, when you're creating a sum, you need to provide this thing, and so we get to see the summary of all namespaces. So this is like each namespace on its own, and this is um some by namespace and.

B

Plus create some by name spaces. We are saving the panel as a CPU container, CPU usage.

A

B

B

um This is currently in seconds, let's pick that one and apply this, and we also want to create a second panel which.

B

um Oh, this is about the memory um we can or we didn't deploy, deploy that yet uh we can just use container memory usage, bytes.

B

We will be deploying a.

B

Service demo in a bit um I think I mixed up the exercises. We want to look into container memory.

B

B

Usage bytes; okay, to be honest,.

B

B

um Many time series results returned okay, consider aggregating with some oh yeah. We can totally do that in a bit um port and container okay.

B

We can, let me see.

B

We have the ability to um we can like select the metric. Then we can apply labels using uh this one and we can do that by container, um which could be. Potentially we can do that in a bit.

B

Yeah but let's just.

B

B

Where is the pedal title?

B

Oh ah there is the panel title.

B

um And save the dashboard and look into this in a bit when we have applied an application. um But this is like also to practice and potentially to take a note.

B

B

Church um yeah and in the end um just doesn't make this doesn't make sense right now. We need to do it later on um for cube State metrics. There's also a way to query that ourselves just to see which prompt calculator we can add a new panel in the kubernetes dashboard, give it a title for the replica status and query the cube deployments back replicas and Status replicas available for the specific job tag is a selector. Now, um let's add a panel in the kubernetes um dashboard use Prometheus as a data source and.

B

B

Tricks no Cube deployment, spec replicas and then speak replicas.

B

And we want to take it by a cube, State metrics. This is like using the uh the entry brackets and the second query we want to add is.

B

The status available, okay, Cube deployment.

B

B

Deliver and same again for the this thing is nice, so I don't need to like think a lot um yeah. This is a replica state or students and try this one as well. So we get to see.

B

The different um containers and their replica state- currently we didn't- we don't have anything deployed yet in our kubernetes cluster, so we're inspecting kind of left, metrics from the monitoring and the cube system. Namespace.

B

um The other thing to keep in mind or to like remember, is that Prometheus wants to monitor everything which has a slash Matrix endpoint, so we can use Prometheus exporters like inside the kubernetes cluster, the node exporter or, for example, using a black box exporter, which is like Pro, a ping probe to monitor or instrumented applications, so applications that already provide an HTTP endpoint using the slash, metrics endpoint, and also the the open metrics format for for exposing the values.

B

And the great thing is around this- that the Prometheus operator provides a service, monitor custom resource definition for this, so you can Define the target endpoints and it also discovers the targets from different namespaces. So you don't need to worry about like how does this work together and this work together? It's just really get like using a customer resource definition. So, whenever you deploy an application that provides a slash, Matrix endpoint ship to service, monitor crd with it and you're good to go um to integrate it into your Prometheus monitoring deployment in in kubernetes now.

B

The thing we want to do now is to practice this one. We want to deploy um a demo service application and before we do that, and the thing is you shouldn't trust, anyone on the internet, deploying your Yammer file and say: hey. Please apply it into my kubernetes cluster. We will be looking what we're doing what we will be doing here so in the end, um oops did I rename something potentially I renamed something um we have.

B

The premises demo service over here.

B

Oh I moved it into a subdirectory okay, so we need to like fix this now live.

B

um This is a demo Service, uh which uses a container that expose exposes slash metrics for some demons, so the containers actually provided or written by Euler's Waltz, one of the the co-founders of Prometheus and I um forked, the project and just enriched it with uh with some builds, it listens on Port 80, and this is like the deployment parts we we're defining three replicas um for that, and the service itself um exposes Port 80.. This is this is about.

B

This is the thing which is about to happen now um and we are going to apply this and before I do that I will be fixing the slides live, because um this is I want to do that and in order to access this one, we are going to open this Raw copy paste. This one and.

B

Also fix the slides, I've never done this in a workshop before, but it's nice that it's it works. So we get to um copy this and it will apply um added into our oh now. The determinant is a little cut, uh we're applying this service into our deployment. It will be using the default namespace just for simply simplicit can I just to like use it, as is as a demo service.

B

So we get to see that we now have to Port uh three Parts being created because three, three replicas- and this is now running in our kubernetes cluster- um so.

B

The application itself provides some metrics, and this is like to verify that, but we really want to monitor that. So um when we open the Prometheus monitoring targets, we can see- or we don't see like something like ECC yet so there isn't something which, which is uh which is monitored already. There are many other targets which are being Auto discovered, but our deployment isn't monitored. Yet, in order to like fix that, we need to create a service monitor, custom resource definition that tells Prometheus hey.

B

Please monitor this and the thing we do again over here is: we want to monitor this one. So um the idea is: provide the service, monitor, CLD, apply the label and it matches the selectors match. Labels for the app ECC demo service, which our deployment and service provides, and the end point, is the port mapping web, which is defined on the other side. Now again, in order to use that open raw and deploy this one into our kubernetes cluster.

B

B

We are, we now have the service monitor crd created, so the thing what should happen now is when we open up again the Prometheus targets so like it's slash, Targets on the Prometheus UI.

B

At some point, we should be seeing the service monitor CLE. um If I didn't do anything wrong um for now. We can also use L get service, monitors.

B

And oh yeah in the monitoring namespace: this is what we have in there and if we want to have it like in all namespaces, we have the ECC team observers in the default namespace. So at some point, um Prometheus will pick that up.

B

It takes a while, because the scraping of the service Discovery targets is not like live it. um I don't know exactly. This is 10 seconds 30 seconds. Something like that. So.

B

To find the right browser window and probably I, should close certain things. No yeah. We have the service, monitor targets for the ECC demo service for all three replicas. So the only thing we did was like a short configuration for the custom resource definition, and now we can actually like monitor these Services. Now, if I click this URLs, these are internal IP addresses in the cluster. I cannot access them.

B

um I could, if I, if I, would be using portfolting, but the more interesting thing is since Prometheus is scraping. These targets there are metrics available already and I can see. Hey I I have labels applied automatically, so I have the container name. I have the end point: I have the instance: I have to draw up the namespace, the port, the service, so this is provided out of the box uh using the service. Monitor crd and I can actually use that data to query for these metrics in grafana.

B

Now, um which is a beautiful shortcut to I, have no no idea how to do it manually. To be honest, um so let's try that and add um a new panel in grafana use the data source Prometheus and in the metrics browser we are looking for demo underscore demon underscore CPU um and we'll also want to filter by the container, which is called ECC. It's an abbreviation for everyone can contribute Dash, demo Dash service, um hopefully I'm not talking too fast, but I'm.

B

Looking at the clock, we are like one and a half hours in um and we have quite a few things to learn so um I'm I'm moving forward with uh the Practical things now so um now I will be closing certain things. This is something we don't need. We want to navigate back into um our kubernetes dashboard, which again we can access from over here, navigating into the cube simplifier workshop and then opening the kubernetes dashboard, which we will be just adding a new panel over here then go again.

B

Add new panel, typically workflow, so we're like we're learning as we go along um new data source and we will discuss later on how to automate the creation of dashboards because doing that manually. All the time is, like think of manual work as a bug.

B

My friend said that a while ago we need to automate certain things, but for now we are practicing our Chronicle knowledge and also how to use grafana in that regard, and we want to search for demo CPU you touch seconds in total, I, think yeah and the label should be, and let's click on the metrics browser.

B

We want to use the label container, which is pre-selected, which is awesome and from there we can say we want to just use. However, we have a lot of containers here. We want to use the ECC demode. This automatically creates like the selection, so um yeah the resulting selector is container.

B

um Should we be using this raw? Should we be using a rate? Oh no. Actually, this is like the hint that we should be um use as a rate query. Okay, this is, we will be using it as a red query, as suggested for some reason. It doesn't like us with the interval, but we can like manually, replace it in five minutes, for example, and I did something wrong.

B

uh No, this doesn't make sense hmm again.

B

ah Yeah, probably ah okay, now I'm learning something new today, if I type it in The Matrix browser above it open it later on, it doesn't copy it to select the metric okay.

B

um Okay, we want to this is this is what I expected. This is like the preview um of the selector so like we are accessing this metric, applying the container label and we can use it as a red curry. This is what I want.

B

um This is um okay, I, don't really understand where this is coming from, but this is probably a default which is not set um for our case. We really want to like focus on this one, and we can see that the the demos service- and this is also the reason why we created three uh replica- sets just to have some data available to play with. We can see that there is a lot of things like going on some demo stuff and um can save that panel as um ECC service.

B

What's up and get to see that over here, Matt am I doing my own exercises right? No, um it says one minute: okay, um you can modify that. It's like one minute sampling or um five minute rate sampling um with that record.

B

So typically, um when you're coming from the Linux or Unix world, it's like CPU usage is like calculating it over time, because the raw value of CPU usage doesn't give you any insight, um but if it like, if the rate exceeds from one minute to the next minute, this can be an indicator whether CPU load is going wrong or not.

B

um Okay, we have applied everything and I was a little too fast with the exercise but yeah. Now we can see certain things from the CPU usage and just to recap. um This is a demo application which has been deployed.

B

um We deployed it in a little bit over here and in order to verify what is actually going on in in the demo. We know that this is like this project, which provides this.

B

So you know that exist is, for example, um can see that this has been forked from us, Repository and um oh yeah I needed to change uh the docker image to dbm because it didn't work in my tests, but in the end this is like this is an open source project which you can inspect um for your own deployments for your own learnings, which really allows you to immediately see something in the deployments and also provides a great insight into how to instrument your goal code in this regard.

B

With this regards to um it's a metrics and see something now um when we move on and I call this like monitoring 2.0, but probably it's like just monitoring um and things you want to apply and want to learn like um we know that, like we have service level objectives with monitoring metrics, we have the key or the the key to indicate things.

B

We have tags and we have the values now, one of the things you can do- and this is more of an advanced exercise because not everyone knows python, I, know python, but I also have zero or little knowledge in node.js or other applications. So it's really like getting to dive into or learning how to instrument application code and use the Prometheus client libraries to instrument your code.

B

So um when we, for example, look into this is the Prometheus client Library, but we can just- um and this is one of the set examples for Googling together, client Library, um you do have the documentation for Prometheus the client libraries that are officially supported and those which are provided by The Wider community, and you can see there is go Java, pyths and Ruby rust and as well as like an instrument, bash, C, plus plus c, and many more, and also like no chess, um which I probably would need to learn um for python.

B

This is the official um Library, which also provides a quick five minute success demo, which you can use and also like wrap around the service deployment or the deployment and the service, and also the service monitor. This is something I did for this exercise.

B

In the end, the end, it's it's really up to what which language you want to learn or use in your applications later on. For me, this was now python and um the python deployment.

B

I'm talking too much losing my voice again.

B

um This thing has um the deployments, which is kind of a copy of the other thing, and um the important thing is like: how does how does this look like?

B

um We do have a deployment which provides us. We can query something we are creating a request um which, in that regard, is just some random sleep time, and this is being uh monitored by that. You can see that I created this for a workshop in the previous Workshop, so one of the ideas was to really do that together now, but since we are not in a room, um but it's potentially a little complicated to follow along in the end, the idea is to really like being able to Fork the project Modified by yourself.

B

Add some code duplicate the things which I implemented here and deploy it and see what is going on now. I will do that on my own pace now in a quick minute, probably not because I'm not logged in, but in the end um this is the python deployment, the application um we can deploy it. So we can. um This is yeah.

B

We deploy the application itself.

B

And the async exercise for you is the factory repository to modify the code as instructed in the slides and modify the deployment or the image location in in the in the Manifest and have the first success with the deployments.

B

Sure why I'm locked out currently but or locked out in my gitlab.com account but yeah, probably because I'm sharing the screen? um Okay, um this is basically what we did before with we're deploying creating a deployment. The service and the service monitor CID for the premises operator, and we should be seeing this deployment in the targets. So, um if I'm moving back into Prometheus into the targets directory, we should be seeing something hopefully soon it's the python is not there yet.

B

Yeah it's the ECC python service. If I would have more time, I would probably create some something in Rust, um but this is not like um getting to know things setting up a development environment if it's not there already and really like getting getting started, will take like up to an hour or something. So um there is a note at the very end of the workshop, like um learning instrumentation code instrumentation for observability.

B

This is like an a workshop on its own, which probably we can do in the future um or someone else does it like. um Maybe Dolton Horowitz from loxyle wants to do it for Telemetry, for example, or someone else just making suggestions now to Siam.

B

um But there is a lot of things to learn, really um and I'm, not trying to replicate everything in this Workshop. Now um I was just trying to figure out why this is not there yet. Oh yeah.

B

We actually have made a good state so for the first uh deployment, this happened already, but the second isn't scraped yet, but this will probably change in a bit now it's script and we also have the metrics and we see that there are like different uh labels applied or relabeled for that deployment, and we also have the possibility now to monitor it again, um which we can request.

B

Processing seconds count: okay, let's try that quickly in the dashboard create a new panel and again Prometheus and the metrics browser this time should be.

B

Good question: what did I say: request processing, okay, I'm a little confused by this many top best processing.

B

um This one, the container, should be this one, so we can actually like use this and actually use it as a red query again and.

B

And we do have only two replicas here, so we only have two like two data points, um I'm, not sure. Okay, this one goes down. This one goes up, um it uses it uses random values. So in the end, it's the uh the python service.

B

Which is exposed it's a metric and, for example, if we would be adding more metrics um or like scaling this up to a third replica. We would be seeing this immediately here. So there's no there's no action needed for like modifying when you, when you scale things up. It's only. What matters is the query: uh the prom curl query and the label selectors, and so on.

B

um Yep, we won't be doing the code modification Instead This is like left as an exercise to the reader um and for later on now um we have talked a lot about like metrics um and metrics are really like great and we've looked at dashboards and oftentimes. You have you if you have an office if you're not all remote like gitlab?

B

um No, if you want to have a dashboard like a monitor at the top or somewhere else like working as an SRE, um You, probably have that somewhere around oftentimes you're monitoring or you're, seeing a lot of metrics, um but on the other side it's like it's.

B

um It can be business critically to get alerted from certain thresholds, and um this is something we call alerting and also Incident Management attached to certain alerts and also like service level objectives, and we will be dealing with that in a bit. The idea is like what what is coming after metrics so like if the definition of failure, the definition of not healthy, um we want to do something because something failed, like a threshold has been violated um and then trigger a rule being matched like notified.

B

Race alerts, notify specific or defined set of teams of personas, um and so on, like the who um and the how is like documentation for incidents. So if there is an alert, it shouldn't be just a message: hey something is broken. It's helpful to provide as much context as needed to really fix the problem in in a stressful situation, which can be like weekend.

B

3 am in the morning, and the customer is calling and nothing is working like having a run book to replicate things, to fix things um and also being able to document the incident in a in a ticket system in an incident management system and also Define corrective actions, for example, and the idea is to really reduce the mean time to response which is called mttr.

B

Some code at the meantime, to resolver or to solving problems could also be the case, but it really needs like iteration or that now, when we look at alerts with Prometheus- and this is kind of the same with Prometheus grounding in kubernetes um Prometheus evaluates alert rules and sends these alerts to the alert manager. The alert manager can be used for grouping. So that, like you, don't have any duplicated alerts.

B

It provides with inhibition, which is like you can suppress alerts when others are fired already. So your mobile doesn't get detosed from notifications and alerts, and it's also possible to acknowledge a problem which is called a silence. So you can mute alerts for a given time and say: hey I'm, working on the problem. If it's not fixed by like five five am in the morning, um then please alert me again about this.

B

um This is essentially helpful for Encore um shift and handovers and and other things like saying, okay, we're expecting this to being fixed um from a rollback or for something else, and if it continues to be a problem, please continue alerting um for the transports. You can use various Integrations like the default, not the default but SMTP, which is for email. You can configure web hooks push over and SNS, which I think is an AWS service, not really I.

B

Don't really know about this, but it's also possible to integrate pages and also use check, uh chat so slack, which had telegram whatever comes to mind um with regards to kubernetes um the uh the premises operator and Cube promises. Deployment already provide certain alerts out of the box, but it's helpful to know what what else is out there or what? What could be interesting. So it could be like an infrastructure alert.

B

It could be um something around memory, CPU or even disk pressure, um the pods, the error rates and the response times from from services and from apis, for example, and the lower reachability I found a website which is called awesome.

B

Prometheus-Alerts.Grab.To- and there are like rules available also for kubernetes, which allow you to like copy paste them and and also inspect them, whether they are being useful for your production environment.

B

Now, if we want to integrate the rules or the alert rules into the Prometheus operator, um because we typically, we don't want to like keep a separated set of resource definitions and and a lot rules um ourselves, um there is the way of providing or using the Prometheus rule custom resource definition, which wraps the Prometheus configuration language into the kubernetes native custom resource definition. Language and allows you to really like have that appliable as a q, a dis manifest and just um benefit from kubernetes native alerts with Pro musics.

B

This takes a bit to really like learn I've, like highlighted certain things which are familiar from the the previous rules, um but in the end um an alert is like defined in a group There Are Rules, which need to be like evaluated so they're alert for this one could be a cube, State metrics list errors um and the annotations, which is a description and the summary of what is going on, which also provides, for example, a runbook URL, so that you can immediately immediately take action on the error.

B

This is important context to provide the expression is written in promquel. So this is like a multi-line prom quill expression, which uses a comparison operator, um whether the rule alerts to Boolean, true or false, to being executed and the alert being fired, and it needs to be active for 15 minutes, for example, so for testing you can like reduce it to one minute, but it's really like defining the state of error. If it, if it happens, for 15 minutes or long, it should be alerted not before, and you can also apply certain labels to that.

B

So, in the end, it's like it's. It's a super useful language to being used also with the premises operator um before defining all alerts by default and have like 10 000 things which can potentially be alerted. It also makes sense to really like see about what you need and what you immediately would be seeing so like in a chat. You would be seeing something immediately, um but for longer term installments in production.

B

It makes sense to discuss and also document refinements, so maybe having some grouping um always like doing a review of what you're getting within Incident Management Systems, because the problem of getting tired of too many alerts, which is called alert, fatigue or fatigue.

B

um This is real and it can also it can burn out everyone or um someone like I, did I'm doing that as well. I have a dedicated folder in my mail in my inbox and there are alerts in there and if I have like 10 000 unread alerts, uh you can probably imagine the problem I'm not reading any alerts anymore.

B

So it's like really important putting out the focus on what is important to fix and not just have 100 notifications, something is broken, and maybe this could be the root cause, and it's like really important to to review that on the I, wouldn't say on a monthly basis, but maybe quarterly or half a year basis to really figure out. What is the best strategy for alerting now?

B

um How do we trigger alerts? Well, there are certain strategy strategies for that. One of them is kill something in the cluster and see what's going on so like killing a port um actually like removing a deployment. This is something we will be doing in a bit and see how everything like works out.

B

So um in the end, it's really about like finding a use case to trigger an alert and how to like yeah how to simulate the production then and break something to see whether the alerts are being fired um or wait for the next production incident and potentially Define yourself or like a test account as as a load receiver and not the entire team which doesn't know what to do with. It depends how how things are going now.

B

um The alert manager itself is using a different HTTP Port, which can be accessed with our port forwarding on Port 30 000, hopefully yeah and in there we're not seeing or we are seeing some alerts and to be honest, there is a part not ready yet so the note export that doesn't start for some reason and um the Watchdog sent something before, but in the end this is like the interface and you can use certain grouping um to filter things, um do silences and so on.

B

So this is the primary interface for the alert manager and um the thing we want to be do we want to do now in the next minutes? Is um we want to deploy a demo application?

B

uh We want to monitor it, we want to Define an alert for it, and then we delete the deployment because for some reason we decided we want to break it, and the expectation is that the Black Box ping probe, which which we'll be using, will alert US based on that and in the end we should be seeing an alert being generated in the premises alert manager.

B

Now this requires a couple of steps and um yeah after two hours. I totally understand that um that it might be a little too much right now um so I'm trying to keep it as slow and as insightful as possible, because there are many switches between the slides and the um the terminal.

B

um The thing the first thing we want to do is we will be looking into an application called Potato Head and the the pot potato is by it's not by accident. The type of this is intended, um so this is like a demo application which provides us and again that's good, um put paid to head.

B

This is a demo application which helps testing continuous delivery um scenarios, kubernetes deployments and so on. um So this is like the one of the cloud native teamwork applications everyone keeps using and it has like a funny pirate head thing, so you can like break the lag and see that it's still working and so on.

B

What we will be doing now is I've prepared a deployment for Potato Head, which is ready for kubernetes and before we deploy that um I think I'll open it over here yeah. The thing is, um it will create its own namespace for Potato, Cube CTL, then the head, then the the legs and so on. So this is kind. The only thing I modified is I think the load balancer. The note port in this example um I think it was the cluster IP actually and again since I move my mouse over here.

B

I don't want this so I'm having a dual I have I'm having an external Monitor and Mac OS doesn't figure this out um anyways. We want to deploy this one, so it doesn't do any harmful things. It's just a demo deployment and we want to decrease and.

B

It until it's being deployed, hopefully fast enough and after it got deployed, we can see the parts in the cube Halo.

B

I think this was so. This has some parts being deployed that had the entry, Left Right, arms and legs, and we can add port forwarding to that as well, which allows us to access the web interface.

B

So um again, let's do it in here. It doesn't lag that for some reason: okay, let's do it in another terminal background demon.

B

The reason that the deployment took a little time- okay um plot for voting, isn't the best way to do things um but again for for the most uh for workshops, it's better or Works, more reliable for me as well, okay. So this is Potato Head running and everything works.

B

Now we want to configure an alert for it or we want to figure out when it's when it's breaking and one of the things is um I want to be using, not the default like I'm, killing a pod, I'm, removing a deployment and then like creating an alert when the deployment is gone.

B

I was thinking of like this is a web service and I can ping that and with Prometheus I'm able to deploy a black box, Explorer probe, which is also um provided by the cube promises deployment um and this one pings The Potato Head web service. So, like the port, we just exposed through HTTP, this gets pinked um and when it's not available um so like probe success in this query down below this is not successful anymore.

B

We are alerting later on, but in the end we really want to see the probe availability and the prop duration. So like what is the response time to the website, this is something you can also like use for monitoring external services or monitoring the response time to a specific website internally.

B

Now um what we want to do or need to do is this one is the probe we want to deploy um just to open it up again, and the main like difference is: this is the of the kind probe it has certain default labels applied. It uses the namespace monitoring and the specification follows the configuration of a black box exporter probe. So um this is like premises. Specific interval is 60 seconds, the module is using HTTP um and it's expecting like uh two two XX means it expects a 200 response.

B

um The probe is the Black Box exporter, which has been or which is already running in the monitoring namespace at this port and um yeah. It's uh we want to monitor the Potato Head, so this is like um the configuration required by the Black Box exporter.

B

This has been prepared and we want to deploy this into the kubernetes cluster as well, and going back.

A

B

Again, no great Dash, F and the URL again and the probe has been created, so it's Cube CTL, get probes, growing, I, think yeah. So now we have created a black box exporter probe, which pings um the Prometheus The Potato Head deployment, um and in order to monitor that, um because this gets automatically added to promises as well the metrics, we can add some grafana panels so um and now I'm cheating- because this is this- is a query. You can't really type you can copy paste it from the documentation.

B

We are navigating back into kubernetes uh into grafana in the kubernetes dashboard, we're adding a new panel and we are selecting Prometheus.

B

We are copy pasting, the probe success ever for the environment, which is like in this range- and this is the uh we are multiplying it by 100, because it's the percentage. So this is the probe probe available.

B

And check whether yeah we we want to add two different panels, so um we don't have any data yet, but this is potentially going to change hopefully, but for some reason, why is there not? Why isn't there any data yet.

B

B

The other thing which is helpful moving into not the alert merger moving into premises, for example, and see.

B

If there is something returned directly from Prometheus or not, um okay, yeah, it's 100 currently so um Prometheus already knows about the data, but for some reason, uh grafana didn't pick it up yet, um which just shows a way how I debug things or troubleshoot things I'm trying to remove as many dependencies as possible and since I know that the data needs to be in Prometheus I'm, looking at uh the premises itself and not at using grafana as an external um way to verify that.

B

um Okay, it's it's great when things don't work, so I can show some things how I troubleshoot this and now we have it here as well. um The other thing which we can monitor is the prop duration, using the same query, but for milliseconds, which means multiply by a thousand just to have something available and.

B

Yeah, this is like again query panel creation exercise copy paste this one and call it.

B

Head duration in milliseconds, typically, typically, you would like look into the legend and other things I'm, just keeping it as simple as possible to being it. So now we see um the preparation like is 300 milliseconds and potentially it will go up and down a little bit in there in the next time.

B

Depending on what we see, we can also enable like refresh again to immediately see something now we know that our deployment is up and the availability of the probe is still a hundred percent, um and this might be flapping between some milliseconds and everything looks good now um in order to like um just think, oh I kind of copied that uh oh, this is like a duplication of the exercise to worry about this um I think I should have removed that, but maybe Google Slides didn't save that, but yeah in the end, we should be seeing something like this like over here.

B

Ignore this part um now the other thing is. uh We want to get alerts based on when something is wrong, with the query for the Black Box prop so, um and I've copied this over here, the alert rule which we will be creating as promises rule um in a bit. This will be a critical rule. The the kind or the alert name is probe failing, and the expression is to check whether the probe is up.

B

um So this is like.

B

um Why is this question? Eight shouldn't be an eight um check this one, let's open this one up. Okay, just tweet is an eight it's a zero um when when up is zero, which means it's down actually all the probe success is zero. Is it's also similar failure, condition and just to see it in the workshop? It should be for one minute so that the alert is triggering and we're using a simple description so like Potato, Head probe is down and so on now again, I want to deploy this.

B

You can just use the URL and go back to the terminal and say keep CTL create Dash F and the URL.

B

And we have a new Prometheus rule created so from Easy PCS rules, um privilege namespace. This will be using yeah, okay yeah. We have it in the monitoring game, Space, Potato, Head, probe alerts as as a rule and um now we're good to go. So we don't see any alerts yet um for this specific type, but we should be seeing something soon, but first we need to break something, um and this sounds like us: breaking things is fun yeah.

B

Sometimes it's fun to really like test the system and see how it behaves when you break something will come to more like automated breaking things in in a bit. um The easiest way to break the deployment now with the probe is to delete the deployment.

B

So in order to delete the deployment, it's basically reversing the creation or the apply, and you could also like kill something. But it's like really like uh deleting the Potato Head deployment, so everything is gone and the Black Box probe points to something which has been deleted. So in the end there is some somewhere. We had this, and this is now gone.

B

um And we are expecting to see alerts hopefully soon.

B

Probably this is the best time um to take a short break.

B

B

um Let's see it's probably um five, the one minute demo effect.

B

um Maybe maybe maybe we get to see it over here as well?

B

Now the probe is still up I think there is a certain like delay: the problems, every 30 SEC, no, every 60 Seconds ah there we go um and the scraping and also like the alerting so like we see it over here, but we should also be seeing it inside the alert manager.

B

Oh now the alert manager triggers after one minute, so we need another measure, another point of measuring um to really to really trigger that. Maybe I should change the exercise to be no um to being more responsive.

B

We should be saving this dashboard once again, um sometimes I'm really like used to everything, Auto saves and when something doesn't auto save everything is gone, um similar to to a kids, stash and hiding things while you're changing branches and then wondering why it's gone, but actually it isn't gone, but I've seen myself writing code in duplications because of that um different story for a different Workshop. Where's now should be seeing this going down to zero and I'm currently wondering when we will be seeing alerts.

B

ah We have a new alert which is called probe filling, which is awesome and we get to see the information Potato Head progress down um yeah we didn't provide any like more things um to be looking at, and potentially we also like need to. If you have uh like an Ingress controller, you should be.

B

You should be able to like, modify and configure that innovates accessible um to you right now, this code, it doesn't work um because I'm not because there is like no port forwarding, but in the end the alert has been triggered and it proves like the alerting or the alerting rule which we deployed earlier um works and we would have. We are getting now uh continuously alerts for the broken deployment. So this is like one. This is one of the simplest examples I've found for learning, except for like removing some ports and triggering something um yeah.

B

So the thing is this was like a simple alert.

B

um If you think of alerts and notifications and problems in your production environment, this is something which is called or which is found under the umbrella under the term service level, agreement objective and indicator and to dive a little bit more into this in the next minute, um because I think it's really important to to learn and understand next to everything else, which we talked about in the last two hours.

B

um An agreement means that you have an agreement with your customer or your user base, you're providing 99.5 of availability of your services.

B

The objective for your teams and for yourself should be higher like 99.9, because 100 is like a move uh in between, like for a series and devops Engineers, um and the indicator which provides the Insight could be errors latencies and whatnot, and in order to really specify that and make this like accessible um you're, using kind of like error budgets So within the error budget, everything is allowed to fail, um but what triggers a violation of the service level objective, which is potentially fine?

B

But if you have, if you have an agreement with your customers and you go um to 99.1 percent availability over the year, for example, you might be uh paying a penalty or the customer doesn't need to pay that much or something else. So it really depends on the contract in the, in the background with the customers um on what penalty or what measurement has an impact. But this is can get tricky, and this can also like harm your business.

B

In the end, the idea is how to like get confidence like with service level objectives and how? How would you apply several server objectives? um We know about metrics. We have seen that we can create alerts based on the prompt queries which is kind of what we Define as a service level objective. So we are expecting, for example, the probe which we have been using earlier to be available in 99 9.5 percent of the time.

B

The service level objective is much higher than that and in order to get started with service level objectives and also getting to know what is going on in your kubernetes cluster. There is the way of defining the goal, the four code and signals of monitoring, which are also defined in the Google SRE book, which I would totally recommend getting and also getting to know how things can be identified, especially when you're looking at certain web services or Services being discovered.

B

Latency traffic errors and saturation are one of like the key factors of um an incident, and when you see uh that the the rest API of your services runs into a saturation problem or the error rate increases. This is a good indication that something is really broken in your environment doesn't really matter the state. What exactly? But it's an overall signal to really get an idea: hey. We need to be looking at a problem and maybe declare it an incident or maybe there's a hot fix available, and so on.

B

um The other thing I want to bring to your attention around service level objectives themselves. Is you don't need to be learning or copy pasting, the Prometheus rules and everything else around it all the time? There are also like initiatives like open SLO as a specification, generators and management with Sloth and Pura I think I pronounced it correctly, which allows you to specify the service level objective for a given service and then generate the specification and Rule set automatically so that you can just like integrate them with existing kubernetes deployments and promises.

B

For example, um this is worthwhile to dive into asynchronously um for having Ops, dashboards and iterating on everything like. There are many many dashboards, and we didn't touch really touch on everything in this short amount of time in the workshop.

B

So it really means to figure out what is important to you, like CPU usage container memory, um other replica sets in a good shape and so on and use that to create your own dashboards, create your own um overviews, which are really helpful when things are broken or even like monitoring the deployed applications and pro have a have a central overview of that.

B

So it needs customizations and it also needs onboarding, documentation and everything else, because when someone new joins- and you have a multitude of dashboards and something breaks and like yeah I have no idea which which signal should be looking at. It really makes sense to also focus on, for example, on the golden signals and have error rates, for example, as as a documented way to do Incident Management, because the goal should be to immediately see what's important during an incident with a lot of moving targets.

B

A lot of things going on customers calling and you should be fixing it yesterday and other teams are like, depending on your service and like it's like the world, is on fire and you shouldn't be stressed to like figure out hey. Is this metric actually showing what I'm expecting it to be, or maybe should I be looking at a different dashboard or maybe doing my own uh prompt queries?

B

It's also helpful to um when you're doing something manually for the first time like creating a prompt code, query either you create a panel or dashboard or you're documented in your handbook or knowledge base to.

A

Have it available.

B

For future incidents avoid doing things twice and just then document it with regards to customization. If you want to modify Cube Prometheus, you need to use jsonnet. So this is something to learn or look into, but it's really powerful. So I tried it myself, adding more rules and creating my own dashboards within grafana. You can monitor different name spaces. You can add your own applications. You can remove applications if you don't use them and just to provide you an example.

B

That's I'm not sure if you can read it in the Stream, but you can just check it out on the uh on the slides which are available. um This is a way of maybe I can can I zoom.

A

B

um Which I can just like there's a the Json net is oh, it's actually linked in here.

B

This Twitter lab defining a dashboard, adding a template with the data source for Prometheus, which is configured in a different um configuration and then adding a panel which monitors the container memory RSS, and this is the Prometheus uh The Prompt calculator which is actually being executed.

B

um So this is something one can uh one can do um to customize uh keep promises and also create your own custom dashboards. The only thing which then is needed like to modify the default configuration um yeah and we're talking a lot about monitoring and metrics, and but there are like certain things like Beyond metrics, because it's not just seeing what is going on in your kubernetes cluster. There are like you need more context. You need more insights.

B

There are certain things really to keep in mind and I know it's been a long Workshop already, but there are interesting things to come and I really want don't want to rush through this, but to provide you some ideas and we might be looking into one or two exercises um which are pretty Advanced. So maybe you just watch watch me. Do them and practice on later on following the steps but yeah in the end, there are more than metrics, and this is like we have metrics. We have logs and events in a kubernetes cluster.

B

um We are thinking about tracing or traces um additional things like application performance monitoring can also mean profiling. Your applications and seeing um where you can optimize an application so like a function is called a thousand times, but actually it could be executed in parallel or just being. Optimized um error tracking is is also an event type which comes to like thinking of. Is this like observability? Well, it's a definition of different observability data types.

B

What can be um but observability is like the providing the look above and looking into things you probably not aware of yet so think about unknown unknowns like you have a DNS latency in your cluster, which causes the deployments to sometimes fail and the apis don't resolve that often. um So. This is the reason why your customers cannot buy items immediately in a shop and sometimes see a 500 error.

B

um Probably you wouldn't be figuring this out by just using metrics and looking into the deployment states of your applications. Now, how can we like fix that and provide more data by default from uh from a kubernetes deployment? This is potentially something which which needs more work, especially because, when we're coming to logs, there are like so so many decisions which makes it super hard to make a recommendation, and by no means this list is exhaustive or a recommendation for any vendor. It's just like providing some examples of saying: hey.

B

You can either inspect the logs on your ports on the CLI, which works if you're trying to troubleshoot something but oftentimes, it's really needed to get a bigger, bigger, View and collect the logs into a Central Lock management solution, using agents um or using demon sets or sidecar containers, which can be, for example, if you're planning to use a self-managed solution, maybe as an example using elastic and the beta's agents and collecting everything and then doing some sales crews in kibana.

B

There are certain alternatives to that, um for example, with grafana, local and promptail, or just using fluent bit or fluently as a sidecar and having a lot management solution somewhere else which then could be used could be stored in the cloud, for example, as a SAS software as a service offering which of course costs money. Oftentimes some have a free tier, some have not, and the idea is like what locks do you actually need so like?

B

Is it important to have a log retention of one year just to see that all incidents happen and just to cover that some data, or maybe it's just a live tailing which, which could be like retention of one day, to be capable of tracking incidents?

B

um So this is like really really or many things need to be decided.

B

um The other thing I want to bring to attention is tracing um which is like a trace is like a log but with more context somehow. So it's like providing a duration of a start and an end time within a span. um You can enrich it with metadata at your own pace. So like providing more context. This is using the docker engine XY set.

B

um This is like the deployment is using a library version or a dependency, or everything you already have available within the application. Context is something you can add now.

B

The thing is adding this by yourself needs knowledge about code instrumentation, so developers need to learn that the other way of thinking about this is doing it in an automated fashion to like inspecting uh what is possible- and this is more easy, with a virtual machine in Java, rather than like doing it in a C plus plus code environment, but in the end there are certain ways of achieving tracing which can be helpful, especially in a distributed environment um and yet yes, moving on from tracing to open Telemetry, to provide a common or a general specification and framework which works, Beyond vendors to really like collect traces and have them stored in your own back end.

B

So Jaeger, for example, is, can be used for traces. There are ways, or there are plans, or actually um it's already there so metrics you can send metrics to open Telemetry and store them in premises. For example, logs will also be coming in even more event types, so you really like the idea is to define a specification which everyone can use and depend upon um rather than 3000 different formats, um where it's super hard to really like past them or use them or even like correlate metrics with traces with Logs with even more data.

B

You will be this in the past. This was or this is still really complicated. um Speaking of experience um yeah, the idea is really like to combine this um and also to provide client libraries, which makes this easier to do code instrumentation and also provide Auto instrumentation. If you want to learn more um I would recommend like checking this out, um like especially with code instrumentation, focusing on traces in kubernetes means, um and this brings me to kubernetes kind of observability.

B

um The components themselves can send traces, so there are there's like I think this is a feature which is an alpha in in 1.25, where the components can be enabled to send traces um just to get an idea of what is going on within the kubernetes cluster itself, um or use app instrumentation with traces, so instrument code or use Auto instrumentation to send to a central, open, Telemetry, collector, store them in Jaeger or whatever other backend is, is useful and then visualize correlate and alert based on traces, similar to what what is possible, with metrics already the reason or an example for traces.

B

If you imagine the Chrome Dev console where you can see like how long is an hdb request going to the server, then it loads some JavaScript and it loads some images and why? Why is it so slow? And this is like mimicking this- a similar view of within tracing, to get to see how long um certain operations take when an HTTP request is sent from the client to the server to them by SQL or postgresql or whatever back end to a rest?

B

Api began to something else, and even more like with a distributed environment, to really get to see why things are slow or maybe they are like using too many requests in the back end, which can be paralyzed and so on.

B

Now um the thing I want to quickly try with you um in this workshop, and this is something I tried last week for the first time myself, um open Telemetry has a demo which allows you to um deploy it in in a kubernetes cluster. You can also, if you're interested in learning more about this uh use, um the open Telemetry operator, the yoga operator to deploy open, Telemetry and Diego in production in your kubernetes cluster, but the helm chart is really like just to get get an idea of of how things are going.

B

So um the first thing we should be doing is- and it's marked as an exercise again, so it needs our attention to do something.

B

um We need to add the open, Telemetry Helm chart and oh I, already added that.

A

B

And I'm, getting some warnings about my cubeconfig again. I need to fix that um and we want to install just using what uh what is there already so it will deploy open Telemetry with Jager and I think there are also some metrics inside, but the idea is really like to have a web application available a sock. No, it's not sock shop, it's a shop where we can do some shopping and then inspect how the uh the backend requests and so on are going forward.

B

Now, in order to access the demo, we will be using some port forwarding commands, and this is something we can actually like copy paste. I would recommend doing them in a new terminal, a new type depending on how you use it, and this is the demo front end and.

B

We also will be using the Jaeger UI, which is used for visualizing the traces, so the front-end UI is the shop. The demo application, the UI is the tracing front. End analysis, application and low cost I think is a it's a load generator which we probably don't need, but let's see um yeah so going much longer than I actually planned, but it's I think it's worth showing rather than reading. So, um if you do agree, let me know in the chat.

B

um If you are exhausted and say, hey I need a break, it's also totally fine, so you can just watch me, do it and later on I've. Also, the last thing I want I really want to dive. In is chaos engineering at least for a little bit.

B

um But now we want to have a short. We want to have a look into the metric, so um one thing we can quickly do is open the front end, and this is just well just. This is a demo shop, so we can go shopping.

B

We can try to buy something um or we can 10 by 10, add to cart. It didn't break. Yet um there is a logo. Okay, so this is kind of like a shopping application, and this is one of the micro Services demo which works nicely because you have a front end. You have. We have certain back-end communication away available. You have like a workflow from I need to go shopping, I'm clicking something I'm, adding something to the card. It can be slow, sometimes and like I want to do a can.

B

I I can place my order. No, it doesn't work. Okay, um Maybe yeah. The thing is really like having certain oh interesting, I had quite some delay, and this is something you potentially will be seeing within uh that you get UI now. So if we open up the Giga UI, which we will be seeing for the first time now that I think I probably need to zoom in a bit.

B

um So the idea of the year uis you have different services and for this is something that race designer spans can Define by themselves. But for our example is like we have the front end and we want to see the latest traces potentially making it a little smaller again.

B

So we can see. um This is like the load.

A

B

um Are the load generator, for example? So there was one error within that Trace and the cool thing is. We can see um what actually happened in there and the checkout service for some reason had an error and it has logs okay, this generated an error.

B

um Dear one of the ideas is next so like analyzing. What what took the thing so long? We can compare.

B

um Traces, this was just this one and this one and then seeing like this didn't work so well, but these items did oh I can actually zoom in I. Didn't know that with two two fingers on the trackpad um but yeah, the general idea is that you get to see something which, for example, took a lot of time. So this is like a trace which took longer than than others, and it also allows to to determine certain animals um of traces.

B

And potentially.

B

Yeah, we can immediately like model that um yeah guys like just one UI you can be using for traces um but just to get started. It's super helpful foreign.

B

And when you have applications deployed which emit traces to open Telemetry, for example, um you can immediately like store them with the configuration in yoga and have the mobile library or, if you're, not using it as a front-end, you could be using something else which is which might be integrated into your um Cloud monitoring service or using a different vendor or something else.

B

So it's like it's really about like collecting traces, sending them to open Telemetry, storing them in in Jager or storing them in a different backend and have them like favorable from different vendors again now, um as a discussion Point, um why should I be using traces or wire traces in more interesting than logs, like this is my my personal opinion, um because we have distributed microservices, so metrics won't tell you that the red is caching for vote for this one load.

B

Balancer root is slow for some reason or there might be an alternate or something else. So you really need to like look into the execution path of a client request, reaching all the different things um and it might not be visible in the Pod log. Actually, so, even if you're seeing an error in the logs, what is the root cause of the problem? So it's really like getting getting a few on top like observing things and not actively monitoring things, um especially when it has an external impact.

B

So if it's like using an internal deployment or using in some internal things where certain delays are expected and okay because they can be fixed um and telling telling each other we're in a slack Channel and say, hey, please look into that open and an issue, but it if it has like a customer facing X incident.

B

This can be really like a problem, because, if users are seeing one location, C is a 500 error on the SR Service and another location does not how which how to correlate that data and um getting an idea, for example, together with error tracking what is actually going wrong.

B

um The other thing is when you're instrumenting a code with tracing you can provide, for example, software versions or Library versions, which you already know about, or specific dependencies and enrich the traces and the spans with that metadata in order to improve uh the ways how to troubleshoot things, because, as you can see, as you saw in the in a short demo before there is a way to provide logs to this bands. So you can also like enrich the spans with additional login, say: hey.

B

This Trace um was doing an SQL query and it failed because of this or the transaction took longer than expected because of whatever. But this is like going going more into application and distributed architectures rather than kubernetes. Now, so not not dying too deep into that. But it's like it's providing a way to really do like white box providing white box insights um and the other idea is like as a developer.

B

You can make your life easier, because if you need to do later, debugging and try to analyze wise, why things are not working or maybe doing a blind guess and doing a hot fix which gets deployed to reduction, and it doesn't work for some reason and doing it again and again again can lead to develop a burnout or just burnout. So it's like it's one way, seeing a value in metrics, seeing a value in traces and getting immediate feedback, maybe in a staging environment. Now.

B

um What is next for this? um Reviewing your code should be one of the tasks looking into how to instrument a code with open Telemetry, but also finding a way to maybe doing. Auto instrumentation focus on traces first with open Telemetry, because it can The Collector can receive more types, so like metrics, logs and so on, are either in stable or um in in draft olympita and evaluate how tracing can help specifically within an SRE or devops workflow.

B

Be it in runbook, speed and Incident Management documentation, be it in I'm, sending an alert and something is going on now, um traveling back in time to the logs again and are sort of like creating an exercise for it. But I couldn't find any which is easy to show because it needs a deployment of specific log management vendor.

B

um It could be like do something, but we need to run iodine kubernetes, maybe elastic cloud like or final Loki or something um or use a sidecar, which then sends the data to different platforms which can be in Island, sauce or self-managed. So, just to give you some ideas around, like loxio data, doc, dynatrace, blank honeycomb, elastic and open search, graylooper, Final, Loki and so on.

B

So this is not a vendor pitch, but just to like provide you some some ideas: um the vendor websites and my experience of overwhelming when it comes to observability, like everything um figure out what you need and don't feel over all the different terms and types and things. um Sometimes it can be a good idea to go to full vendor login and say: okay, I'm, achieving observability using this method now I'm evaluating things, um but really like follow the evaluation.

B

So use use things step by step and if you say well, it's nice that I get to see uh the all the features from when the XY set, but in the end I really want to build it myself and maintain my own Prometheus instance and open Telemetry and whatnot this. This works perfectly fine um yeah in the end logs is really super nice.

B

In my in my experience, it needs evaluation as well um similar to to all the SAS platforms which are available and there are observability and event types. Don't start don't stop there, because I mentioned continuous profiling before Um this can provide application. Performance insights, generates new data and generates new ideas. So if um a profiling value or profiling data Maybe I want to do a loading with it. Maybe I also want to Define service level objectives on profiling data because I'm treating it as a way similar to tracing and metrics many many ideas.

B

um Which brings me to Performance and scaling ideas um which I'm not really sure about like it's hard to make. A direct recommendation provide an example of you should be doing this for for the fastest and for the most reliable and for the for the cheapest architecture. You want. You want to expect out there, so I can just provide you some pointers where you should be looking at similar to like metrics. How long does do you need the the granularity of five minutes for a metric being stored?

B

It doesn't need to be five years unless you have a lot of money which allows you to store that store metrics that long term and also like it's it's a problem of curing a large data set. So in the end it needs data retention and compactation. It's. This is the term used in Prometheus, so that you, for example, have a granularity of uh one hour for metrics, which are older than a month or a week or something, and everything else is like more more granola.

B

This really depends on the storage requirements on the compliance requirements on certain things in your environment, but probably in the beginning, you shouldn't be caring about it but long term or putting it in production this. This might need evaluation and also like having a look inside similar girls to traces um where data is being sampled, so you don't get to see everything, but only a sample of certain things, and there are certain: like providers saying hey, we are not sampling anything.

B

We are providing you with the raw data you can like want to use, but this is also like a um a calculation on pricing and and other things you should be looking at. um How long do you need traces um similar to like deleting the data you don't need for debugging um for logs? It follows the same um approach with looking into archiving things.

B

um So I don't need the logs from one year ago for a specific port or something which doesn't exist anymore. So, at what time frame do you need the data for debugging an incident or maybe doing like a postmodern root, cause analysis, one month later on um think about like, if you have, or if you're, working in a company which has compliance requirements so to Archive everything on a magnet tape.

B

This is different to like an environment where you're working on your own and just need the data of the of the past week, for example, and uh everything else is like Doc. There is a process document, everything and document Incident Management have run books and so on um for storage in the long term. Just for Prometheus metrics, you can look into Thanos or cortex, which provides High, availability and and long long term, storage um and also a certain way of like doing aggregated queries over multiple nodes, um which probably will be needed.

B

Otherwise, it potentially makes sense to look into certain ways of using a source offering or something else.

B

um The other discussion, Point around resources and scaling, can be like um you've seen that the deployment within the cluster takes additional resources, so figure out how much how much cost the monitoring or the obscibility, tooling and and events and storage itself can add to the operational cost so like using something like open course or infra cost to help estimate the costs and also like for persistent volumes plan ahead, select how much storage for Matrix, traces and logs will you need or how to like, manage that, because in having something in memory, something which, which is, for example, I- think the open Telemetry demo does it with Jager.

B

This works fine. But what? If you restart something it starts again, so you need to think about persistent volumes well, and the other thing is like: maintenance maintenance, like I mentioned before, manage everything yourself or have something an agent or remote ride with Prometheus, send it to a SAS platform which which operates everything and abstracts that away. So this is like really. This is like the uh the business decision later on, but you need to keep that in mind um from a scaling perspective.

B

If you're thinking distributed- and this is probably something you might have seen at kubecon at different other reports and large-scale environments- a multi-cluster approach so like have a Prometheus stack in each cluster or use something else like sending the data out of the cluster to a central Prometheus instance.

B

um So, like also similar to which data can we share, with which type of user so, for example, providing a kubernetes service to customers? Should they which observability data should they be seeing so something like restricted permissions, rbox single sign-on policies, restrictions on the data backend, so that everyone can only see their tenant and not everything else which can be leaking security and sensitive details so like a central obserability with SAS platforms can also be an idea.

B

Promises remote right is one way sending metrics out of your cluster could also be using an agent like datadoc as an agent for sending that to SAS platform, other Wireless agents or sidecars demon sets and whatnot. um This is all something to consider in in large scale environments going beyond the scope of a single kubernetes cluster. This needs discussion.

B

This needs I would recommend um going to events and meetups And, discussing with the wider Community to get an idea um and also like- and this is just an example from a previous Workshop looking into how others are doing it. So, like our SRE teams at gitlab, have documentation, have ground books, have uh dashboards or like how dashboards are being built and enriched with specific URLs for context for alerts.

B

um What are like the key metrics to observe like the error rates um as I go in signal and others and how to do scaling, for example. So my colleague, Andrew nudigate did a talk at qcon I think last year.

B

How we are dealing with metrics at scale on gitlab.com I would recommend watching the talk to get more insights and also follow the links on the slide to really like get an idea, how it's being done in our environment at gitlab, but also getting an idea of like how it's done at Netflix, Uber and other vendors, who have like the problem of scaling, things um yeah and since I'm. Taking all the time I need. Today.

B

um I was thinking of having four hours of Workshop material, potentially I'm, going a little bit too fast, but I'm, hoping still that some folks are following along um and Siam is not uh it's not falling asleep, because I know it's late for you, sorry about that.

B

um The last thing or one of the last things I want to dive into is observability and Chaos engineering, because this also touches a thing. I've been like trying in in my recent talks at kubecon and container days like finding a way to break things, because all um figuring out how your alerts are working is probably hard if you do it manually. So if you want to simulate an incident, um you can try to fail. Dns, you can try to break your network. You can try different things, um but it's oftentimes.

B

It's you really want to Define it in a time frame.

B

You want to filter open services, and then you want to measure the impact and the results the retrospective and see it as a way to improve and iterate your observability and, for example, learning rules and service level objectives um and at a certain point it also makes sense.

B

Not only break production and see how it behaves but, for example, integrated into staging environment deployments from cicd workflows and see how the application behaves so, for example, that you as a developer, get immediate feedback and a merch requester pull request rather than the back or regression hits production, and you are being woken up at 3am in the morning. If you, on the other hand, could have seen it in a merge request and it couldn't couldn't be merged.

B

um So within the cloud native environment with kubernetes, we do have a great way of like there are certain projects which create a chaos framework where you can Define experiments inside and they also provide instrumentation sdks.

B

So, like adding chaos, engineering to your kubernetes cluster again costs some resources, but it's worth trying out one of the things you can try out and this is potentially not safe for work, but I found it pretty funny um to use Cube Doom to kill some ports using VNC attached and so on, which allows you to break pods in production and see how the application is actually behaving and the kubernetes cluster. Now, on the most serious note, one of the toolings or one of the Frameworks I've, been using in the past months, is chaos.

B

Chaos mesh the other one is litmus chaos uh which are cncf projects which allow you to create chaos, experiments a rum, chaos experiments either once or on a continuous basis and fail services, and by Services um I mean like you can kill ports. You can create error rates in the network, so trigger the golden signals and see.

B

If your alerts are behaving correctly, you can slow down or abort certain HTTP requests or even inspect the headers and provide different payloads time is one thing which you can also break, which is kind of funny, if probably remembering the old times when time is off, and it's then in sync again and the application doesn't know how to jump between time and then it's either running in an endless loop and crashing or it's leaking memory.

B

um Dns is like one of my favorites, especially because of my nickname, probably but yeah DNS failing DNS, either failing the resolver of DNS or just providing a different IP address and see what the application is doing.

B

This can be quite interesting to actually do and one of the things I really like about chaos meshes that you can Define a customer resource definition just like the Q A's um jumble format, and it allows you to break something and in this example, it just creates pod chaos for a label selector, which targets a specifically deployed, app and breaks that, and we can see that kubernetes is healing itself. So it's actually like crash loopback off in this current state. But it's like it's one of the ways to to be using chaos. Mesh.

B

The other thing I tried for for kubecon and and some other talks this year was my own application and one one experiment or one problem. I had in my I think it's not 20 years of developer experience I was developing an application which only leaked memory when a DNS was failing, but I couldn't produce a reproduce.

B

The problem on my machine because I didn't know that DNS was failing, so I wrote a small application which is linked in this example over here, um and you don't need to be like you don't need to understand, C, plus plus, um to really like follow along.

B

The idea is really to say: hey um I do have an endless loop, I'm trying to resolve my hostname and when it fails or when the result is empty, I'm handling, an error and otherwise I'm just continuing and just to simulate an application receiving something I'm creating a buffer, which is one megabytes and I'm sleeping one. Second, so potentially everything should be working fine, but there is a programming error inside so this one. It just handles the error which is defined over here, but it doesn't free the buffer in C, plus plus.

B

You need free, a buffer which you are located with new later on with delete. So in this scenario, this application leaks memory only when DNS is failing and one megabytes every second. So um this is a perfect example for like showing um how an application problem can lead to memory leaks and how to really monitor that with Prometheus and um use chaos entry to trigger that failure.

B

So um just to do that quickly, we will be installing- and you can just follow- follow me along with that and practice this later on, because this is specifically the configuration in civil Cloud right now um to being able to exit the containing socket.

B

um I will be just copy pasting. The commands as a quick exercise.

B

Yeah my kubernetes cluster is eating quite some resources. Now did it work.

B

Beautiful and sometimes my terminal Isn't So user, friendly okay, this is deploying chaos mesh now, um while it is deploying something we can also open up uh chaos mesh for a second and you can install it using Helm, charts, um I think there is not true if there's a manually installed available but yeah.

B

The idea is to really like deploy chaos, mesh and being able to um uh create your own experiments and it's a cloud native communion project which I really like, because this ensures that it's um in in a good, Community Foundation and everyone can contribute. So it's open source is Apache licensed and um it doesn't go away that easily now. The other thing is, it needs port forwarding. So the front end is using okay, we have created um thoughts already or they are creating.

B

um It has a front end which allows us to um create some experiments. So the idea now is, in this small demo. Small exercise deploying and deploying application, create some alerting rules and then inject some chaos with DNS and see that the memory is going up.

B

This is actually like my cubecon demo, but we are doing it live now in a workshop or kubecon Europe, because uh kubecon North America is not there yet. So um this is like chaos, Mash and the dashboard invites us to create workflows, schedules, experiments and because I want to create something which runs on a continuous basis, I'm creating a new schedule. So I can just click here and then I have like an interface.

B

What I want to do, but um but the thing which I will love about this thing is or this project is and just upload the Yammer configuration which um we can do in a bit. Oh I'm, moving too fast. First before we can create that experiment. Actually we want to deploy the application, which leads memory, so we will be deploying just to inspect this one. We will be deploying an application multiple times, so this one tries to resolve over 11. Why not love this?

B

One is cncf.io this one gitlab.com and specifically for this Workshop we're also trying cubesimplify.com. So typically, this should be working fine um when we deploy it now so um going over here and oh.

B

Let's handle the portfolio volume later on in a different tab. First off we want to deploy this now and if I call that correctly, we should be. We should parents in the default namespace yeah.

B

So let's we were fast enough. It's creating the things and the other thing we want to quickly deploy, and now you can see that the premises rule will be working as a customer resource definition. This is a Prometheus alert for the container memory, specifically for this container and when it's more than uh 10 megabytes, it will alert. So this is something we also want to deploy. Just like this into Infinity cluster right now and.

B

It should be just fine.

B

Actually, making kubernetes observability Workshop into the kubernetes observability and Chaos engineering Workshop now, but I really like to show this, um because I was a lot of fun building out the demo for my talks with that, so chaos mesh is really awesome and Promises as well. Now um coming back to chaos mesh, we can and I'm doing that in GitHub somewhere we can add port forwarding and going back to the chaos mesh top.

B

Yeah we want to actually.

B

Create this um DNS schedule, experiment and I will explain it in a bit just copy pasting this one, um because I want to create it by Yammer, so I don't want to click around the UI, but I can like create this one, and the idea is to really like schedule a chaos demo of the type DNS chaos, the actions error. So we really want to fail DNS and not randomly do something and the patterns I can match specific domains. So I, don't I can fail everything.

B

Okay, but if I really want to test my service or my specific other things, I can fail specific domains. I can also use the namespace selector, which should be using the namespace default and I can also Define the duration. Now, if I click submit the front end allows me to verify what has been selected. So it's an additional verification of hey.

B

Maybe I want to modify this one or maybe add some namespace selectors or something else because probably I'm affecting too many services right now, but it's totally fine um for our demo case right now, we should be seeing the cube, simplify service, which will be our demo Target, so I'm clicking submit over here and submit for the second one. Everything is ready and I can click submit, and now everything is like running, so you can inspect it again. Currently there are no events being found, but the schedule is creating chaos.

B

Experiments now um and the idea is to really like inspect um the logs for the um going back over here.

B

The deployment for the DNS leaker service, Cube simplifier, so we're getting host results and at some point we shouldn't be getting there and, uh as you can see, there are four IP addresses actually attached to cubesimplify.com um and you can ignore the port. This is just needed by the application itself.

B

I think there should be at some point.

B

We should be having some.

A

B

A

B

This is like the schedule, which is running um sorry to call webhook. Oh interesting,.

B

Potential, it's again the demo Gods, which are failing me.

B

B

Let's start again.

B

It should create a new object, which then runs as experiment, which is injected.

B

um For some reason it throws an error. Okay, hmm let's see ah yeah. This is don't mind. The errors in the web interface I was actually like planning for this. So um now DNS is like injecting a failure, and this is not authoritative, so it's kind of intercepting the DNS traffic for the application, and now since this has been running, uh we should also be able to see that the memory is going up. So we can query or we can use Prometheus, um which should be over here to um create a container memory. Rss.

B

Now this oops now this was the wrong copy. Paste, sometimes copy paste, doesn't work in this browser.

B

um Okay, we want to execute this and we want to see a graph, oh yeah, and we can see that the memory is going up at the time we were starting with our chaos, experiments so um the highest one. Okay, this is the gitlab one. This is the cncf one. This is o11y. This is oh Cube. Simplifier currently consumes um a little bit of memory, so it's 30 megabytes.

B

If this stays over 10 megabytes of uh of memory usage to many metrics today, sorry, um we we could. Potentially we should be seeing an alert now um in order to be checking. This I will be closing some of the many tabs I've opened and checking back into our alert manager.

B

Which I think yeah it's in the namespace default and we, for example, can click here to have an applied filter for it and, for example, if I want to see specifically the alerts for for the host resolution for cube. Simplify can check for that. So, um in the end, this triggers a memory leak for the deployed applications, but only when DNS resolution is failing and the reason why it fails currently is because we're using chaos mesh to inject the DNS failure.

B

I hope this isn't like too much mind-blowing right now, but this this is one of the ways I found it really reliable to create some chaos inside the kubernetes cluster and see which alerts are actually being triggered if the applications behave, the right way um and also like have um have a way to being um sensitive on these deployment errors before they're, actually hitting production and affecting other teams and creating many alerts and other things, but instead having a way to really have a defined way of testing things.

B

I wouldn't recommend like what is next doing all these things, all the time so potentially analyzing. What caused which problems, for course, by chaos. Engineering, maybe the alert counts, were too much again and you need grouping or additional context in alerts or even like we find the focus and dashboards, because the alerts didn't provide as much context as needed or even reduce the amount of visible data um yeah.

B

The other thing is like potentially and I'm not sure how this I'm not sure if this like is applicable already, but using chaos, engineering for tests uh for traces again like to get an idea of hey the HTTP request is so slow or the client side request is too slow use chaos, experiments like for stress, testing, for CPU and memory stress testing, as well as low HTTP requests, um I actually made it work a while ago.

B

This is also available for you to test if you want to, and just to conclude with some ideas around correlating metrics with traces. This is called accent plus if you link metrics with traces or even being able to create metrics from traces using open Telemetry, which is called aggregated Trace metrics, also something to keep in mind to keep an eye out for, especially with regards to current developments. How open Telemetry will be advancing in the future and so on and similar to like?

B

How does it play well with chaos, engineering and Chaos mesh and other tools um yeah, and just to just let sum everything up or give you some additional ideas, because um life is a learning exercise, a Learning Adventure um like what could be like future observability with regards to qinators and and other tooling, like you, probably have heard about ebpf, um which is a different way of collecting metrics and collecting values on the Kernel level of the uh of the operating system.

B

Basically so like not having not being required to instrument your code, but instead collect um specific syscalls and events on the Kernel level also to provide some observability and security so something around like psyllium, Falco, Parker and Tracy for different use cases, especially for the reason why.

B

um How does it like fit into kubernetes observability with Prometheus metrics and everything we've learned thus far?

B

um It can be a way and I'm seeing that that pattern, uh with quite a few tools and startups um to provide Auto instrumentation for specific languages, because code instrumentation, is something you need to learn um and it might be easier to use Auto instrumentation with a new technology. Nobody understands yet, but it's a way to get uh to get two results pretty fast, and then we can like see what what other things ebpf is capable of doing.

B

um This is something I didn't really prepare for the workshop today, but I wanted to give you an Outlook or provide you some insights into maybe trying psyllium, tetragon and also pixie for this kind of observability within kubernetes, more of an idea of a developers perspective or seeing it as a way to to also apply security policies to prevent things from happening within kubernetes. But it's like again. This goes Way Beyond.

B

What I want to want to tell you today, but I would really encourage you to watch the recordings from how everyone can contribute Cafe, Meetup um Ultra in future sessions of the Meetup and try it out yourself um yeah, as you can see, to do. Advanced topic um didn't really figure it out yet um the other thing around um service level, objectives and Prometheus. Maybe you want to look into quality gates with Captain uh which can be used in that regard, to prevent certain things from happening when the search level objectives are failing.

B

um Also, a thing I would highly recommend trying out by ourselves and last but not least, obviously securely should be on the first front, but I need to I really want to mention it like q, a security monitoring and getting at it here needs to happen before the deployment, so like think about cicd security, sending scanning cluster image, scanning qvino policies, open policy, agent and so on, um and the other thing is like think about: Prometheus hardening with TLS and basic of with the cluster communication using ntls and so on, um where I totally recommend getting a book.

B

Hacking kubernetes from Andrew, Marlin and Michael housenblast, specifically also with like the security defaults, may need improvements in a default kubernetes deployment. So, for example, you could be using observability patterns and Chaos engineering experiments um for pen testing in your kubernetes cluster and apply those things um again. Many many things to read and learn, but it's now it's your adventure with all the things held up today and probably a scene today.

B

um The last thing I want to keep you keep at. The top of your head is think about how you can instrument your code and make your life easier, either with Prometheus or with open Telemetry or with everything else. We might be seeing future workshops in that regard.

B

Yeah many many more resources, um I do have uh I've created a knowledge base which I called o11 wildlife observability.love potentially check this one out, and if you want to learn about more of my metronome findings, around kubernetes Cloud native observability subscribe to my newsletter Ops indef.news, which I currently read on a monthly basis.

B

Yeah, just let me know how things are going. This is now like thanks for your attention, and hopefully there are no questions. um No just kidding um thanks Siam, for the possibility to do the workshop today. So I really like telling my stories um engaging with you and- and hopefully you learn a lot today.

B

um Let me know how things go and I'm happy to help and yeah I'm talking too much thanks.

A

No, that was amazing, Workshop uh Michael, uh so actually, when you were starting uh just to let you all know my class, like you know, what's the regular time of the workshops uh he said, I have prepared, for you know one to two hours, but it might go a little bit up. I said yeah, it's fine uh as long as the the content is dope and you are going with the flow uh people will be here to stay, and people are here.

A

um They are learning uh till the end. So it's amazing and we are like close to three and a half hours and Michael has been speaking continuously, so we have given him just a bit of rest for now. So thank you. Michael for this amazing presentation. First uh great exercises, uh great demos that you showcased uh great live kind of debugging whenever it was necessary and you fixed. The issue showed us. uh The alerts and seeing those coming in uh at that point of time is is very good.

A

I think people people should take a lot of things from this particular Workshop, like you know, playing with Prometheus Prometheus queries, grafanas, grafana dashboards, setting up that writing your own grafana dashboards. Alerting! Writing your alerts. Black Box exporter instrumenting the application: how to get those alerts in Prometheus how to do chaos, engineering and alerts so I think there are many many parts to this particular Workshop that you can actually go back, break it down and then revisit and try to do it at your own pace, so it can be overwhelming at first.

A

If you have heard like everything you know there might be a lot of new terminologies I, always say for kubernetes as well. When you learn kubernetes, there's a lot of new terminology that you have to like you know, take take hold of it's it's not easy um same as in in mathematics. When you did trigonometry it, then there's a lot of terminologies that that you have to you know uh grasp. So all those are in the similar lines and for this Workshop as well go back revise this Workshop. Do it in parts?

A

Do the complete demos uh you can use any kubernetes cluster or Co kubernetes, whatever you feel comfortable with, and you know, install uh Cube straight metrics install the metric server uh see what all stuff you can do see how you can instrument like uh in the previous slide. uh Michael mentioned. You can instrument your own application.

A

um You know there are many aspects which are covered and I think this. This uh presentation will be available um to the people right Michael. This is not going soon away.

B

No, no, this is public information and um the thing I will be doing after the workshop when, like uh the Youtube video is available and we combine all the resources, um I will be also adding them as a resource to the gitlab developer. Evangelism handbook so um just to like um can I talk today anymore, but um we do have like our own handbook and one of the things and this needs projects I've been doing kind of a kubernetes monitoring Workshop in the past.

B

So um all the workshops and everything I'm doing of our team is doing is like linked um in this section and it's somewhere at the bottom. There is like yeah that was the first iteration of this Workshop. So um um when everything is locked done, I will be adding it here and also making it available on.

B

You will be seeing it also here then, where I'm like sharing what I'm talking about, and you might see some overlaps with my talks, and this is intended because I love to repurpose everything.

A

Can you can you share that link that gitler projects link so that I can share that with with people as well yeah.

B

I I assume you've shared the slides already with folks. Yes, so yes, yeah I will I will share them with you in the chat, so you can copy it over to uh YouTube and also- and it's done this one just like my website- has all my talks and and everything you probably want to know. um If you want to see my background in a little more other lights. This is my workspace.

B

um If you're planning on on getting some some Lego for your background as well, yeah.

A

As far as social media, yep I saw one comment: uh that's a very nice Lego. In your background, so there was one comment uh that came in.

B

Yeah the thing is, or just a short story around the Millennium Falcon. This was a gift from uh the monitoring Community while ago, so I did a lot of community support and engagement and being a maintainer, and some friends granted me this that as a gift um and I was like oh wow, so I I tried to make it really like uh pulling it the best way and always having, in my background as a great memory and also yeah, it's a nice break and have a conversation as well.

A

Nice, so everyone I hope you really enjoyed this Workshop I enjoyed this Workshop I was here the whole time uh and I did many of the exercises as well. uh So I hope you were able to do uh at least 50 60 of the exercises and you'll go back and do the rest of it. uh You follow uh Michael on Twitter, he's very active over there, sharing all his learnings all the new new cool tools that he found.

A

uh You know he find in the cncf ecosystem, observability ecosystem, so make sure to follow him also tag all your learnings, like all the screenshots grafana dashboards with him, so that he can, you know, see those what all the stuff you have uh done uh during the workshop. What are learnings you have taken away from this particular workshops. So do that tag tag me as well, so that we both can retweet that, like that- and you know uh we all- can share uh good stuff. So yeah here is uh his um Twitter handle as well.

A

B

Yeah, the other thing I want to mention is I will be uh in Detroit at kubecon. So if you are traveling to to Detroit, I would love to meet in person um other than that um I. Think I will be mostly in Germany this time. But let me know if there is like a meet up in my area or maybe you're around, um so we can do something together in person or just catch up for a follow-up, coffee chat or, let me know on Twitter publicly or even on LinkedIn yep.

A

Michael is very approachable, so I think you can. You can ask him questions, uh don't buzz him too much. You can ask him.

B

I would say um ping me publicly on Twitter.

A

B

Dms are like chaos, um so it's chaos, engineering, my DMs um I I, prefer working in public. So um if you, if you are able to ask in public, do it in public and we can Loop in more folks to contribute.

A

Awesome, so thank you, Michael. If you are at kubecon, uh you know feel free to reach out Michael and if you are at hash account I'll be there at hashicon and also Dubai GI t e x. uh Both the events I'll be there. uh Unfortunately, I won't be there at kubecon, because I will be at two events uh in October already. So third one I won't be able to bake it to the third one uh but yeah uh it will.

A

It will be nice meeting Michael at some of the other events coming in future uh and uh yeah. Thank you. So much for this Workshop make sure to follow. Cube, simplify all the next set of workshops.

A

um I, don't know when the next one is I think there is one in October and after that, I haven't planned to be honest, uh but yeah streaming keeps on continuous. The videos will keep on keep coming on and the workshops are gold. So you can watch all the gifts and private workshops, and you know just if you want to learn something. The resources are available. People are reachable, so do not let the resources be the barriers to your learnings.

A

If you want to learn something which is not there at all reach out and we'll try to fix that, and thank you again Michael for you know, spending great amount of time, great amount of knowledge and this amazing session uh with all of us and thank you all who stay till the end uh with that I'll leave you all to the rest of today's night, so bye, everyone.

B

Bye thanks for having me.