Cloud Native Computing Foundation PrometheusDay NA 2022, 1 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: ⚡Lightning Talk: Leveraging Prometheus Monitoring to Maximize Business Metrics with Iter8 - Alan Cha

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

⚡Lightning Talk: Leveraging Prometheus Monitoring to Maximize Business Metrics with Iter8 - Alan Cha, IBM

Prometheus has grown to become one of the most popular tools for collecting and aggregating metrics for Kubernetes applications. These days, you’ll be hardpressed to find a Kubernetes application that is not using Prometheus. In this talk, we will present Iter8 (https://iter8.tools), an open source tool that takes advantage of Prometheus’ ubiquity in order to perform SLO validation and A/B tests for your Kubernetes apps and ML models. Iter8 conducts metrics-driven experiments to ensure that your apps and models are performant and maximize business value. During an experiment, Iter8 will query a metrics source, such as Prometheus, and use those metrics to determine which versions of your apps and ML models work the best. Best of all, you can set up these experiments in a simple and declarative way. We also want to show some of Iter8’s latest features, such as autoX, custom metrics, and multi-loop experiments, which allow Iter8 to automatically detect new versions and launch experiments, use metrics from any database or Kubernetes resource, and conduct experiments over a period of time, respectively.

A

Hello, my name is Alan cha and I'm. A software engineer at I've, been research and today I'll be telling you about iterate and how it can benefit you as a Prometheus user. So before anything, we need to answer the question. What is iterate iterate is an open source, kubernetes experimentation platform.

A

It already makes it easy to ensure that your community's apps and animal models form well and maximize business value. So what are some of the things that iterate can do? Well, here are some of the use cases. Itera can conduct performance tests and SLO validation for your HTTP and grpc surfaces and for your performance tests. You can use custom metrics from external databases such as from Prometheus.

A

It can also perform a b and ABN testing and it can generate load for your tests and much much more, which will go into more detail later. So how does iterate do all that? Well, iterate introduces the notion of an experiment which is a list of configurable tasks that are executed in this specific sequence.

A

For example, let's take a look at this diagram. Perhaps you have multiple versions of an application and you would like to know which one is best. You can create an iterate experiment composed of three tasks: collect metrics, assess versions, promote learner to help you figure that out when you.

B

A

This experiment, it will collect my tricks, perhaps from Prometheus and using those metrics it'll assess the different versions by comparing them to your slos and, lastly, it'll recommend you the best version.

A

This is a very simple experiment, so keep in mind that iterate can do a lot more. For example, we also have the concept of a single Loop and a multi-loop experiment.

A

A single Loop experiment will only run the tasks once, whereas In a multi-loop experiment, the experiment will run the tasks periodically. Perhaps you want to run your sermon over a long period of time. Maybe you want to see how one version of your application handles live traffic, where the amount of traffic depends on the time of day or even the course of the week. This will be one reason why you would want to run a multi-loop experiment recall that an experiment is composed of a number of tasks.

A

So, let's take a look at what kind of tasks are currently available.

A

B

Have the ready task.

A

Which checks, if your deployments or services are whichever commit these objects you're using, are ready. We have the grpc and HTTP task which generate requests for grpc and HTTP services and also collect latency in early metrics.

A

We also have the assess task, which allows you to Define, slos and checks. If your app versions satisfy them, we got the slack task which allows you to send a message to a slide. Channel.

A

We have the GitHub task which allows you to trigger a GitHub workflow and lastly, we have the custom metrics task, which fetches metrics from a database or from a breast, API and I think this is a task that will be most interesting to all of you who are attending Prometheus day, because many of you already have your own way of collecting metrics and are using Prometheus to store, said metrics and the custom metrics task will allow iterate to plug into all that pre-existing data to help you ensure that your apps and demo models are performing well, you can use all these tasks in order to compose an experiment that best fits your use case.

A

For example, you can create an experiment where you first ensure all of your resources are running using the ready task, then using the custom metrics task, you can fetch metrics for information database and after assessing your different versions using the SS task, you can trigger your GitHub workflow, which will start some process for your git Ops Project, based on the results of experiment in this manner. Iterate experiments are declarable, easy to use and don't require any additional programming.

A

You just need to list the tasks you want to use and provide whatever additional arguments are needed for those tasks. So let's try out iterate.

A

First, we can do a basic experiment where we do load testing on a kubernetes ECB service and validate some slos. To summarize, what we will do, we'll have some ECB service running.

A

We will check if the app is ready, then generate HTTP requests and collect latency and error later metrics and then finally, we will validate those metrics against slos. So to start, I will need to create some sample HTTP service in this case I, will create the HTTP bin service.

A

Then all I need to do is launch the experiment.

A

This experiment has three tasks: ready, HTTP and assess. The ready task will ensure that the HTV bin and service are ready within 60 Seconds.

A

The HTTP task will generate load to the app in this case via the link and collect latency and error related metrics. The assess task will ensure that the mean latency less than 50 milliseconds in the error count is zero and, lastly, setting the runner to job means that this is a single Loop experiment where the task will run just once.

A

I will also add a node download flag, and this is to ensure that I do not need to reinstall iterate resources.

A

Foreign for a bit we can get the report.

A

As you can see, the experiment has completed, there are three tasks and we completed all of them. The app has met all of the slos and we can also see additional metrics that were collected as part of the experiment. We can also produce a nice HTML version of this report.

A

And this is just the same data.

A

Now, let's try and every experiment where we use the custom metrics tasks to query metrics from Prometheus.

A

This tutorial describes how you can utilize metrics from istio, specifically metrics, that issue generates and stores in Prometheus via is add-on.

A

We can see here that in this experiment, instead of generating load and collecting latency and interrelated metrics we're getting metrics directly from the Prometheus database, this will also be a multi-loop experiment where the task will run periodically to set up the experiment. I will need to create another sample, ACP Service, as well as generate some load, so istio can start capturing metrics.

A

Now we will generate some load okay, so all that's left is to launch the experiment.

A

A

We have two tasks: custom, metrics and assess the custom. Metrics task also requires you to define the metrics that you would like to capture as well as how to do so. This is done by providing a link to a configuration file.

A

We provided a template for getting ACL metrics from Prometheus and we plan on providing more templates in the future, but it is simple to create your own configuration file. It is simply a list of ECP requests. The assess task will ensure that the error rate is zero and the mean latency is less than 100 milliseconds.

A

The runner is set to Cron job, which means that this will be a multi-loop experiment, and in addition to that, we need to provide a context schedule. This means the experiment will run the tasks periodically every minute and, as I mentioned before, I will add a node download flag.

A

We can see the experiment report here and, as you can see, we had two tasks and they were completed and we were able to meet the slos.

A

So to summarize, we saw how easy it is to get started with performance, testing and SLO validation with iterate. We learned about tasks and how they can be composed. In order to create an experiment. We got to see how iterate can also utilize metrics from an external database, such as from Prometheus.

A

We have some fun stuff in the works such as an ABN SDK, as well as the way to automatically launch internet experiments after detecting version changes, I hope you enjoyed this talk and I hope you enjoy the rest of Prometheus day. Thank you.

B

Hello: everyone thanks for having me this is engineer working for cultural park today, I would like to share with you my experience in preparing for and taking the all-new Prometheus certified associate certification exam.

B

The provision certified associate certification, as defined by the cncf, is a pre-professional certification designed for an engineer or application developer, with a special interest in observability and monitoring the main areas the exam covers are the following: observability Concepts, Prometheus fundamentals from ql, instrumentation and exporters, a learning and dashboarding.

B

Why should I prepare and take this exam? Well, there are good reasons for that. I think it strengthens the foundation and it motivates to delve deeper into learning more about the cloud native monitoring ecosystem, besides getting a level of credibility under the belt.

B

So this is where it all started. The opening remark: skinout announcements, Con Europe 2022.

B

Following my interest, registration I was invited to take the online Proctor multiple choice, privileges certified associate data exam in July, now figure out where to start could be daunting along a proper curriculum eliminates most of the trepidation that comes along with learning a new skill as it helps to not miss any of the Core Concepts, as I prepared for the exam I created a publicly accessible repository on GitHub, where I have linked curated resources to each of the exams topics, mainly from the official throwdowns documentation of Prometheus itself.

B

In addition to some excerpts of the Prometheus upper Running book by Brian Brazil, before starting with Prometheus, it's worth getting more acquainted with the absolutely Concepts and terminology such as metrics service, Discovery and the service level objective, to name a few Prometheus Beyond doubts is a major pillar of cloud native stacks. This is certainly not Prometheus in two minutes, but it's more of an overview of the Prometheus ecosystem.

B

What capabilities it has and what to use each for the Primitive server components are the retrieval component that scrapes metrics from endpoints offered by the discovered targets, the time series database where the metrics data gets stored and the Prometheus hdb server. That offers a rest endpoint along other systems to query Matrix data regarding instrumentation, primitive, scrapes, metrics from estimated jobs, either directly or via an intermediary push Gateway for short-lived jobs to collect application, specific metrics, a client Library needs to be added to an instrument: the code in Paris programming languages, to instruments, a third-party system.

B

An exporter can be installed on the hot server or next to an application. Another aspect is a learning. Prometheus alert manager is capable of routing alerts to different channels such as email or Slack, for the visualization grafana can be used additionally Prometheus remote right, which is basically sending data to other systems like a long-term storage, and this is where Thomas, for instance, comes into the picture.

B

If you haven't been working with practice for some time now, I would recommend taking any getting started course or you can also watch Prometheus introduction at kubecon, Europe 2020 by Julius balls, the co-creator of Prometheus as per product URL. It amounts to more than one-fourth of the exam questions, hence the increased importance of knowing how to use it properly, also beyond the exam. Here's an example of a query to get error ratio of an HTTP server for the last 10 minutes, a shout out to prompt labs for the bronchial cheat sheet.

B

Needless to say, it is not to be memorized, but it does help to get more familiar with the query structure and, last but not least, a learning and dashboarding open source tools like grafana bring live to data through intuitive visualizations of different metrics being monitored.

B

Grafana makes it feasible to see the current state or collected metrics as a graphical representation and to create custom dashboards.

B

Monitoring is often paired with alerting tools. The identify Engineers when metrics have exceeded or Fallen below certain thresholds specified in accordance with the service level objectives.

B

A tool like Prometheus alert manager can be used for this purpose and also it gives ways to filter others before sending notifications. So what happens after taking the exam?

B

There is also the exam will be emailed to them in some taker within 24 hours after taking the exam, and if passed, a certification at a badge will be issued or calls that I received the results for about five weeks after taking the exam as to other one. That is no longer the case, as the exam was made generally available last month.

B

I hope you find the information and the resources useful and I wish you all a great and thoughtful coupon. Thank you.