Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022, 1 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Metrics as a First-Class Citizen in the E2E Testing Landscape - Matej Gera & Jéssica Lins, Red Hat

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Metrics as a First-Class Citizen in the E2E Testing Landscape - Matej Gera & Jéssica Lins, Red Hat

Exposing metrics through the metrics HTTP endpoint is the de facto standard when it comes to exporting application metrics in the cloud native universe. But did you know this well-known pattern can be used for more than just collecting metrics? This talk will show how instrumenting applications can be leveraged to improve your experience with end-to-end (E2E) testing. If your application is already instrumented for collecting metrics, why not take it a step further and use the exposed data to bring your E2E testing to the next level? This has the enormous benefits of giving you detailed insight into the state of the tested application and allows for more complex testing scenarios, which are not possible without observing the behavior of an application from the inside. The talk will also describe how this framework has been employed in several open-source projects, including Thanos and Observatorium, and what benefits have been gained from the change.

A

B

A

uh So we are very happy to be here. Hope you had a good time at kubicon um today we are here to talk to you about metrics and entry and testing how those things can be combined together, but first, let's present ourselves.

A

So my name is jessica. I'm a software engineer at red hat, I'm also a contributor and approver for portuguese content at the cncf glossary, I'm one of the maintainers of the observatorium project. This is our observability backhand. I will see more about it today and I'm currently working with go. I'm also interested in distributed systems and also observability.

C

So, hey everyone, I'm mate, I'm also working with jessica at the monitoring team at red hat, I'm a contributor and a triage member at the at the thanos project, as you can also get from my cool tunnel socks that I got yesterday from the cncf store and I'm also maintained observatorium uh together with jessica, so yeah, the usual interested in distributed systems. Observability and all of those things.

A

Cool, so let's go through our agenda today, so the idea today is that we want to understand what are metrics. Why are they important? Why do we need them? Then we will jump into understanding, metrics and then trend testing as a concept so like how this differ from the conventional engine testing. What are the advantages? Which patterns can we use and we also go through different types of tests?

A

Then we will show how this can be used in real world applications and at the end, we'll have a demo. So we can understand how all of this can be used in practice.

A

But first I want to ask everyone who, in this room, has your application instrumented with metrics cool nice nice? I see some hands. I first want to share my personal story. How I got introduced to metrics on the first place was not very funny but yeah. I pushed a new service to production.

A

The code worked, the prs were review. Everything was okay in my head at least I had the code there. The tests were passing and then the service went. Live people started to use it. I saw some logs okay, I got some requests, but I had no idea how everything was going right. I had no idea how my application was performing.

A

It was a new service so like. How am I supposed to do that? Everything is fine right like how many requests do I get like. I had no dashboards like how many requests per second, how many errors um and what about resources right, like I don't know if I have enough cpu or memory or actually, if I had more, if I provision more so that's when I noticed that metrics are really important.

A

So why do you need metrics right, so we need metrics so that we can get performance insights about our application.

A

So, for example, you can have your application instrumented with different types of metrics like number of http requests, duration of requests, number of errors and metrics help with making your application more resilient and gives you a better understanding of how your application is performing.

A

So why metrics also because you can answer questions like how much load your application get, how many errors? uh How slow is it right? So you can measure your latency.

A

What about resources is important to understand how much cpu and memory application is consuming. So, as I mentioned before, yeah you can see if you need to provide more or less resources and configure your application properly.

A

So how do you instrument? How do you do this right? I think um in the last talk we also saw the importance of metrics and it was, I think, at the end. This was also an example of how you can set up your application, but ideally you instrumental application with metrics.

A

You expose a slash, metrics, http endpoint, and then there use a client library that then will help with your code instrumentation and once you have this endpoint with your metrics exposed, then um you have a server that then will scrape those metrics and possibly could store them somewhere. So, for example, you can use prometheus for that right.

A

Cool like we, we saw matches, are important, but what all of this has to do with testing.

C

Yeah, so jessica will test through, like the typical use of metrics that we often think about, and we understand that this is what the promise of observability brings right as engineers. We want to be able to infer information about the internal state of our applications from the signals we receive, and this is of course, extremely valuable for our day-to-day work, as jessica was explaining when it comes to monitoring the state of our infrastructure, debugging or tweaking performance of our applications.

C

uh Inherently all of these scenarios and use cases have one aspect in common: uh we're talking about some sort of live application right, we're observing an existing system in motion, whether we equate this to a sandbox environment or the actual production deployment.

C

However, when it comes to observability signals, we should not be limiting ourselves only to established systems, although this is the primary use case that often stands at the forefront of the discussions. It is not the only one how about monitoring our applications before they're actually running in the live system. Could it bring us benefits as well and as we'll propose in this talk it can we don't often look at it this way, but tests are sort of systems as well.

C

If we think about it, running a test equates to a creation of a system of components. This can be like functions, mock services or actual applications, and subsequently the system is evaluated and asserted against the set of conditions. Although these systems are only ephemeral, it does not preclude them from benefiting from observability patterns.

C

So the idea we put forward in our talk is simple and powerful. Let's take what we're used to from running our systems and apply it in the context of end-to-end testing. We want to increase awareness of metrics as a part of the testing landscape and to show that as engineers, we can reap immense benefits from incorporating metrics into our testing strategies and what is even better. We already get this for free with instrumentation for our live systems. Right three things are the best things in life. We do not.

C

We do not need to put more upfront investment into our applications. No code changes are necessary if you already instrumented applications. The testing patterns we'll talk about today are automatically applicable. So why not get more value in your testing experience for free? Let's talk about this in more concrete terms.

C

Having metrics makes it possible for us to make more detailed test assertions. Think of a more typical end-to-end framework. You spin up a couple of binaries or containers, and then you observe them and assert externally.

C

For example, if you wrote your tests with a bash script, you might base your assertion on the application's exit code, so exit code, zero means success and other codes mean failure.

C

However, just asserting based on how applications are behaving from outside might not always be enough sure the application exited successfully, but can we be sure of all of our assumptions about how the application behaved during the test run?

C

In contrast, thanks to metrics, we can externalize the internal internal behavior of the application and assert on this externalized information.

C

Instead of checking the exit code, we can check, for example, whether we always responded with http status code 200, whether we reached an expected number of responses or whether the expected amount of data was processed and so on.

C

On top of that, external data does not only greatly improve our capability to assert. We also gain better control over test scenarios. Thanks to metrics, we can check our preconditions more easily, maybe our application for an issued on an event that will occur in some uncertain time in the future. It is not trivial to check such preconditions without knowing the internal state of the application.

C

If you have metrics, this gets easier. For example, before proceeding with testing, we might have to wait on an application to be connected to a certain number of replicas of another service. If you have matrix telling us this value, it's trivial to leverage this for our precondition check.

C

This also allows us to fail fast if the precondition is not fulfilled, or on the other hand, we don't have to fail fast, but we can wait until the metric reaches a certain value before we continue with our test or if, after certain time limits, the the limit has not been reached, we can just terminate the test at that point and even better for more advanced cases. We can, for example, restart our applications. We can retry certain operations all together if our metrics are not still not showing the expected values.

C

So when we start to piece all of these patterns together, we can see that we can create more complex scenarios easily, and these do not have to be related only to internal state of a single application, but also to links between the applications based on observed metrics. We can run one test with service started other tests with a service stop and compare the results. We can test how applications behave when a dependency goes offline, for example, and then how it recovers when the dependency comes back online.

C

And lastly, we benefit also from collecting various extra data points about our tests that we are again getting for free. In this case, this can be related to various metadata, such as such as our applications version, or we can gain insights into applications performance characteristics since it's typical to collect, for example, data about about latency of our request. These metrics can give us some idea about the performance, and here we're moving a bit into the benchmarking territory, about which we'll also talk in a bit later now before we move on to the practical application.

C

Jessica will talk us through some of the test types to which we can apply these patterns.

A

Cool, so let's go now over different types of tests like first, I want to talk about the benchmark test. This is a very useful test to test performance, so it's the focus of this type of test. It helps you identify bottlenecks on your application.

A

For example, if you have any, if you're introducing a new api to your architecture, it can be useful to then test how slow this new api is right. So maybe you can create a benchmark for it and the way to do it is that in general, you have a function that is as executed multiple times and then for each time you test.

A

If the output of this function is compared to a certain standard um yeah- and I think it's nice, if you are using already gold link, you have this from the testing package by default, so it's it helps you to like just easily create benchmarks, but the idea also, is that you don't optimize prematurely right. So it's not nice. If you try to run benchmarks for every piece of code you have like. So it's something to just to keep in mind.

A

um I think the other type of test this one- I don't know if it's very well known, at least for me before it wasn't, but this would be the interactive test, so this type of test. The idea here is that you pause your application at a certain point and then you kind of play with the scenario in progress so like.

A

If you have a lot of services, that's maybe a good idea to then see how those services interact together um and today I think it will get more like we will understand better once we show a practical application with it. Our demo would be also using exactly this kind of test, so I think it it's a bit easier to understand um cool, so we saw benchmark test interactive test. I think those other type of tests.

A

They are also very well known, which is the unit test, integration test and entrance test, but maybe, let's just refresh our minds a little bit so the unit test. The idea is that you test each unit of your application. So, like you in general, you have a code base and you have a lot of functions and then you can create one unit test for each function of your application. So it's really testing a unit.

A

It's also a less expensive type of type of test. So then that's why you can also have them a lot right, because also you have a lot of units. Probably you also have integration tests and the integration test. The idea is that you, you test how those units work well together. So let's say: if you have a function, a and this function, a has a function b as a dependency would be very nice to understand if those two functions are working well together and that would be the integration test, the entry and test.

A

On the other hand, this would be really like spinning up all the components of your application. So all of your containers, so it's a more expensive type of test um and I think also they don't. Ideally they won't happen very a lot. So you won't have a lot of entrant tests on your application because yeah like it's, it's not not as the unit test, for example, and again today, we'll focus exactly on this type of test on the entrant, and another thing I want to mention is also that you can have so.

A

I mentioned the benchmark class the interactive test, but you can also have those tests combined with those three tests here. So, for example, we can have an entrant test that is interactive or you can have a unit test that is also benchmark and so on, and today yeah, we see also an interactive test and also entrant tests. So we see how those how they can be combined together cool. So let's then uh see how this can be applied in practice, then I will hand over to matthew.

C

Yeah, so, let's move on to the practical part, we introduced the concept of using metrics in end-to-end testing in a theoretical manner, but let's see how the how these ideas are put to practice. What we want to show is that these concepts are universal and can be applied within any programming or scripting language, but the concrete implementation we'll be looking at is the end-to-end testing framework written in go.

C

So this is a framework which is hosted under the umbrella of the efficient go project, which is a collection of utilities, utilities and tools that are readily available for use by any go project, and the end-to-end framework specifically is intended to run complex workload scenarios in isolation with help of go and docker. The code of the framework was originally developed with the cortex project and later was donated, adjusted and moving to a separate module, mainly by our colleague martik potka.

C

We must have seen some talks from him because he had like 10 talks at the kubecon, so he's very famous.

C

So the framework has few like defining features that that are interesting. For us. It focuses on testing cluster workloads, uh so meaning primarily cloud native microservices, while keeping in mind test scenario readability, the scenarios are run in the containers environment with help of docker. Naturally, the framework also focuses on uh on putting metrics front and center, which is the feature most relevant for our for our talk. Today.

C

Not only does the framework support, asserting and controlling the scenario test scenario flows via metrics, it also provides self-monitoring out of the box by leveraging the c advisory utility. You can monitor the test containers directly and gain even more control and insights over your test scenarios. Lastly, it also supports on-the-fly containerization of your code.

C

So, if you're not running a fully fledged application, just a piece of code, you can containerize it and run it directly through the framework, and although we refer to this project as an end-to-end framework, there are actually multiple usage modes beyond plain end-to-end testing, as jessica has introduced some of these other types of tests, the frame the framework can be employed to facilitate this as well.

C

The framework enables users to benchmark with the native go benchmarking package while incorporating external dependencies, or you could use it for macro benchmarking, as it allows for running testing scenarios repeatedly and tasks, users can observe behavior and performance of their applications from a macro perspective.

C

Last mode is the one that jessica referred to as interactive setup, so this means that you can manually interact with the application and monitor them, and this is also very useful for engaging new users, because you can quickly spin up the whole setup just with one one command and it's all running on your local machine, and you can uh you can also. You can also monitor it right.

C

Let's look at some real world usages. uh We will talk about two projects that me and jessica are both involved with, uh that is tunnels and observatorium so tunnels, if you're not familiar with thanos, it's a cncf incubating project for highly available prometheus setup and long-term metric storage. uh You could have seen us at the boot here at the kubecon as well, and we've been using the end-to-end framework here to build and run this scenarios for all of our components, because tennos is actually a number of loosely coupled components and they need to communicate together.

C

The end-to-end framework is perfect for satisfying such complex scenarios with a large number of components, so it's not always true to ensure that the tests run successfully, because normally the components are long-running services. It heavily relies on asserting on metrics as well as metric driven test run control. Thanos project features a full-fledged interactive setup as well.

C

So this includes everything from synthesizing data for users to play with, to spinning up all necessary components with monitoring and tracing in place, as well, second project that we will mention today and uh where we mean just as we said, our maintainer is the observatorium just uh tldr and observatory. It is a one-stop shop for all observability signals.

C

It is a collection of tools which makes it possible to easily run a whole observability stack with some additional functions on top, such as multi-tenancy authentication, authorization rate limiting and so on, so feel free to check out the project if you're interested and here as well, we have been using the end-to-end framework to a great success, uh mainly with our core component, which is called observatorium api, and we use the end-to-end framework to create uh complex test scenarios across all different signals that we support metrics, traces and locks, because this requires spinning up multiple uh dependencies right because we are providing authentication.

C

So we need the authentication we need to connect to authentication provider, because we, for example, can use uh open policy agent for authorization. So this is another dependency we're also depending on external service. For for raid, limiting uh that we need to communicate with and last but not least, we also need to back this up by an object storage for which we use menio, and we again just run this locally through throughout this uh through this through the framework.

C

So having said that, now, let's look at the demonstration of all of the features that we set and for this we'll use a more simpler example than than uh the ones in the real world, and this will be just like just to demonstrate and see how this is how this works in the code and then how the actual tests uh are run. So, as jessica's mentioning, we will see an interactive test I'll just quickly switch this to mirroring so that everyone can see what I'm doing on my screen.

C

D

Okay, let me try one more time.

D

D

Let me try a couple more, I'm not sure. Now it's not reverting back.

A

Oh, it's like that with demos.

D

Okay, sorry folks, oh.

B

There's something there did.

C

It work. Let me try to reconnect sorry about that.

B

Yeah you can just like put that.

C

Okay! Sorry about that, I'm not I'm not sure if you're gonna be.

D

Okay, now it's completely.

B

If I try again.

D

Okay, let's one: let's try.

D

It's not reacting at all.

B

A

So now it's not connecting anymore.

D

Smile, my home, like.

A

I don't know if.

B

It's the cable or anything. I don't know much about.

D

Can you try yours yeah.

A

D

No, it's the whole screen screenness.

D

Wait something's happening. Okay. Now I have three screens cool. Oh it's.

B

uh Okay, thank you.

D

It's still not.

D

Now I have three screens sorry about that.

B

Oh yeah, that's it.

D

C

Want to mirror right.

D

Now it's just everything is a bit messed up.

D

D

Now my display is not there.

D

It's not recognizing my screen. I think actually tried that sorry about that.

C

Yeah in case it's not working there's there will be also video in the presentation, so you can watch it for yourself and there will also link to the to the example test, because it's it's part of our uh repository so uh yeah. Hopefully, if you're not able to make it work, then you can still see the results but yeah. Sorry about that.

A

Which one is it.

B

Hdmi actually.

D

Yes, you could see it. Okay,.

B

D

Can try mirroring.

B

D

B

A

Can I say, let me just check.

B

D

I can so I can try to. I don't think.

C

We have that much time but yeah. So, as you can see just this is like a plain uh go test.

C

Is it's a go test file we're using the native testing uh package, as you can see we're simply starting by setting up the docker environment, and um then we have another method to uh to start our application, we're actually using uh example, app, which is a which is a simple which is a simple http server that is instrumented for metrics and we'll we'll be just observing uh the request these are coming in and checking the metrics on that. So doing some additional things, making sure that we can.

C

uh We can access the service on our host machine, so we're exposing the port we're providing the image for the for the service that we want to use in the start, options and the init and then we'll go we're starting and waiting for the search to be ready. Right. Second thing that we will spin up uh is the wait. Sorry, it's okay, second thing: we'll spin up is the it's prometheus here.

C

You can see a simple, uh simple conflict for scraping we're providing the endpoint from from the app we previously set up uh for for for scraping. So we'll see the metrics in prometheus again we're starting prometheus setting the conflict waiting until it's ready and then what we actually do is here.

C

We will see a pause in the test, and this is intentional, so we will be waiting until prometheus is scraped until primitives until prometheus creates something- and we will do this based on the metric which, which uh checks how many head samples have been appended we'll wait until this is greater than 50, and this shows how you can leverage this to use for checking preconditions right. So before we move on, we want to be sure that we already have some data in there.

C

We'll also then open the example application in our browser, which is another nice feature of the framework. We have this antoine interactive package, which makes it possible to also directly open the service or the or the the ui that you might have like a web web ui directly in the in the browser, so that will automatically pop up for the user once it's ready and what you'll actually want to do here, which is the interactive part that we want to show is that we will wait unless, uh for at least five requests right.

C

So the test will not move on until uh jessica has made five requests on on the in the browser. On that example, app we do this, based on the http request, total metric, we're not constantly hitting the application. We do this with a back-off. So each time the condition is not fulfilled, we back off and we try again and we'll also wait on missing on the missing metric. So this option ensures if the metric is not there from beginning that we um did. We wait until the metric appears right.

C

Second thing that will also open up prometheus see what has been scraped actually which matrix we have there to gain some extra insights into our test run and that's it then we're done we're finished. We can close the test and now jessica will walk us through the cool.

B

C

Since we don't have it set up on this computer, maybe we can. We can.

B

C

Show the video or.

B

No, but it's fine, you can try here cool.

A

So just a bit hard for me to see, but like now we run this entrance right like we created.

A

So let me just get into the right folder.

A

Cool so now we will read, we run our make target.

B

A

It's just my command so that we can then run the test.

A

Cool so now we see already like some logs uh the example app is started. It's exposed in the port 8080 that then it's mapped to this another random part. Then we see already prometheus being started. We see some logs now we see that we are waiting right because we want to make sure that prometheus prometheus already scraped, something like on my screen. I already see the browser, uh so it's just open after prometheus scraped, some metrics, then we open in a browser and then now we are waiting for the five requests.

A

So I'll. Try to then move my browser here. So you can also see.

A

So that's that's our example. Application very simple, just prints hello from example, application uh and then is exposing some metrics. Now we try to do some requests, so I'm manually refreshing the page now.

A

And then now prometheus is open right, so I already reached more than five requests. uh Let's maybe check like the metric right see if we.

B

If we get something.

A

Cool, so here you see the time series. The metric name is http request, total all of the requests were 200, that's a counter metric right, so we had of like 12 requests more or less and and then all of them were get requests.

A

You can also check the graph like if I change this to five minutes, so you have a better view here. You see. The peak of the request coming in can also check the rate.

B

A

Those requests like.

B

B

So I can check like for the last five minutes.

B

A

See that it was like about 0.03 requests so like not so much um another thing I wanted to show. Also, I think, which you have time, um is this lash metrics endpoint. So this is the end point that our prometheus is scraping, and then here we have all the metrics that are exposed. So here you can also see the counter metric that we were requiring the http request, total and also all the metrics.

B

um And as a last thing, I also want to show.

A

How so this so as we we mentioned, this is the is an interactive test right. So that's why it's paused, but then, if you hit this end point this address, then the test will be finished.

B

I don't know why now I cannot do that, but I can also maybe yeah.

A

Okay, so now I hit the address, then, if I come back to the test.

B

I see that the test also finished.

A

So like, if yeah, if you want to automatize this, you can maybe like set up something that calls this address automatically and then the test finishes um yeah. I think, with that we saw how to set up engine tests using the engine test, the engine framework like how can we easily spin up prometheus instances? You can also spin up thunders and then, with that we had an example application that was exposing some metrics and in our tests we could assert all of those things right.

B

A

Also wanted to yeah, like I don't have this light here, but you can also check online like we uploaded the slides, uh so we left the links for all of those resources for the entrant framework and also yeah for the observatory project and also the thermos project, but yeah. Thank you so much.

C