keptn Keptn Tutorials, 22 Jun 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Automating performance and load tests with Artillery + Keptn

Description

Francesco Lentini from Artillery shows how you can use Artillery to run load tests and smoke tests to help your services maintain their SLOs.

Artillery: https://artillery.io/
Integration: https://github.com/keptn-sandbox/artillery-service
Announcement: https://artillery.io/blog/shipping-friday-170621/#Keptn-Artillery

Learn more: https://keptn.sh
Get started with tutorials: https://tutorials.keptn.sh
Join us in Slack: https://slack.keptn.sh
Star us on Github: https://github.com/keptn/keptn
Follow us on Twitter: https://twitter.com/keptnProject

A

Hi everyone um today, I'm really excited to see how you can automate performance and load testing with artillery and captain. My name is jurgen. I'm a maintainer of the captain project and today with me, is francesco lentini from the artillery team. Hi francesco, hi everyone, and I'm really excited to see what artilleries, uh how you can write load tests in artillery, how you can configure it and then, of course, how you can run it with captain. um I know you have prepared a couple of slides and a very exciting demo.

A

I had the chance to already see it once so. I'm handing over to you and please give us a short introduction into artillery and also how to use it with captain.

B

Yeah, thank you, hello. Everyone. My name is francesco, and today I'm going to talk to you about how artillery works, what it is and how it works with with captain a little bit about me. First up till recently, I was an independent contractor working as a node.js developer. I was helping companies transition to microservice architectures and I've been using artillery on several different projects over the last few years, and I am now part of the artillery team.

B

So what is artillery it's a performance and a load testing tool?

B

It's a command line tool designed to test the performance of backend systems, so we can use it to make sure that our services work as expected and they can handle traffic spikes and sustain increases in workload by emulating user behavior.

B

This way, it makes it possible to test transactional apis and complex systems in the same way as your real users would.

B

A

B

It supports the most uh popular protocols like http and web sockets, and others are supported by our plugins, and it already allows you to send metrics to external monitoring systems such as statsd influx db, datadog, etc, etc.

B

This is done through the metrics plugin, which is officially supported and more target platforms, keep being added to it.

B

So artillery is built on top of node.js and while artillery artillery tests are written in yaml, they can be customized using javascript and they can rely on the npm ecosystem.

B

So what runs in node can be run in artillery. Essentially so. Test supports hooks that can be used to inject custom javascript functions and they also support constructs like loops and variables.

B

This makes it uh easily extensible it's easy to write a custom plugin to to fit your needs in case a plug-in for your target platform or protocol doesn't exist already.

B

So we have a blog.

B

You should check that out for more in-depth articles and if you are interested in artillery and or need help with it or you just want to have a chat about performance testing, you can join our discord channel and, of course we you can find us on twitter.

B

So, let's, let's have a look and how it works.

B

So, as I said, artillery tests are yaml files and.

B

They are made of two main sections: one is the configuration the test configuration and one contains one or more scenarios, so we can see here uh our target service, what we're going to hit with our with our test and the load phase.

B

This means that this test will run for two minutes and it will start with five user five users per second and it will ramp up to 10 users per seconds, and it will sustain this load until the end of uh of the test.

B

Each virtual user that is created will will go through this. uh This flow and it will first get uh get a card, and then it will get the items in this card and it will post an item to to to our uh our cart.

B

I'm I'm using the for this test, I'm using the the subshop uh cards uh microservice. So if you are familiar with the captain tutorial, you should be really familiar with it.

B

um As you can see here, uh artillery supports inline variables. um This set of variables will be used that will be made available uh in the uh in the in the rest of our scenario, and uh the values here will be picked randomly for uh for each uh for each user.

B

uh So yeah, uh let's, uh let's oh yeah- uh it's also possible to uh have uh uh to get this data test. Data from external data sources and out of the box artillery already supports uh reading data from csv. So it's easy to do something like this.

B

Let's make it like this and then we can, and then this stuff will will read the data from our csv and and the customer id variable will be available in the rest of our test scenario.

B

Okay, so running a test in artillery is as easy as.

B

This and uh as I said, this will run for uh two minutes, 120 seconds and uh artillery will will gives us will give us an update uh of how the test uh is doing um by the way, since artillery is a command line tool, it's uh easy to integrate with your ci cd pipeline.

B

It supports validation against the aggregate response time latency, so you can do stuff like this.

B

And artillery will return with non-zero if this condition is not met. If the p95 is more than 100, milliseconds uh artillery will return with the node zero.

B

um If you are interested in how this works, we recently published a detailed article on the blog on how to do this with circleci and how to generate reports and the pros and cons of this kind of approach, and you should definitely check that out.

B

So our test is running here.

A

I think that's already pretty pretty powerful to write a whole user scenario with only a couple of lines. I think it's around 10 lines of code and you have three different end points that you want to check and one is even for adding items into a shopping, cart.

B

A

So this is, uh this is pretty exciting, I think um even having the um yeah the possibility to define the exact uh payload for post requests uh here and uh also the uh the integration with the csv uh that you showed us. I think it's really powerful to kind of scale.

A

The load tests also to to really a lot of different users that might have different characteristics.

B

Yeah, I think uh the fact that tests are written in yaml uh make it easy for make it easier for um also non strictly deaf people to to read them and modify them so yeah and uh yeah. So our test is complete and we get. uh We get a report here uh of how this uh this was doing.

B

um We have an aggregation of the response time. uh It's also possible to generate to generate a report here that can be used in the.

B

In the report viewer, so you can, uh you can load the report. This is a for a print from a previous run.

B

This report viewer is available as self-hosted or you can find it online at that url, and it gives you some.

A

B

And some information about your test run with a lot of colors. So how does all this fit into the captain ecosystem? So uh we we just released uh an integration for captain the the artillery service. You can find it as part of the captain sandbox, and it's quite easy to install you just need to apply the service definition here and uh artillery will will come with it. You don't need to install uh anything, but but uh we apply yours the service to the cluster.

B

um So let's see how it works in detail. I'm gonna trigger a delivery here. On my.

B

A

So I think what I just want to highlight here, uh while we're waiting for the for the um delivery. um What's really cool is that uh when francesco is showing the demo the left hand, side of the demo with the artillery load test uh will stay the same. So we are using this uh that we've already seen doing this from the local machine. We're now using this file.

A

uh It's already added to captain and uh we're just triggering now captain everything will now run inside captain and triggered by captain, um but it's it's basically the same definition, so you don't have to change yes by your definitions. You just add this to the git repository and we will see also the benefit of the whole thing. uh Just in a minute.

B

Yes, exactly uh yeah, that's a good point. The tests that are running uh that I just run are the same. That captain uh is gonna run for us, so uh the artillery service will react to the test, triggered the cloud events triggered by captain and now it's it's, uh it's uh deploying our our service. Let's see what is uh going to load, so we are on the dev branch at the moment.

B

And uh we have an artillery configuration that's available for all the stages, so, depending on the test strategy, it will trigger a specific resource so for a function for for this test strategy, which is the functional one.

B

This is the test we are triggering, so artillery can also be used to do smoke smoke testing uh via the plug-in, the expectation plug-in- and this is slightly different from what I showed you earlier.

B

It will just create one single user that will run through all these steps, making sure that our response codes match what we what we expect. Okay, uh so for this endpoint, it expresses 200. We add an item to our cart and we expect this and we can also do some more more complex uh uh comparisons. So we can capture the a variable from the uh response payload and we can make sure that uh that it matches our expectation.

B

Okay, if it doesn't this bit here, will fail and uh and our uh our delivery uh will be stopped. It didn't fail in this case, so our delivery is being promoted to to the staging part. Okay.

B

So in staging it's loading, it's using a different, different, different scenario. So we can check it here.

B

It is exactly what I showed you earlier same set of endpoints but different different configuration.

A

And I think, what's also worth to mention here, is that uh you won't find here the concrete url that captain is uh that is basically or not not captain, but where the where artillery is sending the load to, but this is injected- and this is part of the of the artillery integration with captain so that whenever you do a new deployment uh and maybe the url might change- or maybe you want to reuse this test file in from staging also, maybe you want to use it in production, you can you can use or reuse the same test files um because the url will be injected then uh in basically, uh when the deployment is finished, captain figured out the the url.

A

Then uh this deployment url will be used, uh so you can easily reuse uh these files. I think that's also really really powerful yeah.

B

That is correct. uh Captain will uh override the configuration here and it will inject uh the service url uh at runtime. So we don't need to to define it as part of our scenario, so our tests are still running. um There is no way currently for us to to read the uh artillery output, uh but it's in the pipeline, uh so we cannot have an update here of uh how our tests are doing, but we can uh yeah.

B

We can of course uh get it from uh from our pod, so yeah, that's just completed our test run and.

A

Actually, um this can already be done, so there is a um a change or or an update on the status. This can be sent to captain uh and, as francesco said, it's uh it's already in the pipeline. So this also the uh all the the status changes and all the different updates uh can be already uh sent to captain, and this can be shown here.

B

Yeah, so the tests have finished and uh for this for staging, we have an slo file here. So we can see that uh that our configuration is defined here.

B

So we have an slo that that is going to be used to our uh by our evaluation step. Here, evolution has passed and we can have a look at a more detailed view on how captain does the um how the quality gates work in in captain.

B

So this is pasta, it gets compared to our previous run and yeah. We can see that that the data is is within our thresholds, so our deployment can be uh promoted to the next stage, which is uh production.

A

And I think uh what's what's really powerful here now, is that, uh on the one hand, side we've seen how you can do, or already, uh let's say, a check directly in artillery uh if the the tests are successful or not by adding uh by using the expectation, plugin and doing directly the checks kind of a smoke test directly in artillery. But maybe you have, you want to have a larger slo evaluation, uh not only on the response time but failure rate, maybe even some kind of infrastructure.

A

Metrics like the cpu usage or ram memory usage, then it really makes sense to use the captain. Quality gates, put all the slos into the quality gate and then, after the testing phase have captain do the evaluation uh of of the tests and basically of the of the whole test run.

A

What whatever the time was. So if you uh make it longer, then captain would also, uh of course, respect the longer test run uh and do the evaluation for the longer test run.

B

Yeah, definitely uh you can set up uh so artillery is focused on uh latency uh latency metrics, uh while with captain uh you have a.

A

B

Spectrum of metrics you can, you can check with the quality gates.

A

And I think in this example, uh the data is coming from promethouse. So, although artillery um does not have the prometheus integration today- and I know it's on the it's also on the on the roadmap, but um we can still use promethor since uh we are doing the um prometheus is scraping the data directly from the clock, shop application, and so we are kind of taking the round trip. uh Yes, we are firing. The load from artillery to the sock shop.

A

Prometheus is monitoring the sock shop and captain is querying data from prometheus and doing the whole quality gate evaluation on prometheus and as we as we see, uh we did a couple of test, runs or actually francesco did a couple of test runs. So we can also see the previous evaluations where we have different kind of data- um it's not always 100 percent. It also depends on the ability of the pod like the uh how the how fast the infrastructure is. uh It is running and we can see all the different kinds of evaluations here.

A

um Yes, it's always good that we have no errors in the software application, but it's not always really fast in responding. That's that's what we see.

B

Yeah, so um exactly so, we we just. uh We promoted this to the production now and uh we just have a light uh smoke test in production to make sure that all endpoints were are working as expected in the same way as as dev that works yeah and yes,.

A

I think that that's pretty cool uh can you show us again the artillery uh github repository, because I just want to highlight how to get started and um it's really easy, um the the team and especially francesco from artillery they have done a phenomenal job in providing this integration and to use this integration.

A

You just have to install this artillery service into your captain installation, and you can do this by going to the captain sandbox on github, going to the artillery service, and you will have the deployment file and just with the cube ctl apply, you can directly install it into your captain cluster and with that artillery is already there already installed.

A

It will listen to all the events for a test triggered events as you, uh if you're already familiar with captain everything is event based and whenever there is a test triggered, then artillery will uh the artillery integration will start its work and uh we'll actually look for uh and for the the artillery files in the captain, git repository uh for the demo uh francesco has already added those files, as we've also seen in the in the github repository.

A

So I think it's really easy to get started um and uh and also how to write uh artillery tests. uh um It's it's. It's really uh easy to use. uh Maybe one question from from mind: are there any best practices on how to write artillery tests and load tests.

B

uh Well, uh this is, from my personal personal point of view, um not particular best practices on how to write tests in artillery uh just uh start as soon as you can uh make it part of your development process. uh Since it's quite easy to run this from your laptop, it's, um it should be part of your development cycle and then, as we said, once you've got these tests running locally. It's going to be super easy to to add them to your captain sequence.

A

Cool yeah, I I think so uh as well. It's uh really get started with artillery uh pretty early. um You already get uh all the http response code as we've seen. So, if you already get the like the 400s uh well, while you're still developing, you can already fix this. uh It is very, very powerful uh and then later on, just add those files uh to captain, and you can even build some kind of self-service um scenarios with captain plus artillery, which is the trigger a new delivery with captain it will deploy the new version.

A

It will test the new version and it will give you back the evaluation of the quality gate. So you can either do just the single stage, deployment and delivery with captain we've seen a more complex one that francesco built with the multi-stage delivery. But if you want to do it more in the self-service kind of way, just do one stage deploy it test, it have a quality gate evaluation and then you can reiterate and often uh on the quality of the service.

A

Yeah, I think, uh we've seen a lot uh in this uh like 20 25 minutes. um Thank you so much for for uh presenting this. uh Is there anything else we should know about artillery and captain.

B

um Well, uh the captain uh integration is um is pretty pretty green, pretty new, uh so if you're interested in it, please uh check that.

A

B

uh Give it a go and if you we are of course open to contributing contributor uh and um yeah, we have a roadmap. So if you want to participate in that, please uh please get in touch.

A

Cool, thank you then yeah. I just want to also emphasize this. Please um give it a try. uh I think it's all, although it's it's it's pretty new, I think it's already pretty powerful can already be used. um Please let us know how it goes. um You will also find us in the captain slack on the slack.captain.h.

A

We are happy to uh to engage with you there and hear your stories how you, how you're using artillery uh in captain, um then francesco thanks so much for presenting this. uh Hopefully we hear some success stories uh pretty soon, and then we can do another video on this. For today. I really want to thank you and uh give it a try, uh dear captain community, and do all your load tests now with artillery.

A

Thank you thanks. Everyone bye.