Cloud Foundry Cloud Foundry Summit EU 2019 - The Hague, 19 Sep 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Cutting-edge Continuous Delivery: Automated Canary Analysis on CF using Spinnaker - Andreas Evers

Description

Cutting-edge Continuous Delivery: Automated Canary Analysis on Cloud Foundry using Spinnaker - Andreas Evers, Pivotal

Using Cloud Foundry, we’ve learned best practices about how to deploy software at velocity — things like automated releases, immutable infrastructure, gradual rollouts, and fast rollbacks. One of the most sophisticated techniques of going to production quickly, with confidence, is called automated canary analysis. While canary deployments themselves are becoming a commodity, having an autonomous judge at your disposal, makes all the difference. The judge analyses metrics of both baseline and canary deployments and automatically rolls forwards or backward. Meet Spinnaker, an open-source multi-cloud continuous delivery platform which embodies these core principles of safe, frequent and reliable releases with contributions from Netflix, Google, Microsoft, Oracle, and Pivotal.

For more info: https://www.cloudfoundry.org/

A

Welcome am I talking about contagious delivery, I'll be talking about automated canary analysis, 3 use of the tool called spinnaker, but first let me introduce myself I'm andreas eavers I'm from Belgium a solution architect at pivotal, I work in the application transformation practice for amia and when I'm all our customers, I'm active in pivotal spinnaker team, which is a collaboration between the spring team and foundry RG or pivotal R&D, and you can reach me on Twitter with my handle andreas eavers. Also after this this conference. If you have any questions, let me know all right.

A

So let me talk about context delivery and when we talk about continues delivery. The ultimate goal that we want to reach is to deploy software quickly and automatically, and we can't really do that if we're not able to push code without fear. So this highlights the the eternal struggle between two opposing forces in software development where, on the one hand, you've got speed and only our hands, we've got confidence and somehow they don't seem to like to work together, often so, especially with micro service development, because we are now doing Microsoft surveillance.

A

It's gets really hard to to make sure you have confidence when going to production, because micro services move that faster production. It's very hard, if you have an acceptance of runs to make sure whatever you're testing, it's not a moving target and it's not already changed on production. So we got to start looking at our alternatives. Other ways of testing and one of the ways that we can start adopting is to test in production and it sounds a bit crazy.

A

But if you use things like automated canary analysis, it can actually be very, very beneficial and once you I can do that is spinnaker. So, let's take a look at spinnaker. What is spinnaker spinnaker is a multi cloud, multi-region deployment tool built by Netflix that supports multiple abstractions. It supports kubernetes, but also Cloud, Foundry and also VMs or bi-metal. So all the major cloud providers are supported and they've all contributed integrations for them, especially Google they've I think with 14, after es a team of 14 people on this project.

A

So it's their main containers delivery approach, which is really powerful, so kubernetes is well supported and pivotal has supported splat, foundering so we'll see how that works in our demo as well. Netflix, of course, have supported or have integrated Amazon Web Services, because they run everything only AWS and you can run an openstack compute engine, App Engine or to cloud this us even Alibaba cloud is coming up so, as you can see, there's a whole ecosystem around spinnaker.

A

Of course, there's cloud providers that have already mentioned there's also CI systems like Jenkins Travis, worker concourse and more coming up. That will integrate really well with spinnaker for mortaring, because, if you're going to be doing automated canary analysis will be using monitoring, we can use data dark, prometheus single effects, effects, it's tag, driver Atlas and there's many many more coming up.

A

The only important thing here is that it supports multi-dimensional time series so, whatever solution you want to integrate, if it has that and it's fine and in our demo we'll be using data log, you can use, of course, notifications and artifact support artifacts. Our deliverables could be like binaries. Things are you're going to deploy and that could be a docker image. It could be coming from Google, Cloud storage and could be github and it should be called s3.

A

We will be using s3, Amazon, AWS s3 will be using for artifacts and artifactory and our installation, so spinnaker itself will be running on pts and we'll be using cloud foundry on two different on two different foundations. So, as you can see this in many different clouds that we'll be combining here and because that's kind of the nature of spinnaker that works with all these different clouds under the same concepts, the same abstraction model which is pretty powerful, so ah one thing I shouldn't forget there CI systems there.

A

So you might wonder: isn't this the same as Jenkins? Isn't this just another CI cd2, and actually it's only CD, it doesn't try to do CI. It doesn't try to build your applications, sure it could build. It could bake a VM like Amazon like Netflix. Does that, but but it kind of delegates the work to these CI systems that have a lot more integration as possible. So you don't have to you.

A

Don't have to do, Java builds or anything like that in spinnaker and sure you could ask well I could deploy to my cloud environments with Jenkins as well. Why would I need spinnaker and, and the difference lies in state management's Jenkins doesn't contain any states of your applications and where they're deployed it doesn't know that it has to make calls and figure that out.

A

But if you have multiple clouds or just one and on premise perhaps- and you have multiple regions- multiple availability zones, multiple accounts, multiple environments, it gets really tricky, and especially, if you have multiple versions live on production, and so things like rollback, sophisticated roll backs depend are dependent on which version is currently running, at which version what is actually which version is actually correlated to which commit, and that gets really hard.

A

Some systems do that by tagging and get, but that only supports a certain flexibility and especially if you have multiple versions running in production that can get really tricky. So spinnaker differentiates it here by by having this inventory, where it keeps a keeps track of. What's currently in production of your application and use that information uses that information to drive pipelines in more sophisticated ways, if you're just doing a CF push sure you can do that in Jenkins.

A

But if you want to go further and have more safety built into your pipelines and when you're going to production, then this is crucial to have.

A

So, let's take a look when we go to the cloud and when we deploy- and we do many more deployments than just a simple CF push. What actually is important to have, and it continues that every tool- and the first thing we'll realize when we do scale this out, is that cloud deployments can become really complex and especially if you work with communities. You'll know that this is actually not easy and so and if you have multiple people and multiple different teams, making multiple different changes to multiple different services that gets tricky.

A

If it's especially if it's across multiple different cloud providers, multiple different availability zones, regions, environments, accounts that gets really really complex, especially because these providers, they have different terminology, different concepts. Different abstractions you'll, have to kind of match. All that, together in your pipelines and sure you could say well, we only have Amazon or we only have Google or only Azure or whatever right.

A

We have just settled on one sure, but you also still have on premise probably right: there's not that many companies that completely migrated so having the view across all your different environments, including on-premise, that's an important part and for for Cloud Foundry that becomes really useful. It becomes really useful to have a composite overview like there's an spinnaker, a multi foundation view. So you have your application. It's deployed to in that environment, testing, environment, staging production.

A

You can see all that together in one dashboard in spinnaker across your foundation, so you don't have to go login into each foundation to check where which version is deployed where or just through the CLI is just have to switch between your foundations. That's not it's not really great, and here you can use that information to drive your pipelines and when you're creating these pipelines, it's sometimes hard for new teams to onboard with fun with glad family.

A

It becomes easier if you use kubernetes it's still hard, but in general, if you want to have sophisticated pipelines that use metrics, then then, having guardrails in place like template, a paved road, it's really useful to have them an easy, onboarding and spinnaker offers that, through use of split of templates, where you can simply have a pipeline predefined and just you can just push safely to production.

A

So that's one of the key features and when you're in production things can go wrong, it happens so having the ability to easily roll back it's important and having just this button, you can press or you can have an automated rollback.

A

That's really useful to have, because when we have row backs is when we were under duress when we have stress and then we don't, we think clearly I mean at least I, don't so having something like this available, where you can clearly see which steps are going to be undertaken and in in an automated fashion, not something that you have to do manually and think about when you're under stress it kind of helps to get things back to normal and helps to take the next step. For it.

A

Has anyone seen this image before it's quite a lot of people? Sadly where's it from github yeah when kid ups down, they show this picture I think they want to say that it never happens because unicorns don't exist, but unfortunately it does and they really try to make. With this picture, they try to make it clear that you need to make sure that whenever you go down that, as as least amount of people are impacted, so you got to make sure your blast.

A

Radius of your changes is as small as possible, and you can do that by things like deployment windows, where you can look at your traffic of your application and then correlate that with the time frame in which you want to deploy, of course, added with the idea of mine or people in the office, because if you do that at night of course, then it's harder to troubleshoot and something does go wrong so having that availability or that that that feature available in your in your spinnaker installation is is really cool, because you can use that as part of your pipeline.

A

You can say, let's just let's prepare everything upfront, let's make sure, make sure everything's automated on my pipeline and at the moment that I want to deploy to production or the moment when I even want to switch traffic. That very last point. We are not gonna make that decision. I'm gonna, wait and wait until deployment windows there. So that's pretty handy too. To make sure a blast.

A

Radius is as small as possible and when you're creating a deployment when you're going to production, then having strategies is a key feature and blue/green I think everybody knows about Bluegreen. We've got plugins in cloud foundry that do look real, um that's pretty straightforward! You just pull up a new version next to the existing one.

A

So do you have two server groups and you just just switch the load balancer the advantage there is that if things go wrong with the new version, you can switch back to the old version, no worries, but the downside is that everybody's exposed to the new version, because you switch everyone over so rolling blue-green is the next step. It's an incremental fashion, where you will be taking one instance at a time offline and a new version online. So that's an incremental cut over now.

A

When we do these steps, we can have a manual verification, whether it's going fine and we can automate that with smoke tests, for instance, that's all from what we're doing, um but it doesn't come. It doesn't go as far as canary testing, so we all do canary, releasing and in a way rolling.

A

Bluegreen is, at the first instance, a canary you're, deploying a canary as a new version, two small set of users so in a way you're doing canary releasing, but it doesn't mean that you're doing real canary testing in the full possibilities that it offers and canary releasing how spinnaker sees it is much more sophisticated it will.

A

It will do an incremental increase of of traffic towards your Canaries, but whenever there is an increment is going to check whether the metrics of the new version compared to the metrics of the old version is a positive change or a negative change, and based on that it will automatically roll forward or roll back, which is pretty powerful, and it's going to do that with what both seed in the demo, how it's working and it's pretty cool actually and, of course, these pipelines.

A

When you built them, you can just script all that and groovy scripts and Jenkins, or you can write your own course scripts or whatever you want to do. You can do that right. It's possible, but I bet that 80% of the scripts that we write is always the same script, and we always want to do some kind of blue green or something like that um and some smoke tests. And it's always the same thing, we're writing and now why don't?

A

We have that built into some kind of framework or some kind of tool like spinnaker spinnaker offers this, um where you can just define your pipeline with a set of stages, and each stage will have like this I set a set of steps and each step will have a set of tasks and each tasks will translate into low-level operations, and these things are contributed by the cloud providers themselves, so they're definitely tested and they're definitely going to work with us with the specifics of each cloud.

A

So you can do this by applying with Amazon with Google, with with Azure with with Oracle, whatever you want in one pipeline, and it's just going to look the same. So it's very powerful. You don't have to worry about all these fermentation details, and you don't have to you know, just in a trial and error, to try an error approach where you're just gonna test it by running it, and this is definitely baked into the system. It's an important thing to have now who's.

A

We can be looking at canary bird, sir who's, who is a bird lover here. Okay, we've got four people so for you, I've got special slide in here. The Latsis appreciates birds and especially canary birds, because we're going to be killing a lot of them. So let's take a look. How cute they are I mean look at the cuteness. It's amazing right I mean this. One is the same as the first first one, but come on, but we're going to be killing them. So enough of that, and actually how we're going to do.

A

That is through the use of a score. We're going to give this canary bird a score of how trustworthy it is, how reliant reliable it is, and then we're going to move forwards or backwards based on metrics. Now this is just a screenshot, we'll be seeing it in real life in just a minute. Now we also have chaos monkey integrations, so this tool is coming from Netflix. They build it originally, so they of course, have deep integration with their simian army that they've built of chaos monkeys. So you can also integrate that pretty easily.

A

So, to conclude, we can go faster with safety and there's a number of features that spinnaker offers and specifically the deployment phase where it lets. You have more safety, while still being able to go faster production, and so that we've seen automated rollbacks deployment windows chaos monkeys deployment strategies in an automated way and automates canary analysis, which will demo as well and but there's more there's close to rock in traffic guards manual. The manual judgements conditional pipeline and staged executions and notifications, and these first two I might have to say a little thing about it.

A

So cluster locking is really about when most Popeye plans are acting on the same cluster, which is kind of a server group in in the terminology of spinnaker.

A

Then it could be that you're having side effects or you're having certain combination of factors at play, where you don't really have a predictable situation, so cluster locking allows you to lock a certain cluster when you have a certain when you have one operation acting on it and traffic guards make sure that whatever strange combination of sophisticated pipelines are running.

A

At the same time, you will never delete the lost server group in production just to make sure right, because you know there could be the strange combination of factors that does delete the last one, and then you have downtime. So you can have more of these features in there that allow you to protect yourself. So let's go to the demo right. That was always more interesting, so for this demo, I've got a little scenario. That kind of helps us to to understand.

A

What's going on and I'd like to use Rick and Morty who's, who is familiar with big amore? Oh wait! Who's! Not familiar Rebecca Mori, okay, this is so quite a few, so Rick and Morty is a cartoon. It's kind of crazy. It's about interstellar, multi-dimensional time, travel, stuff and you've got this. This crazy engineer here on the right and this little kid that's always taken along for adventures and there's two adventures that we'd be talking about today and they're gonna be defining our blue and our Korean versions.

A

So, on the one hand, we've got our green version, which is pickle Rick. Now in this series, pickle Rick, um actually it's just Rick who turns himself into a pickle because it doesn't want to go to family therapy. He just hates it so well, actually, because he's a pickle he starts to roll and he rolls into the sewer and has to start fighting rats and cockroaches, which is not great um but eventually comes out victorious. So that's amazing, he's very strong.

A

He's predictable he's a stable green version, so we want to keep that in production because he's on our side now, for our greet our blue version, we've got the blue antagonist, which is mr. meeseeks and he's a gene in the box who only exists to solve a task at hand. So it sounds great because he'll help you out, but the problem is: if he can't help you out, because the task is just unsolvable, then he become aggressive. He'll become dangerous, for us he's a chaotic version and we want to roll him back.

A

So the blue version is something that we don't want to keep in production, um so the backstory I, if you haven't heard about it, so we missing mystics is being spawned to solve the task of getting this guy to golf better. But he just can't play golf. So he spawns an army. Six, the music spawns an auto mistakes and they spawn other mystics to help them out.

A

But you know this guy just can't pick up, so it's pretty bad, so they start to get aggressive start to take people hostage and eventually we gotta roll this back right and it gets really bad. So, let's take a look at our demo, you can serve to this specific end points. This is production and you can take a look at what's currently in production, so you can also take like a QR codes screenshot and to see what's going on in production.

A

I just really hope this is set up correctly because sometimes I test this out and forget to roll back. But nobody is fine.

A

It's pretty simple: Big Dawg do /tf summit all right, it's green all right! The green version is currently applicable is currently in production, amazing, and so let's go to to lets go well. We all know it's now currently green crates. Let's look at our application. What's actually driving this green page um now, I've got a spring booth, page a spring which sites or spring good web application whoops.

A

It's really simple: it has five classes, and only one of them is important and that's the controller. So the idea is that it takes a call and it then returns view super simple and the view that's currently being returned is that this visible in the back great it's going to pick a brick, as you can see here, all right pick a records, the green one. Alright, so that's that's great and we've got some random sleep at at the moment, which is between zero and 100, and the chaos is zero.

A

So that means it's from zero to a hundred that we're going to sleep and then we're going to return. So if you refresh the page you'll have between 0 and 100 milliseconds of sleep before you receive the page you're not going to notice that, because it's it's not that much, um but we're going to change that we're going to introduce our second character, which is mr.

A

meeseeks, so I'm gonna change this application, of course in github, because that's what great developers do and me seeks there we go and our me6 application is going to act a little bit more erratic and the way we're going to simulate. That is by increasing the time it takes to load the page by about 50, so our chaos will be 50. Now that's 50 milliseconds. Just remember that our initial pickle Rick already slept for between 0 and 100 milliseconds.

A

So it could be that if you look at the matrix that you think you are you're dealing with the music's, but actually you might not so just to make this a bit more unpredictable. But then we're gonna be looking at trends right. So that's going to be interesting. So now we're going to increase the trends, to always add 15 milliseconds. So, let's switch to meeseeks all right there we go so I'm gonna commit this.

A

What's going to happen, is it's gonna be built by Travis here, which is for free, so I'm, using that one and it's gonna build my application that will now be a music application once it's done that it's going to trigger a pipeline in spinnaker. So let's take a look at spinnaker.

A

First of all, we can go to infrastructure where we will see our application and how it's currently deployed in all our environments across all our different foundations, and here you can see if I zoom in a bit there we go that we have a Sonoma foundation, that's the name of the PCF foundation and that we have running make left-hand refoundation and sir a quick answer. A quick is used for dev and tsunamis used for prop and in production.

A

Currently, four instances are running on Version three I have version two just there to roll back to, but I don't want to do that, because four and three is acting great. It's our green version on dev, we have version 22 and it's currently working. It's currently green I can go there if I go to my load balancers. If I go here, it's just sort of quick and I go to this application. It's also green. So my development version at the moment is also green great.

A

So now, if I built, is actually working, it's triggered it's now building. Then it should deploy something to death. Let's take a look at how that happens. So we have here delivery pipelines, so you can take a look at what my pipelines are so I can configure the deploy to dev pipeline, which is gonna act on sir Creek, our deployment, our dev deployment environment.

A

So let me take a look: how that's the so the configuration here is actually looking at a web hook and the web hook is going to be triggered by Travis if the builds were successful. So let's verify whether that's working in a second. Now, if I look at what's what's going to happen once this web hook is triggered, it's going to look at an artifact from an artifact repository and there can be many of them, it can be. Jenkins could be art factory.

A

It could be as a tree could be custom, artifacts could even be github or Google Cloud storage or whatever you wanna use. So these are all these artifacts repositories. So I'm going to look at an s3 bucket on Amazon, where I'm going to take my built out, artifact I'm, going to deploy it to death and the way I'm going to do that is through using a strategy called Highlander. Now is anyone? Does anyone know what Highlander strategy could be? What could put the Highlander strategy me exactly?

A

There can be only one, so the strategy is gonna. Take it's sort. It's gonna cut off the heads of all the existing existing instances. I mean that's kind of what it means, so it's great because on that we don't need to be had available or fault-tolerant or whatever. We just want to see that one instance working so that's gonna happen and it's going to deploy that with Java build pack. So this is kind of what you're put in the manifest.

A

You could have a manifest also from an artifact repository from wherever you want to get it or you could define it through. A form and I opted that for I opted for the form the route is going to be the same route for sort of creeks or the dev environment, and here you can see all my keys, which I'll be changing after this presentation of course, and you have developed deployment type here, which is going to say this is just the development instance. You don't have to look at this for my further stages.

A

So if I look at the pipeline, it's been triggered and let's take a look at infrastructure. As you can see now version 23 is now coming online. I don't have to go to my CLI to figure out. What's going on on on that foundation on cloud foundry I don't have to go to the to the interface or anything like that. So that's great. So it's deploying that and it's going to remove version 22. So the points you brought. That's the next step.

A

Let's say we're going to go ahead with this, we'll see once the deployment finish, because it still has to tear down the old version with the strategy of Highlander diploma to prod is now going to do more sophisticated stuff. Let's see so it's first going to deploy a baseline and then it going to do and at the same time it's going to deploy a canary.

A

These two things are going to be: are our two instances actually for cuz I'll be deploying two of each and where we going to do our automated canary analysis on, and why do we deploy a baseline because we already have production running? Why?

A

Don't you just use production and the canary is because when you have a long-running application, it could be running for months already, then you could have memory leaks that are build up, or it could be that when we start the canary that we have certain spikes and CPU and memory, you could be warming up caches. So those things we don't want to compare it to a version, that's already running for a very long time.

A

So we also want to have this level playing field for the for the baseline for production, and we want to spin up baselines as well, and then these two are going to run next to each other. So the baselines, just gonna clone from production, that's simple and the canary is going to take the same version as as it is on development. So this is going to be hopefully our musics.

A

Then we do canary analysis and, with the canary analysis, we're going to be using a configuration which we'll take a look at in a minute and we're going to run for six minutes and we're gonna have a delay of one minute before starting the analysis, because we really want to get rid of these first, this first minute of startup, because the Mavericks there are a very difference than what you normally have when you run the application.

A

So we were just going to ignore that part and then every minute we're going to have an analysis we are going to check. Is this actually good or not? And then, based on our results of this analysis, we're going going to add a roll back or roll forward every step, every 10 10 seconds we're going to have a step where we're going to receive metrics from data log because we will we'll have metrics in data talk. So let's take a look at what's currently going on.

A

So it's almost finished here. It's gonna be taking over 10 seconds, I guess and then it's gonna kick this off. So let's take a look at our config for canary. So it's this relief okay. So this is shrinking the cluster right now, so our canary calm that we're referencing in our canary analysis phase looks like this. So here we've got two different groups: we've got the Rick and Morty metrics and we've got system metrics for Rick and Morty metrics.

A

We've got the rendering time average and the rendering time maximum, and so this is going to look at how long it takes to render a page. And if the tool is successful, it should look at trending trending between metrics, coming from the first version and coming from the second version and compare these trends and not just freak out.

A

If there's something something wrong with the with the initial metrics that you get so it should really be looking at some statistics and persistent metrics, we've got CPU for the CPU usage for the system for the process, some memory usage and also looking at log back events anything in the logs, if there's errors being generated, if you want to roll that back as well, okay, so these are basically are the different metrics that we are going to track and we're going to then analyze whether we want to go forward or backwards.

A

The metric group wait we're going to assign more importance or more weight to recomm or D, because that's our business metric in a way that's the metric. We define ourselves while system metrics, well also important, of course, but for our demo here we were only attaching 30 weights on that right. So this is. This is kind of triggered. Now, yes, great so now, baseline and canary are being deployed now, I'm gonna have to ask all of you to refresh your page once this, the first men in the form of is finished.

A

I'll tell you one, because we need a lot of metrics. We need some data from production. If nobody is visiting our website, you don't have used it. We don't have metrics for our rendering time and to help all of you out. I'm gonna be also generating some random loads so that we actually have a nice graph of nice metrics no money on production. You have some metrics, so here you can see already.

A

If you refresh, you can sometimes see the music's page and sometimes the picker rec page, so I'm also going to do that I'm going to production, so production load, balancer and, as you can see here on the production, we now have four instances of the production environment, two instances of canary and two instances of baseline.

A

If I go here well, then I now have the green version. Great. That was already there the far refresh enough because of course, it's 2 out of 8. Then we will have mr. musics. This is mr. music site, the blue version, um and this version should normally generate some bad load or some average really higher rendering times, as you can see here, dated August receiving metrics from the applications awesome.

A

So, if I'm looking at my pipeline, ok, the deployment of the baseline is still ongoing. It should be finished right away. There you go so now the canary analysis kicked off, which is pretty cool. I got a have to make it a little bit smaller, because otherwise it's not really showing it nicely.

A

So here you can see, we have six minutes, so there's gonna be six six times and analysis and analysis now, of course, if the first analysis already shows that this application is not actually working at all or these, these rendering times are clearly higher, then it's just gonna abort, because we don't want to increase this. The blast radius remember so this is this is happening. We are it's. What is actually going on now. Is the warm-up wait, so we there are 40 minutes into the first minute of warm-up.

A

Wait we're just going to wait until this application is becoming healthy. Then we're gonna wait for one minute to receive metrics. So in seven seconds you can start refreshing.

A

We can do it an hour already already, if you want alright right so now, the metrics that you're going to start generating are actually going to be part of the analysis, so this is pretty cool. So, let's take a look at the other thing that we're seeing here this orange one. This is a manual judgment. It's actually just a big red button to press. If you want to continue or not now this is the difference between continuous delivery and continuous deployment.

A

All right, so the more cool thing, I suppose, is to be completely automatic true, but you can't underestimate the value of manual judgments like if you have human, interacting with the pipeline. The benefit is that this human has a gut feeling and computer doesn't have a good feeling, so it could be that the human senses that there could be something wrong. Maybe they maybe he heard something from a colleague that there might be some firewall changes coming up just during this timeframe.

A

It might be possible that is just right now and it might impact, especially if you're changing your version, so you could say, like okay, the canary analysis turned out to be successful or my application isn't working, but actually I don't want to go ahead and or it could be, that you want to have marketing, maybe align this with some kind of email, app campaign or anything like that. So then you can say: okay I want to go ahead or not. Oh, so, let's see our canary analysis has already reported that something's wrong.

A

So let's take a look. The canary has run the interval was one minute and let's take a look at what this generates. So now we go to our canary reports and our canary reports here show me a number of different metrics. So, of course this is all the same ones that I've created in my config you've got JVM memory usage, which is almost exactly the same family.

A

But if you just look at a certain moment in time, if you just compare this manually the canary with the with the the version on production, then you might see what this is completely different shara, but the trend over time indicates that actually this is very similar and due to garbage collection, you can't predict when things are going to be similar or not. It's just a trending, that's very important.

A

So the engine here, that's going to judge whether this is successful or not actually takes into consideration trending of your metrics and log back events and there's no, no events on another side. No errors. Application is working amazingly well, then, for CPU usage. Also, there you can have spikes, but eventually, if you look at the the differences, it's pretty pretty similar and we can also look at the bee swarms block box plot and you can see that the standard deviation is kind of similar and, of course, you've got outliers.

A

As you can see here, the CPU sometimes can spike, but we don't care about that, because the standard deviation is similar, then, for the run wrong time. Well, I think this is pretty clear now and then looks like this this huge difference, because, of course you could have with your baseline from 0 to 100 milliseconds and your canary.

A

Let's say you have a hundred milliseconds where you have to wait and your canary is just by a chance always ending up on 50 milliseconds, so 0 plus 50, then you might think your baseline is acting worse than your canary. Well, it's actually the other way around now. Here, of course, our metrics are are smart enough or our system is smart enough to to judge the metrics correcting. So now, of course, it's going to roll that back.

A

I think it's already done that if you refresh your page to door, only be pickled right now, yeah, it's so like two minutes ago it throw down all the other stuff. So if I look at my my infrastructure now, everything is gone again. Super simple automatic, the blast radius as small as possible. Alright, so that's the end of the demo. Let's go back to the slides, so, as you can see here, automatic marinelle's can really help you to start testing on production and not having to have lengthy release cycles with a lot of different tests.

A

You want to run an acceptance environment, and this kind of indicates this this struggle between speed and confidence that you can combine them and it's using a spin through like spinnaker office. You a lot of different features to get to production fast or making sure you still have confidence.

A

If you want to know more about this, then I advise you to read this book. It's available on spinnaker lay your website for free, it's about 75 pages, it's a really quick, read, and but it goes more in detail about all everything I've talked about today and also about the concepts behind it.

A

So if you don't want to use spinnaker, you could use an order tool as well, but just the concept of making sure you can actually have safety while going fast, alright, and if you want to see the slides they're on speaker deck, you can reach me on twitter. If you have any questions for the whole sprinter community, you can reach us on slack stackoverflow, just attack spinnaker and there's a lot of documentation on spinnaker name. So if there's any more questions, I suppose we're out of time but I'll be around.

A

So you can definitely ask and thanks for coming over.