Cloud Native Computing Foundation ArgoCon 2022, 21 Sep 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lightning Talk: Open Source Canary Deployments with Application Metric Analysis - Hannah Troisi

Description

Lightning Talk: Open Source Canary Deployments with Application Metric Analysis - Hannah Troisi, New Relic

Deploying new code to your production cluster can be stressful. A canary release can reduce risk and give you more control over the progression of your application updates. However, canary releases are more of an art than a science. How much traffic should the canary get? What type of analysis should be run? Which metrics should I analyze? How long should the analysis run? A carefully designed canary configuration can maximize your chance of catching an issue before the new application version is fully rolled out. Luckily, Argo Rollouts provides advanced deployment capabilities that can be customized to your application’s needs. This session will cover the following topics: - How to do canary deployments with Argo Rollouts - The different ways to perform analysis to drive progressive delivery - Selecting the best metrics for analysis (this is very application dependent) - Practical demo: Canary deployment with application metrics from Pixie

A

Hi everyone, my name, is Hannah and I work at New Relic on the pixie team, contributing to pixie, which is an incubating cncf project.

A

So I'm here like you, because my goal is to make deploying code to production, less stressful and so uh like, no matter how much you test your code, there's always a risk that when you expose it to real customer traffic, that you'll surface an issue you hadn't found before so today, I'm going to show you how you can do a completely open source, Cloud native Canary deployment, with application metric analysis and we're going to use Pixi to provide the application and Metric analysis.

A

So a canary deployment strategy reduces Risk by diverting a small amount of traffic to your new version and metrics from the canary release, inform the decision to increase traffic to the canary version or roll back to the staple version and this diagram simplified. But you can have multiple analysis. Metric collections at that collect and analyze metric set.

A

So Canary analysis is more of an art than a science. A canary release doesn't guarantee that you'll identify all issues with your new application version, but a carefully designed one can maximize your chance of finding any issues. So you should make sure that your Canary um analysis, your Canary pods, get enough traffic that you're going to surface issues, but not too much traffic that if you do discover an issue, you expose your users to it.

A

So the rule of thumb is that you should give your Canary at least five to ten percent of your traffic, and you should try to overlap with your Peak traffic periods.

A

uh The trade-off here is that the longer you run, the analysis you'll potentially get better data, but it will reduce your development velocity. So there's a lot of strategies around this and the strategy we use is to do a short First Step that has a fast fail and then longer sub subsequent steps, and also you should make sure that the services that are most critical are getting the most analysis on them.

A

So for API, Based Services, the common metrics to analyze are latency error rate and throughput, but it's totally up to your service, for example, at New Relic. We use Kafka as their messaging system and we use agar rollouts with Canary analysis to monitor Kafka lag, because if you use Kafka, you know that consumer producer lag can definitely cause some incidents.

A

So you can actually manually perform a canary deployment using native kubernetes, but the benefit of Argo rollouts is that the controller manages all this for you and so Argo rollouts is the controller plus two crds, the rollout resource, which defines it's a drop in replacement for the native kubernetes resource, and it contains a recipe for splitting the traffic and Performing analysis on the canary version and then the analysis template which contains the instructions on how to do the metric analysis and defines the success criteria for that.

A

um These two crds provide tons of configurability we're just going to do one example today. So here's our rollout resource on the left and you can see at the bottom the steps we're going to direct, initially 33 of our traffic to the canary, which is way too much traffic.

A

But this is, for example, purposes and then we're going to pause, 60 seconds to give our analysis some time to run and then we're going to increase the traffic to the canary to 66 and then pause again to run more analysis, and we just have a background analysis running where on the right. You can see that we're basically um uh uh every 30 seconds after an initial delay of 30 seconds, we're checking for the area of our Canary pods, and if that errorary is greater than five percent, we're going to fail fast.

A

So how are we going to get the error rate for our Canary pods? So here's where the open source application metrics comes in so pixie is an open source observability tool for kubernetes applications. It's a cncf incubating project and one of the main features of Pixi is that it is automatically traces protocol messages. So, for example, this demo makes HTTP requests and instead of instrumenting the application we can just deploy the application in an environment that already has Pixi deployed and Pixi will trace. The Linux Cisco is related to networking.

A

So um all the app anytime, the application, makes a network request, we'll trace it at the Linux Cisco level. So I'm going to just show you this in um in person rather than spend time explaining it can I go to the demo.

A

Okay, so here's our demo app the Argo rollouts canonical demo, application on the right and I haven't changed it at all. Basically, every bubble is a request. Our browser is making to the back end and the back end responds with the color. That's indicating the version of the application we have deployed, there's a bar chart at the bottom and right now you can see that all of our requests are being handled by the stable Blue version.

A

um So we have pixie on the left, and this is just a script that shows us all of the HTTP requests. And again we didn't change the Argo rollouts demo at all, we're just looking at the requests here and we can filter all of the requests to the canary pods oops.

A

So now we're looking at all the canary demo requests and if we expand it, we can see the end point it's being sent to the request. Headers the request body, the response status code and response body and pixie is very scriptable, so we could, for example, group all of these requests to see like which.

A

Okay, so we can see that we're only getting response from the blue um version and we're only getting the 200 status code and over on this uh Argo rollouts demo, we can increase the error rate and that stuff shown with the little red square and if we rerun pixie.

A

This is just looking at the last 10 seconds or five seconds here. We can see we're starting to get 500 status code and we can also increase the latency.

A

And we can rerun pixie and you can see that our latency has dramatically increased here. So we're going to use Pixi for our metric analysis and it does this all using evpf.

A

So now, I'm going to go over to Let me refresh this page, okay, so now I'm going to do the actual Canary demo um rollout, which again is in the Argo repo and basically I'm, going to like demo a roll out of a new application. By um let's see we are on the blue ones, so we'll roll out the yellow uh thing and ignore that gcpr, okay. So now up above here you can see this is the cube cuddle plug-in for Argo rollouts, and you can see our Canary. Is this Yellow Version?

A

The Blue version is our stable version. We've spun up a canary replica set here and it's running and we're going to do our uh analysis. Every 30 seconds after an initial 30 seconds, oops.

A

Oops sorry, there was the latest one okay and this green check here. Let me refresh this over here. This green check here means that we've done one analysis run and it's um succeeded. We should be seeing yellow over here all right.

A

Well, oh there we go okay, so now we can see that some of the traffic is being directed to our new yellow application version, and this is just going to keep running analysis in the background, according to our analysis, template, but we can inject some error and see what happens and, according to our analysis, template if we have more than five percent uh HTTP error status codes, we should fail. So this error is represented by these red boxes here and we've got to wait 30 seconds for another analysis to happen.

A

Fingers crossed this works.

A

Oh there we go great okay, so you can see. We had two succeed, um two metric analysis that succeed, one that failed and we're tearing down our Canary replica set and our rollout has been degraded and up above you can actually see now it's aborted and it tells you why, and if you want to see why it was you can go, look at okay, uh you can get the analysis run.

A

And you can see the actual metric values, so you can see our error rate for the first two runs was very low, but then for the next run it was at 60 percent. So keep switch back to the slides.

A

So that's showing you how you can do a completely open source, Cloud native Canary rollout, um so pixie can provide a lot of other metrics for all of the protocols listed. You can get latency error and throughput and you can also get those metrics, By Request path, so take a chance on pixie. Try it out today. It's another cncf project and we'd love to hear back from you.