keptn Keptn Tutorials, 1 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Tutorial - Canary Releases with Argo Rollouts and Keptn

Description

Canary deployments are a very popular progressive delivery model where new versions of a software component is safely released in a "phased" approach (e.g: 25%, 50%, 75%, 100% of traffic) instead of a big bang (100% of traffic). Between each phase an evaluation takes place deciding on whether to continue with the next phase of rolling out the canary to more end users or whether to roll it back to the previous version.

Argo Rollouts is a very popular open source progressive delivery controller for Kubernetes (k8s) and provides both Blue/Green as well as Canary rollout models. If you want to learn more check out their introduction video of Argo Rollouts: https://www.youtube.com/watch?v=hIL0E2gLkf8

Link to tutorial: https://tutorials.keptn.sh/tutorials/keptn-argo-rollouts-dynatrace-08-on-k3s/

Learn more: https://keptn.sh
Get started with tutorials: https://tutorials.keptn.sh
Join us in Slack: https://slack.keptn.sh
Star us on Github: https://github.com/keptn/keptn
Follow us on Twitter: https://twitter.com/keptnProject
Join our Keptn community newsletter: https://keptn.sh/community/newsletter/

A

Hi, I'm andy grabner part of the captain team and I just finished a new tutorial called multi-stage delivery with captain and argo rollouts check it out. This is a quick overview now. Why have we built this because argo rollouts is really great in blue greens and canary deployments with their operator. Captain, on the other hand, brings a lot of things around delivery, automated testing, automated validation, automated notifications, automated monitoring configuration which argo rollers itself doesn't bring, which you would need to bring in anyway through other tools.

A

So what the tutorial includes is really a way to show you how captain and auger rollers can be used to roll out new versions then use the canary capability as well as the blue green capability. Captain can then bring in automated testing with different testing tools can also make sure that the observability is correctly configured whether this is prometheus or any other tool will then do some advanced, slo validation continuously with every rollout and then with our promotion assistant. Allowing you to say: do you want to automatically promote into the next canary phase?

A

Do you want to roll it back, or do you want to have this automatically decided, or do you want to actually have a human to still make a decision, depending on your slo validation? Captain, as you probably know, is an event driven system. That means you can also integrate your chat ups tools, and we can also do all this multi-stage all right, so this is kind of the overview of the tutorial check it out now quickly. I have followed the tutorial, so if you go to the tutorials page, this is exactly what I'm showing you.

A

um I have followed all of my instructions here. This is part of my tutorial series captain on k3s, so the only thing I need is really a linux instance. In my case, it's an ec2 instance and then I walk through all the different steps to just install it. Now I have just installed it. You can obviously install this on any kubernetes cluster, but I just ran my installation script. The tutorial installs several use cases, the one we are focusing on right now is the one that's called canary delivery use case with argo rollouts.

A

So this is the section of the demo. The first thing I want to do is I actually want to kick off a deployment of my build number one. I have a sample application with four different builds, and I'm now telling captain please trigger the automation sequence for delivering this particular version, start in staging and then go on now. Let me quickly walk over to captain, so this is my captain install the tutorial, as I say, includes multiple use cases.

A

The one we are focusing on right now is the demo rollout before I go into this project check out this here. Captain internally holds a git repo for every single project and I've also included kitty as a git client, which means we have here access to all of the different git repositories right. So under captain these are all the different projects and the one we're looking at right now is the demo rollout.

A

You see here the shipyard that I've used. This is the shipyard file that specifies what are the different automation sequences that I want captain to automate. I decided to do a two-stage delivery in staging. I want to do some delivery, where I want captain to first configure my monitoring monitoring is code. Then I want to do a deployment then, because it's a blue green deployment.

A

I want to release the blue green. Then I want to run some tests, some evaluations, and then I want to have the smart approval assistant to always ask me whether it should go into the next stage in case anything goes wrong. We do a rollback if everything goes well, we go into prod where we again have a delivery stage that is triggered on the success of my staging delivery, where we can do deployment test evaluation approval and then it goes into the individual canary phases.

A

So this is the sheepbed file check it out very important, because this is an argo example. If you know captain, then captain creates a branch for every stage so for staging.

A

I have my one service that I've onboarded called simplenote captain is organized that every service has its own subfolder, and in here we have the configuration specific files for the individual tools that we use. So, for instance, our rollouts are specified in helm, charts. So I have a helm chart and in my template file I do have my rollout definition here for my service. So let me click on rollout, so you can see here.

A

This is for staging for my simple node environment and here I'm using a blue green strategy with the active and the preview service. Now, if I quickly switch over to my production in production, not using blue green I've decided to use an argo rollout with canary strategy with these steps. Now the nice thing is: captain takes care of the deployment because captain can trigger helm, and then we have a captain argos integration. Captain algorithm that takes care of the individual step.

A

Promotion in between it will do testing it will do evaluation and it will then include the smart delivery assistant where we either do automated promotion, or we can also ask a human being now, let's have a quick look at what captain is doing so far. Let's look into the project demo rollout we have staging, we see delivery is currently going on. So this is the overview of my stages, so I get a feeling what's happening across stages. I also have a service specific view.

A

I have my simple node view and I can see here that in staging something is currently happening here, I would only see the information that is really necessary for me to make a quick decision of. What's going on, I see my quality gate evaluations. I see any approval steps. If I click on few sequences, you can see we're switching over to few sequences. I can now see every individual task that I specified in my automation sequence.

A

It started with configuring my monitoring doing the deployment releasing testing a and the evaluation just completed with 100, which is great. Now let me actually show you I completely forgot about that. I have a little file here. That shows me on the left side, what's happening in staging right now on the right. I have four iframes that are constantly refreshing and showing me what's in production. Now nothing is in production right now, because this is build number one.

A

Let me approve build number one now, because it seems we have achieved 100 points, so we can now approve it through the smart approval assistant into production, which means now the captain workflow continues. Captain is now moving into the production stage and in production is executing the individual deployment, sequences or automation sequences that are specified, starting with deployment which, in my case, triggers helm.

A

If I go back over to my file here right, we can now see, because these are refreshing every three seconds here I can see build number one is currently deployed in production already now. This is pretty much it for the build number one, because with argo rollouts, whether you do blue green well, there's clear, but for canary, so the first wheel will automatically be promoted to 100 of your replica set. I will let this run through all the way, and once this is done, I come back.

A

Show you the result at the end, and then we go to build number two all right. I quickly click on pause here. All right captain finish testing in production. Everything is good now it gives me the approval option, because that's what I specify but remember, build number one is a complete rollout anyway already so there's nothing really. I need to approve. We got we're all good here. Let's go and deploy build number two I'll, deploy it again or kick off the automation through the captain. Cli now remember.

A

I could also do all of this through the captain api. This would allow you to integrate captain with your let's say, jenkins, uh gitlab azure devops. We see a lot of uh users do this because we have an api that does the same thing in the back end. Now, let's see what's happening now, I just kicked off build number two monitoring, as code has kicked in configuring, monitoring deployment happened, uh which means, let's see here, we go build. Number two is already in staging remember.

A

This is just my sanity check of five iframes on the left staging on the right four in production. I just see what's happening, it's a refresh of every three seconds. This will be reloaded now. I will pause again here and see if captain thinks that build number two is good enough to be deployed into production because remember now, tests are executed, quality gates, slo-based quality gets evaluated, and then I get a notification whether I should or should not where I can or cannot, deploy or promote into this next stage.

A

All right I'll be back in a second and we're back, and we can see that build number two is still in staging, not in production. Why? Well? Actually we will never get there, because captain told us that this build is no good for production. We only achieved 50 points, which means if we have a look at it.

A

This is the overview of our slo evaluation, the quality gate- and we can see here that error rate is at 12 12 and a half percent, and the criteria, though, is one percent. This is why captain says this is no good build at all, and in this case we don't even get the option to promote it. This is the way we have configured the smart uh deployment assistant.

A

So, in our case, what we do- and this is also in first of all- the intended build number two had this built-in problem to really show you that captain can be used multi-stage to stop bad builds early on now. Let's try build number three and see if build number three is worth it to go all the way in build number three: let's kick it off.

A

Let's also have a quick go back to demo rollout right. This is also nice here we get always get a nice overview about what's currently running in production, what's happening in staging the service overview. I cannot stress this enough. This is really great because we can see build number one build number two build number three. We get the history, we get the high level overview of what's important like I get. The slo results here immediately see in staging there's just no way that this build goes into the next stage.

A

Now build number three is going on. We will see the high level overview here soon. If you want to see the details, let's click on view sequence and we see exactly what captain is currently executing and last energy check. Before I go into a quick pause, build number three is in staging. I will pause here until build number three is tested in staging and then we'll release it to production, and we are back build number three has made it through the quality gate.

A

We achieved again 100, so this build is good to go and we can improve it now. I'll show you we can either approve it here. I told you. The overview here on the service level also makes it rather easy, because we see the quality gate in.

B

A better overview immediately and then approval.

A

So now we approved build number three into the next stage, which means it should start in production.

B

With remember deployment, then testing.

A

Then evaluation and then in this case it is a canary rollout. So what we should already see if we go over here, we have a 25 percent uh rollout phase for the first phase, which means on average now, because these are four kind of iframes. I should see about one lighting up with build number three and uh yeah. This is now the first phase. The first phase will take about two three minutes, because captain will run some tests will then evaluate and based on that again give me a decision.

A

Do you want to continue with the rollout here? I'm also going to show you the smart delivery assistant. So let me stop here. Until captain is done with the first phase of testing all right- and we are back- captain has finished on production. That is the first uh rollout. We got a 100 quality gate and we got an approval, so we can approve it to roll out further. Now. This is something I've built into the shipyard file. Now remember.

A

If I go back, the shipyard file is that what defines all of the different sequences, so you can check it out ship, your dmo. This is where I've specified my staging delivery: automation, sequence in production. I've also specified for delivery. Hey start with the deployment then run some tests. Now the deployment will immediately roll out for the first 25.

A

Then an evaluation will happen, and then here I have an approval and I've specified it to manual. If I want to, I could even say if it's pass immediately approve it so that the next canary phase starts. I just showed you that this is an option here, where you can configure the approval process to have more human interaction or less human interaction, but everything is based on the result of the previous evaluation. So now going back to captain, I have approved it. Now captain goes into what I've called canary phase one.

A

You can call this whatever you want. I've called it canaryface technically, it's kind of the second canary phase already, because we are in the second wave, which means now. I should see a higher chance of more yellow, more build. Number three, because now we are going to 50 all right good. Let me pause once more and I'll see you in a couple of seconds and we're back, and you can see now build number three is completely rolled out in production. Now, let's have a quick look at what captain has been doing.

A

Remember we kind of went into canary phase one canary phase two now we're in canary phase three already there's still some tests going on, but let me also show you the release task here is the one that is actually interacting with argo rollout, that's the argo service that we've created and what we do here. This is the service that is then advancing to the next step. So in this case we're basically doing a promotion and we also get the output of the rollout.

A

So this was for this particular step, and if we go to the last one here, this should then basically say hey we're at step six from six and it is just switching over from the remaining pots that are still on build number one to build number three right, so this is really cool. Now I really encourage you, you know, go and run.

A

The tutorial also run build number four, because with build number four, you will find also a new use case, something surprising something it makes it in production, but maybe there's some issues in production and then you see the power of captain rolling back in the middle of a canary rollout.

A

Give us feedback join us on the captain, slack and together. I think we can even make this better come up with some new use cases, but shout out to the argo team, great stuff that they're building shout out to the captain team amazing stuff together, we can truly bring value to those people that needs to automate multi-stage delivery with automated testing, automated evaluation and all the integrations with your different tools that you have in your pocket. Thank you. So much.

B