Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Progressive Delivery in Kubernetes - Carlos Sanchez, Adobe & Viktor Farcic, CloudBees

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Progressive Delivery in Kubernetes - Carlos Sanchez, Adobe & Viktor Farcic, CloudBees

Progressive Delivery makes it easier to adopt Continuous Delivery, by deploying new versions to a subset of users and evaluating their correctness and performance before rolling them to the totality of the users, and rolled back if not matching some key metrics. Canary deployments is one of the techniques in Progressive Delivery, used in companies like Facebook to roll out new versions gradually. But good news! you don't need to be Facebook to take advantage of it. We will demo how to create a fully automated Progressive Delivery pipeline with Canary deployments and rollbacks in Kubernetes using Jenkins X, an open source platform for cloud native CI/CD in Kubernetes, and Flagger, a project that uses Prometheus and your service mesh of choice to automate Canary rollouts and rollbacks.

https://sched.co/ZesH

A

Hello and welcome to this talk about progressive delivery in kubernetes. My name is carlos sanchez.

B

And I'm victor farzig.

A

So I am a cloud engineer at adobe I wrote some open source a lot of it like the yankee supernet is plugging and I've been a long time open, source contributor at apache, maven, eclipse, foundation, puppet and other things and victor.

B

Victor works in cloud base and as a product manager, and then he published some books and courses and he's a google developer expert, all the good stuff, a lot of open source, a lot of public work.

A

And we're here today to talk to you about progressive delivery. When have you heard what have you heard the first time about progressive delivery? Victor.

B

I heard the first time from you.

A

Well, I heard the first time from lunch starkly they wrote this vlog about the history of progressive delivery, so that was the first time and then james governor from redmonk known chips in twitter. He was pushing it and I said: well I love this term. It takes. It defines very clearly what we're trying to do. So, let's steal it right.

B

Exactly we're not going to invent out anything new right right so.

A

What is progressive delivery is it's just a new term to coin a bunch of different things that have been around for a while. So it's groups together all these diplomatic strategies that try to avoid the all or nothing deployments where you know you replace something, and if it's broken it breaks all your customers.

A

So we are trying to avoid that and that's all these techniques fall into the progressive delivery password. I say so when you deploy a new version.

A

You are not replacing existing versions then, but you just deploy things that are running in parallel with the previous versions that you have to deploy, and then you can decide whether you want to send live traffic. This is important. This is all in production happening, and so you can send live traffic to the old version or to the new version based on whatever parameters you decide, and then you evaluate whether this new version is behaving correctly and if it's so, then you can send more traffic to the new version.

A

But if, for whatever reason, the new version doesn't behave as you expected, then you can start rolling back to the old version, and this is very immediate or fast and just a percentage of your users could get affected.

B

So the main advantage really of continuous progressive delivery or running in parallel.

B

It is really about avoiding downtime, because if you do not run two releases in parallel, it is physically impossible to be able to deploy a new release without downtime, so running in parallel is a prerequisite for any item to do no downtime or zero downtime deployments, and then, on top of that, we have all the benefits that carlos just mentioned, that we can decide how much traffic goes to one release, how much to another and increase the reliability and confidence we have in in rolling out something so.

A

It's all about avoiding downtime and limiting the blast radius. So something goes wrong. Just some people notice what the issue and also allows you to have a faster like idea to production type of workflows, where you are more confident that things are gonna are not gonna cause a big issue in production.

A

So there's a number of techniques that you may heard of this is nothing new, so we have like things like rolling updates. So this is something that you could have done with virtual machines. You can do it very easily in kubernetes, with containers with pods and basically it's just you have 10, contain pots running, and then you upgrade one to any version and instead of operating all the thing at once, you operate one check that that one is successful or to have some maybe metrics.

A

Then you operate two three four five and so on until you operate all of them, so you don't.

B

A

Of control, but at least you are not affecting all the users at the same time,.

B

So rolling updates is not that much about control, it's more about avoiding downtime by replacing one release at the one replica at a time. There is always a replica running bit of the bold or a new release or both. So it's not that much about having control you, don't really control who gets to see what and uh you cannot even control much whether you roll back or roll forward, it's more about reducing downtime than than anything else.

A

Yeah and as I said, this is very easy to do in kubernetes.

B

Then we have it's a default deployment. It's a default deployment mechanism in kubernetes. In other words, if you, if you use kubernetes deployment or stateful set, you are getting rolling updates, no matter what you, what you do, unless you explicitly change the deployment strategy to something else,.

A

And then we have longer in deployment. This is something that I think came from. Martin folner, I mean he was the first one kind of going through the definition where you have two different sets of what clusters or machines one set with the old version, one set with the new version, and then you have some router load balancer in front and when you want to deploy to the new version, you just flip the switch on the load balancer and say all the traffic goes to the new version.

B

The main issue with blue green deployments is that it somehow assumes that the best way to roll back or to roll forward is by having everything duplicated it assumes in a way, and simply because it was created long time ago. It assumes that uh rolling backward back, for example, would require. Creating of new virtual machines would require a lot of man. Many many different steps.

B

uh So in order to avoid the downtime, uh no matter whether we're going forward or backwards, everything is to run in parallel. But now, assuming that we are using kubernetes for most of our workloads, then the difficulty of going forward or going back is is basically non-existent.

B

So the whole idea about running everything twice in parallel forever and ever is kind of uh based that could be probably avoided. So it is much better suited in in environments where deployment takes time and scheduling takes time and what's or not,.

A

And then we have canary deployments, which is like probably the best way of doing things today, where you have a sort of life same thing of with blue green deployment, where you have two versions running at the same time. It's just that you don't need to have a whole copy, and you start you are able to start sending users to the new version and you can start sending like a few percentage at a time and based on different metrics.

A

So you could say typically what companies do is: okay, first, only employees access the new version. So if there's anything wrong with the new version employee is detected before it reaches to external users, and then you could start saying 10 or 5 and then 10 and then 15 20, 30 40 of users are going to the new version, while the other ones are stuck with the old version, then you keep rolling the percentage of users that are going to the new version and another.

A

The other percentage goes to the l version and you can stop this roll out at any time and start and switch it and send everybody to the back to the old version.

B

So the main difference, I would say, between rolling updates and country deployments, is that withdrawing updates, you're you're using very simple health checks, normally to decide whether to go forward or to go back without really control. Who sees what and how well with canaries. You would basically use um heavily rely on metrics to decide whether whether something is successful and if it is successful, whether to move it forward, another 10 20 or whatever the percentage is uh so it is. You can say even that country deployments is a form of testing.

B

Where you are uh you create you deploy a new release, you allow certain percentage or certain region or certain users to see it. Your test test, test and uh and by by testing, I mean evaluate metrics of our query, metrics, evaluate the results and then decide to go for forward or backwards.

B

So it is in a way similar to what will what we would do when deploying to integration, environment, staging environment production, except that this is all happening in production and uh the use cases or test cases are based on input from the real from real users rather than imaginary uh test cases.

A

And the last way of doing things is and that you can combine. This is interesting part about feature flaps that you can combine them with any of the other ones and feature flags and canary deployments make a pretty good combination.

A

So there are like future flashlight toggles, so switch that you can turn on and off anytime when at your application. Runtime. That's that's the interesting part, so you can deliver new features that are off by default to production and then you can enable them or disable them.

A

If there is any issue with it right, so you build a new nice, I don't know like ui and then you can flip the switch and start showing that new ui to specific subset of users, and if something is wrong, you just switch the flip the switch again and go to the old version.

A

So it's very nice because you don't even we were talking about rolling deployments and rolling applications and so on with feature flash. You don't even have to compile anything again or deploy anything again. It's all happening in production.

B

You still need to run multiple replicas of your application, do some form of let's say, maybe rolling updates uh to prevent downtime, but you do not necessarily need to uh to measure uh to actually decide who sees which release, depending on where you're holding forwarding traffic, but rather by enabling and disabling features in that new release.

B

So it's usually a combination of running multiple replicas in some form of rolling updates calorie deployments, also not combined with the fine grained ability to enable or disable specific features, while canary deployments are allowing you to enable or disable visibility or specific release. This is rather on the future level.

A

Yeah- and I think we we heard this before right- monitoring is the new testing, in the sense that, if you have good monitoring, you can know when users are experiencing issues in production and not only that, but you could react to the issues automatically. You can rely on metrics to decide whether you want to continue a canary rollout, whether you want to disable a feature flag because uh is broken, and you can do very cool things.

A

Unlike a devops board, I uh you used to say to make error: is human to propagate that or tool server in automatic way. That is what devops is, and this is the first time we mentioned devops in the whole talk, so that this must be a record.

B

I bet no other talks managed to get this far without measuring those and.

A

Something I experienced- and I learned from that is like if you have an automatically destroy something by mistake, that means that you are not automating enough.

B

Oh yeah, actually, if you want to be secure, the best way to do is to do nothing and then the second best level is not to do anything automatically. And then you get to automation.

A

So, let's go on to an example on how you can you practically do progressive delivery on kubernetes, specifically canary deployments? Let's talk about istio. If you haven't heard about istio, it's a service mesh, you probably heard about five different of them.

A

Istio is one and isti allows us to do very interesting things with um with your services in kubernetes, so istio connects services allows to secure those connections, doing like automatic encryption and also control the flow so deciding what traffic goes to what service and providing us observability metrics on these services, and the interesting part, is that you don't have to do anything to your application. You all get you get all these things for free when you run a service mesh like istio, similar with other ones, but this example just uses istio and another interesting project.

A

The most interesting project on the sense of canary deployments is flagger. It's a open source project that automates the promotion of of canary deployments using istio and prometheus metrics. So when you deploy a new deployment of kubernetes flagger, is there listening to events and sees the new deployment?

A

If you have an evolved flag around it on it copies that the plane will offer, so it keeps two versions of this deployment, the previous version and the new version, and it uses istio control to start sending traffic instead of sending all the traffic to the new version gradually rolling out from to the new version, and you can customize whether you are sending you know: five percent, ten percent, fifteen percent, on whatever number you want to set this new traffic, goes to the new version and the metrics from mr ending in prometheus and flagger looks at those metrics and says you know this metric is okay or is not okay, so based on those metrics, it does continue their rollout or otherwise doesn't roll back and start sending all the traffic to the previous.

A

Version, so it does this by creating virtual services in hto, primary and canary services and deployments of your application and continuously looking at istio virtual service, metrics and prometheus, and you can even get messages in slack when when these rolls are rollouts are happening, because I mean who doesn't need more slack messages.

A

You also get graffana a girlfriend dashboard where you can see side by side old version, a new version. So again all sorts of things like response errors response times, and you can see both all version and the new version.

A

And it's really cool it's a really cool project. You should totally check it out.

B

Yeah one additional note is that uh you were hearing about uh flagger in combination with istio, but if that's not your flavor, that's okay, because flaggers support basically almost any service. This is a serious mesh available. On top of that, it supports uh even without service mesh. It can work with engineer singers and a few others so think of eastern. This context is example, not necessarily the requirement for.

A

Flagger and now let's go on to the demo.

B

Yeah demo time, let's take a look at flagger definition. First, it is kind connery that that's not really anything special. What matters is is that we are setting provided to istio. In this specific case, we are specifying that the target reference, the application- is running as a deployment and it is called go demo. Seven flagger will be monitoring, metrics, impromitus and, depending on the outcome of those metrics, decide to move forward or roll back.

B

We have an application, it's a kubernetes deployment, we have flagger and we have istio gateway everything else required for that application, like uh destination rules and virtual services and whatsoever. All that will be created by flugger out of the box. The moment we uh deploy the first release. Actually we already deployed the first release, so we're going to see how the second release comes along because that's the more more interesting part. So, let's take a look at the deployments now.

B

This is important, because here we can see that we have a database that one is not imported, but we have got demo seven and go demo 7 primary now go demo. 7 is the deployment that we deployed.

B

The other one the primary is created by flagger, and that is actually the release that flagger will consider the production release, while whatever we deploy ourselves will be uh copied by flagger and converted into next production release when the time comes, even if all the conditions are fulfilled.

B

Another thing important here that we can see that we have hpa horizontal pod autoscaler, and here we have services that are already that were already created by flagger right all those here except the database were created by flagger managed by flagger. We don't need to do much over there and the most important one is canary resource, which there is only one right now and it is initialized.

B

A

Mean that's the beauty of flyer right, you all! You focus on your application and then it will from the crd, the canary crd. It will create all these things for you magically.

B

So what we're going to do next is deploy release o2, and I will show you here the yaml. This is basically exactly the same yaml. As of the initial release, the only the only real difference between this deployment and deployment it is currently running in production is in the tag on the right side. You will see very soon. A loop request will be sent to the application loop so that you can see how it progresses right.

B

The current version of the application, the one running in production, is supposed to output, something something version 01 and, as scandal restarts rolling out the new release. You will see increase in of the percentage of the outputs uh that state that the release that is running is all volt2.

B

Okay, so as as you can see, all the outputs are from the existing release from it says version one. So what we are going to do is apply the new deployment, the deployment of the new release. So the only thing that we are changing in the system ourselves is updated version of kubernetes deployment, we're not changing anything else, we're not doing any other action except deploying uh overwriting the existing deployment, with the different image with the different tag, and we can see that we have two deployments the new one, all seven god demo.

B

Seven is currently running zero replicas that will soon increase when horizontal product scaler kicks in. And if we look at the virtual services, we can see that we still have only one.

A

And this makes a lot more sense when you have multiple replicas and even the hpa, the horizontal polar to scale it running. So you have more.

B

Room and if you look at the service right now virtual service from mystery. In this case, we can see that there are two destinations. There's the two routes, one sending hundred percent to the existing production release or primary and zero percent of the traffic is being sent to the country release to the new release and the reason for those zero percent is because canary flagger did not kick in yet soon. You will see the change of that percentage once uh new deployment is operational and flagger discovers that there is a change of that deployment.

A

Yeah flagger, basically, is stopping your new deployment from getting accessed and then slowly turning around for the users.

B

And now we can see that the percentage is changed. 80 is going to the production release. Why 20 is going to the new release and if you pay attention to the right hand, side of the terminal, uh you will see that occasional outputs are now saying version. O2, approximately 20 of one out of one of every out of every five requests are responding. Currently with version o2.

A

And this is random, but you could also set by like http, headers and other sort of rules.

B

Exactly currently rules, if I remember correctly, the definitions are based on uh the error rate. I think, or something like that- uh simple one, but yes, definitely in real production usage. You should change that to be real metrics, real queries, real error, detection, user satisfaction or whatever else you deem important to decide whether to move forward or not.

B

So let's describe canaris one more time, and now we can see that it increased to calorie weight increase to 40.

B

So it saw that there are no problems, no errors, no issues with the new release and it decided to move forward another 20 and if you pay attention on the right hand, slide term inside of the terminal, you will see that approximately 40 percent of the responses are version 02, while 60 percent are version 01, and we can see that canaris is now progressing and 60 are now 60 of the traffic is currently being redirected to the new release by default, and this can be changed once it reaches 60.

B

It is configured to finalize the release. It assumes that if 60 of your users of the traffic or whatever the criteria is uh has been valid, there is no point probably to continue progressing to 80 percent. It will jump straight to 100 and penalize uh the deployment of the new release so soon within a few moments.

B

All the traffic everything coming into this application will be redirected to the new release.

B

That's about it now it is promoting meaning that it is removing the country release and it is changing the primary release to be the new release, and basically this is it even though the process did not finish yet. Basically, it is finishing our application. All the traffic of our application is now going to the new release and you can see from the output on the right hand, side of the screen that all the responses are now all too.

B

That's it. We're successful. We've proven that it works in production, just apply applied, and that's it. Okay,.

A

Well, thank you for listening to us today and we hope you learned that you can do progressive delivery canary deployments on kubernetes. It's very easy. You don't have to be netflix or facebook or a big corporation. To do this. You can just use open source projects to do it right. Victor.

B

Absolutely basically, it's almost too easy. The main difficulty is whether you really understand under which conditions uh a release is successful, because you will need to tell in this case flagger what those conditions are, and that is probably the most complicated part, because it's very easy to say: okay, just move forward twenty percent, every five minutes, but figuring out what are the potential errors that the machine system needs to discover in order to decide whether to move forward or move back is?

B

Is the complicated part outside of that? So if you know your system well, if you know under which conditions something is a success or a failure, then it's fairly easy.

A

Yeah- and it's not only metrics about you- know http traffic. You could also have your own metrics, like business level, metrics like how many items are people putting on their shopping carts or how many uh credit card transactions we have, so you could also- and it will be really nice to have those type of business level. Metrics that mean more to you,.

B

Exactly well with all that, okay, so thank you. Thank you, yeah. Thank you for.

A

Being here virtually and see you next cubecom.