Cloud Native Computing Foundation ArgoCon 2022, 21 Sep 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Argo Rollouts at Scale: How We Brought Automated Rollback to 2100+ Micro-serv... Joseph Pallamidessi

Description

Argo Rollouts at Scale: How We Brought Automated Rollback to 2100+ Micro-services at Monzo - Joseph Pallamidessi, Monzo Bank ltd.

At Monzo, we massively invested in the micro-service philosophy and shipping small and often: we have 2100+ services and deploy over 100 times a day. We have great tooling and a fast deployment pipeline, yet our deployment strategy was still trailing some of the modern best practices. We were too reliant on testing, alerting, and engineers discipline: as a regulated bank this exposed us to a lot of risk, fines and potentially having the UK financial regulator taking action. This talk describes why we chose and integrated Argo Rollouts with a complex and established backend. You will see how we migrated 2100+ micro-services safely with minimal change despite the heavy customisation of our cluster. We will address how we improved our tooling to make the experience seamless to the backend engineers, and moved to a world where automated rollback became the default strategy (and not canarying !) By making deployment safer, we guarantee a better outcome for our customers: they will see fewer issues when using our platform. For our engineers, that means less time to deal with incidents, more time for delivering value, and reduces the anxiety and cognitive load of deploying.

A

I think we can start yeah perfect, um so hello. Everyone thanks for joining me this morning, I hope you're, having a great time so far, and today, I'm gonna talk about our Google Earth at scale or oh, we brought automated rollbacks to more than 2 100 micro services at monzo.

A

So a little thing about me and why I'm doing this talk in the first place, so under the paramedic, a picture of me with like much more air um I'm, a back-end engineer working in the platform team at monzo I like this migration project, where we essentially replace our default deployment strategy to basically use our goal out and automated rollbacker countering by default.

A

Some and I was detective of the engineering Effectiveness team, as we have a couple of discussions this morning during the keynote, so the Medicare team doing everything in the backs all the tooling that engineer used to use to interface with the platform and also deployment pipeline, CI, CD and observability stack uh and I. You could probably hear already I have a strong accent and unfortunately, I cannot do anything about it. So please bear with me um so first of all what I'm gonna talk about this morning. um This is a study case of oh.

A

We make it migrated all of our services to our Google out, as I said, they can become like a default deployment strategy at monzo. uh Oh, we did it safely because we're a regulating Bank in the UK um and them of the bank is is monzo and that we're actually quite happy with what we uh what we got and what we achieved with our Google art, and there will be a lot of emojis because it's part of monzo branding, so uh so yeah a little bit uh scene setting a little bit of context.

A

So what is manto? It's a regulated Bank in the UK. It's seven years old. It has more than 5 million uh customer. It's one best, British Bank um this year, I believe um it is a mobile only banking up, as you can see um like what the basic art look like and if you've been in the UK you've, probably seen those very flashy, odd, Coral cards that we're famous for and that. Basically, um what's it look like from the tech point of view, which is uh yeah? The tech stack point of view.

A

The more interesting bit is uh monthly, really brought in the micros into the microservices philosophy. We have more than 2 100 macro services or application, depending, if you recall them, uh we have over 200 Engineers um a good most of them back engineer and that's where, like the interesting number come in, we did last year, 20 27, 000, Pro deployments and those are actual Pro deployments. This is not conflict change. This is actual code that we push to and to production.

A

Everything is going and the other interesting bit is everything is running in in one Mega Community kubernetes cluster of like 300 nodes, uh 20, 000, pods, um massive scale. It's it's a really chunky, uh Community cluster. Another entire platform is quite centralized and quite uh homogeneous. Every golden Services looks the same and basically use the same libraries.

A

um So as I mentioned basically deploying and monzo is a bread and butter. We made that process like incredibly efficient if we uh like artificial, as fictionless as possible, for engineers to be able to deploy, um and basically our deployment pipeline is enrolled uh and you can deploy um in less than two minutes and that's include building the images. So it's very efficient, but we kind of like realize that really it's something we do like soft and on a daily basis or the percentage was kind of simple.

A

We still rely on communities, deployment, Community native deployment and use Like, a Rolling update strategy, which basically means that the engineer when you deploy stuff and because we do it so much you kind of like needed to we had like alerting genocide and also like um business level alert but Indiana kind of, like add to BBC deployments, look at graphene dashboard, which is, as you know, not super interesting and also with a current setup. Every time you do like a new release, you basically release it directly to 100 of the customers.

A

um As I mentioned, we are in a banking is a regulation. Industry downtime is very costly, uh financially in terms of reputation. I mean for all of us in this room like you, don't definitely do not want done time, uh but for us it's also bear like the risk. Some like regulatory risk, uh if we're down too often or if, like some very important payments as a system, are down for a long enough time. That might actually show you some regulatory action and those are quite unsorry. We kind of want to avoid that.

A

So, in order to do this, we kind of like try to try to stick to like um best practices, and we have like this cheese approach to like change management.

A

So uh we try to catch badge changes, but 10g is an error at multiple layer um could review um two lack of CI pipeline uh having a staging environment that is actually quite close to the production environmental editing, but we kind of realized that we did not add anything after you deploy you basically deploy and then it's it's a new except for, like obviously a smaller thing, so we kind of found environmental deployment uh safer.

A

We wanted to reduce the connective overhead of like deploying and also to a lot of like automation, so just like continuous deployment in short I can make the point safer and quicker.

A

um So we onboarding this uh Progressive volat initiative and the big idea at the beginning were to move gradually move traffic gradually to a new version of the code of the club and also uh automatically rolling back, but a bad change. If that happened, um I need to do like a bit of introduction about our Google ads. So this is like stolen from the from the project documentation.

A

So algor alerts is a kubernetes controller and set of CID which provide Advanced deployment capabilities such as blue, green and Canary Canary deployment, and um what I got a Google ads? More or less is it's. uh It can be used as a replacement of kubernetes native development objects that basically essentially provide uh more feature, and you know those uh Advanced um deployment capabilities such as doing uh staged stage rollout to kind of ring and things like that.

A

But in essence, it can be used either as a replacement for kubernetes Native deployment or alongside them by essentially referencing an existing deployment uh development objects.

A

So this slide is a bit like there's a lot of things going on um what I really wanted to like convey here. Is you have the rollout controller? That's basically managed a different, followed object, but the central thing of um I go out. Are the rollout objects, like, like I say, can be used as a replacement for like Community native deployments? That's why you default. You can Define like your um spec um annotations, but also where you define.

A

What's the deployment strategies would be like like, for example, a canary shift traffic 10 for like the first five minutes and then like 100 after that, and that, where you define all those things, that's also where you can Define, like some analysis, so an analysis essentially like a condition uh using some Metric providers, some metal provider in our case permitters or you cannot Define rules such as a rollback.

A

If um you know that metric is above a certain threshold for like a couple of this amount of time and so on, and so on so yeah in essence, that's basically what a Google ads look like if you've been to the workshop, yes or yesterday, you probably played with it and you put them so you played with it a little bit and that basically, all there is to it, which is great so great and simple tool.

A

um And what did we decided to choose our goal? A lot in the first place is uh because it was easy to understand and experiment with uh very well maintained and well documented, and it kind of had this kind of like Plug and Play quality to it, as you can just like, install it alongside your like existing cluster Oculus has, like I, said, very big, very complex. We pushed a lot of like business logic at the community level, especially every single related security.

A

um We have like a custom like Network isolation. Capability built into I was a community leader at monzo, but even through all of that, you can just easily add a Google out and you don't need much requirements for it to be working. Fine. It worked better if you have like a service mesh that allow you to do like final grain uh traffic shaping.

A

I find our grain traffic shaping, but even if you don't have a service mesh, it will work like traffic shipping still work. It's just like a bit more accounts and it's highly customizable. You can basically Define a custom rule for each of your, like rollouts, for each of the services, which is what we are looking for in the first place.

A

um So now we need to talk a little bit about like our journey, how we uh all we did get to automatically roll back at monzo.

A

um The first thing is a bit of a bait and switch so I talk about kind of being a bit, so coloring is other one. Economic deployment is when you shift traffic like progressively uh to a new version of the oven up um and that's why we're looking for in the first place, but then we ask ourselves the question: is that really what we want? Is that really what we needed, because we're going from like a really simple process of you, know, engineering and not having to like think about that could be rolled back.

A

uh You only have like one couple of assumptions. You only have like one version of the code like running the cluster and so on, and so on, so something much more complex and there was a clear need to like you know, educate, engineer and the rest of the of the org and those are consistential tooling and we're like. Is that really what we want? Is that really what we need right now um by the way?

A

This illustration is made with Daddy, which is a game changer when you actually need to put illustration in in your like presentation and as you can see, with this kind of fluently, giant, Canadian giant cage.

A

So um you know reviewing canopy deployment, we're like active. What we care is reducing the impact of a bad change and what we're really actually looking for is automated rollbacks. So that's what we decided to implement. First, we kind of using a blue green, like strategy. At the moment it still use Argo canopy strategy, except you still go straight to 100 of the shift, the new version to the 100 of the traffic.

A

But you have this analysis period where you basically run the stack to know if you need to roll back or not, and we did that to basically allow for us to move to real Canary in the future, because we got most of the infrastructure in most of the work in place to be able to do that, basically going instead, going from, like ruling updates strategy to just having automated rollback and later on, move to um canary- and this is what the deployment workflow look in this new world essentially and essentially an engineer started deployment that updates the World objects.

A

The new version of the app become live, then for like a short period of time. This is analysis, step uh where, like the different rules like because I'm rollback rules um evaluated and if the successful the engine is notified of success, we do that in slack, because we also like um we use basically a lot of flag stack UPS. We have a lot of processes in slack and, if it's not, if there is one of the rules actually evaluate to false, we automatically roll back to previous version where we notify the engineer and the team.

A

um This workflow allows us to choose a strategy: uh the different strategy per service and per deployment, so for each individual deployment that you make that allow you to potentially change what type of Department you want. Do you want to donate it rollback? Do you want Canary in? Do you want something more fancy in the future and that's what this flow allow us to do so again. This is probably not super interesting right. Now, it's more like if you're reviewing this talk later on and want to like look more about the details.

A

um This is what it looked like from like an architecture, point of view. So, as I mentioned, everything or the print pipeline is enrolled, so we have a service that basically generates a manifest, and so you think, Engineers use, CLI, tooling called cheaper to to get a deployment. That's trigger. That's basically make one of the service create a new manifests for it. Put that negative people, and then this is applied to another like Services another service. That applies that to kubernetes and updates about objects and then creates a new replica set and so on.

A

So basically, this is just to prove that we actually use githubs and yeah. There will be a complementary blog post about this talk with, like all the nice gritty technical details, um but all the aspects of the work that we did um for all of you yeah. If you actually love the ammos.

A

um So that's what the tooling is look like. So at Momo we have like this philosophy of back-end Engineers should not know about the platform. They should be able to interact with it, but they should not actually need to know things about. You know kubernetes, like networking and or even Auto scanning.

A

um So we have this one tool called shipper to deploy and the output is not super interesting here again here, but what we just wanted to show you is you basically just deploy by calling shipper, giving it a commit, Ash, a name of service and deploy it for you and we basically completely abstracted away Argo a lot.

A

Our Google ads itself provide like great, tooling and great dashboard, but we decided not to like expose that to our engineer and basically abstract all of that away, which basically required us to require to integrate with our goal out in a slightly different way same thing here. That's basically what our stack notification look like when you start a deployment. I basically tell you. Oh, you started your payment with this strategy. Here's a dashboard to by the end, the logs.

A

If you want to see what's going on, um and um you know, and the notification kind of success, of a successful deployment or like failed deployments, um so I talk about rules and rollback rules, so we actually started with only one rule and we try to keep our rule as simple and generic as possible. At the moment, all the services in our backend use assembled back rules, uh so they have to be like quite generic um and we started basically with.

A

um If the total um error rate of a facilities is above a certain threshold, then you need to uh to roll back um and about the rule themselves. We decided to basically tweak them and add them based on our incident handling process. So basically, every time we have an incident or something that looked like an incident. We of course have the question: should that be caught at deployment? Time of could that have been called a deployment and, if, yes would have, we made a rule to actually catch that issue.

A

The issue with like rules when you come to electron, knowing an automated rollback, is a false positive, is annoying for your engineers, because the change the deployment is well back. Customers should not see much problem, hopefully, but it's really annoying and it's not quite a growth ux and you don't definitely want to avoid like a false negative, where you're not rolling back, where you actually really should see.

A

So the takeaway here is basically, if you're planning, to move your entire organization to something more complex like I'll go out, do it in Step start as simple as possible? You don't need to go from, like um you know, simple process to something like state of the art, with like very complex Canary strategy, uh just start with what actually matter for you.

A

Org, and in our case that was automated rollbacks um ago, is relatively easy to integrate, as you can see with our Custom Custom tooling, because they provide a lot of thing out of the box like it will choose notification managing the ago.

A

Rollout other controller like metrics, like the robot metrics and it's CLI and dashboard, are quite useful when you're, like in development phase in the development phase, now I'm going to talk a little bit about audio, actually migrate to our Google out and, um as I mentioned before, Argo rollout allow you to do it in a manicure mental incremental fashion, where the rollout objects essentially reference an existing deployment.

A

So the world object essentially defines the strategy and a couple of other parameter, but the put template, spec and everything related to the actual deployment is still on the deployment object and as well as just reference it, and this allowed to basically migrate travel ago a lot in a safe manner. That's avoid on time, because you essentially just bring a new rollout object that reference the deployment object.

A

At that point, you have two replica sets one managed by the department, Resource One managed by the world resource, and then you can safely decide to set the number of replicas deployments to zero. So you just have like a replica set both managed by the rollout object and doing this. It's all over, like you know, downtime immigration, which is great and also doing it. This way, allow you to easily revert uh reverse the process, if you don't want uh wallet again or you're, just in an experimental phase.

A

um So that is great. That is super easy to do. Cube CTL command, probably like two three minutes relatively safe to do, uh except as I mentioned earlier, we have like 2 100 applications, so we need to do that a lot more, which kind of make us.

A

Actually, there was a clear need to or to the automated process, and through this automation we also decided to like unsolve the tone Force, even more safety, because, as I mentioned earlier, we are regulated Bank, who cannot really allow downtime and um and also part of our like Company motto, make money work for everyone, and there is nothing more like a scary for our customers and having the card not working in the supermarket, because you know product is down.

A

um So we wanted to make to make sure like we're. Extra safe and the world process is repeatable observable and inappropants.

A

So that's what basically, what the immigration looked like for us is. We first notify the service online stock. We create a new product object for that service that we're migrating with freezy deployments, because we want to limit the amount of change we'll be doing this process, which is not available.

A

And only then we decide to scale down the deployment to zero replica because we know that we have enough enough rollout. Maybe we can manageable by the by the balance object um and we then checks that we don't have any deployment pods um but managed by the depends, object left and then, if the establish using HPI, we switch the target of the HPR from the deployment to the rollout and for the deployment notify people of owner of success.

A

um That is great. So that's, basically what it the process looked like and we decided to like execute on it by going before, like a migrator part, a migride immigrator pattern where essentially all the step I just described, are Independence, um because certain other pattern that migrator said is that we're using? uh Will we try all the steps.

A

Without so much services to migrate, we only wanted to do them like doing business hours.

A

uh We want to spread the immigration across like the entire day, because we want to avoid adding too many too much per turn in a short amount of time, for example, and just you know, to basically control the amount of of risk um and we yeah develop that through, like immigration service, where the user decide which side is to migrate, and then we are the Chrome every day that um triggers immigration for the services and supporting them out through the to the day. We did it by using your service tiering.

A

So we went for like the lowest tier services to the IST services, so the less important service to the most important service and by the time we actually got to migrate. Our TSO services, including you know, The Ledger service, which is you know, no big deal like just managing like 4 million quid.

A

um So you want that won't not be down, because if it's down all the payments um or the payment system is actually done um by the time we got to those tiers, there was hard disease. We already immigrated with that process more than 1700 services, so we were quite confident about it being quite safe, um so take away if make it safe, automated like. If you automate it, um you can unforce like some additional like safety and safety rules.

A

um Obviously, if you have like 10 or 20 application, you can still do it by hand. It's probably much quicker, but as as, if your like backend is like big enough, that might be worse actually um adding some automation.

A

um The migration is actually what took the most time uh the act of integrating is actually what to do the most time, because we had to do it slowly and carefully um so and and where to invest in tooling and monitoring, to make sure that we could do this process safely. So in terms of like deployment could be like half the time was just actually running the immigration after we developed and developed everything.

A

um So what did we learn like moving to agoral Art and running it at this scale? um First of all, some numbers because everyone like numbers, so we have in our production cluster right now: 2143 rollouts, um so 2143 Services running a production. Probably a couple more inside, like this I did the slide. Like a week ago.

A

We have seven rollback rules that cover like some things that are very specific to us, but um are they enough to catch a large class of large classes of problem and we've been running our Google ads in production for essentially every services?

A

For like four months now, which means we did 7000 production deployment with a global out and 22 000 production deployments in staging because I told you, it only did 27 000 production deployment per year, but the truth is we did like more than 100 100K different staging last year, so yeah we have like we built like the confidence that our Google art is actually capable of operating at this scale, um and we are like 63 rollbacks, um since we basically um start using our Google out um yeah.

A

So a little bit like less than one percent. um A couple of them are like full uh for enough false positive. That's why there is a new equipment to actually kill your roads.

A

um So what about operating role with a scale which is like the whole point of this presentation right?

A

um Actually, there was very little thing for us to do it just kind of scaled uh really well, with all kind of like unconventional architecture, like it's very unlikely that we have like one kubernetes that big and that many people out running when, um but the aguilac controller did not have any issue with our scale of like operation at monzo, which was quite surprising. We even had a small incident in staging where we did like 5000 departments over the course of like three hours, and it worked, which is great.

A

All the stuff broke black Primitives, but because of the per term, but other than that. It actually worked quite well at, like this very large scale.

A

um There's a clinic to like share the knowledge about our Google out, because it's become a new critical component that your opt-in platform team need to understand quite well um and as well as the rest of the organizations that need to understand that the process is a bit different.

A

Now um there are still one little performance issue about leftover, like analysis for an objects, but we are in the case and we hope to be able to push a PR like soonish to actually fix that issue um and uh yeah, and it's important to invest into our lighting and observability from when operating ago, Alert in general and especially at this scale, and that the project make it easy because it already provide like most of the metrics. You can think of that. You would want to to monitor.

A

um Would we do it again, and should you the answer to that fast? But the question is yes and the answer and for the other question? Yes as well. So for us, it was a clear positive outcome. uh We would use connective uh the connective load of like deploying we catch loads of issue across like tons of deployments, uh we're quite happy with what we got in return and and also with all easy. It was to act to this set.

A

It set it up in given the amount of customization, and you know like business, specific stuff that is running in our cluster um and it's a nice addition for like um swiss cheese approach, like that one's in-depth strategy to control from Bad changes and the most important thing is like Engineers- actually love it. So this is one feedback that we got from backend engineer.

A

um That's basically got paid for initiative deployment and it was automatically rolled back in I, think less than a minute, uh and that would have been like a rather nasty um and nasty problem with basically the MasterCard 3DS processor and on our end, that would have um um predicted like every every request, which we basically mean that a customer will not be able to pay with a monzo carbon line for the duration of that incident. But that was automatically reverted within within one minute um and yeah. Thanks you very much for listening to me.

A

uh We have a sometime for like a few questions. If you have any and like I mentioned earlier, like um we will do like a complimentary blog post with all the actual technical detail of how we enabled um the different aspect of this migration and our setup, and if you have any question in general about the gritty details, please um you know grab me after this got me doing the conference any any questions.

A

A

um I yeah I kind of um I got the essence of what you're asking about so about the the migration process. When we scale up and down the difference exactly.

B

Yeah, so when you're scaling up and down the deployment versus rollout I think sometimes what I've heard is that there can be some kind of cast repercussions, because you have two workloads. You know that are both scaled up fully or you have to have downtime to scale one down to roll out the the rollout um did. You guys was that ever a conversation in terms of that process, especially with the scale that you guys were rolling out these micro services at.

A

So the question is like was it? Was it a problem for us in general,.

B

Yeah I guess was it. Was it a topic of conversation like a concern about maybe like lack of resources in duplicating these workloads or something.

A

um So we kind of like didn't add that much of the problem, because, um due to the nature of what we are like a bank, we basically run with a lot of overhead in our clusters. So we had like a loads of like uh capacity um capacity left over like the whole point of doing this migration.

A

You know, flexibility is bringing out immigration across, like the entire day is to make sure, like you know, to control for those kind of problem where yeah sure you're gonna run double the workload for like a short amount of time for a couple of minutes and if you spread them out, you know across across the day, uh for example, for lowest tier Services.

A

We did batch of 200 like spread out of like eight hours, which basically means you have like one migration, every five ten minutes and the immigration process itself only lasts for like two three minutes. So that's all we control for it and that's why we, you know, decided to go slow to be to be extra safe but yeah. Indeed, that's something to take care of, because you don't and that also the wordpad about doing this.

A

Amigo top having this migrator pattern and checking that we are the right number of posts that we're expecting, um because for very, very large deployment um with like 500 bulbs. Sometimes it would take like a couple of free tries to you know it would take time for us to actually um come up due to scheduling pressure so yeah. That's why we yeah. Thank you 17 seconds any question.

A

Okay, cool perfect. uh Well, thank you very much. Thank you very much and.