Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022 [End User Sessions], 1 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Keynote: Push It to the Limit: From Canary Deployments to Canary Clusters - Henrik Høegh, Lunar

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Keynote: Push It to the Limit: From Canary Deployments to Canary Clusters - Henrik Høegh, Platform Engineer, Lunar

At Lunar bank we had a good problem, our customers rely on us to move quickly and provide new features and to do so in a highly reliable manner. To meet their needs we set out on a journey to move from canary deployments, where we could test new features in a safe fashion, to canary clusters. We envisioned a world where our production clusters were truly disposable and after 3 years we finally achieved that goal. In this session we will share how we did it, and how you can too. Today any engineer at Lunar bank can fail over the entire platform in 40 minutes. By deeply integrating with our infrastructure provider, writing some new custom operators, and moving most state out of the cluster Lunar is in a position to make disaster recovery a day to day operation. Listen as Henrik shares the successes, key learnings, and challenges we faced along the way.

A

Hi, what's an I.t crowd?

A

I'm going to talk about how we went from canary deployments to canary clusters, basically how we do failover clusters in luna. It will be slightly technical, but I hope that's okay.

A

I work at a company called luna. No, I don't think many people here know about it. So I took the liberty to post this slide. We have half a million customers, uh 650 employees almost 700. Now we make a app with a bank and you can invest in stocks and stuff like that. We are a tech company, so at luna I guess you can call me a lunatec or something yeah.

A

We are apparently a unicorn. Now it's something about evaluation, I'm not an expert, and apparently we are times two. I don't know how that works with unicorn, but yeah. We'll look at that.

A

My name is henrik, who I'm from the nordics slightly different climate. My genes are not made for this climate, at least I learned. So that's it. I'm a co-organizer at the cloud native or whose group, together with my colleague, casper who's, the organizer um yeah. I speak at different meetups conferences and stuff. I really like doing that sharing knowledge.

A

um I worked as a consultant for many years, but now I work at luna. I've been there for nine months and my hobby is dungeon and dragons. So I basically.

A

So, basically, in to people who don't know what that is, I paint small plastic stuff and then, with other grown-ups, we play with them roughly yeah.

A

One of the things I really like about dungeon and dragons is that one thing you learn is that a strong team is a diverse team. Four wizards won't survive the tavern so and everybody that's apparent to everyone who played it so so yeah. That's me, um this is the meetup groups we have in the nordics and we actually combine them to the cloud native nordics.

A

So if you like, cold weather or rain or in the area by all means, join us go to cloudnativenordics.com and we have a slack work group. I think it's called where we communicate and exchange speakers and stuff like that. So it's really awesome community.

A

So the at the end of today I'll talk about our tech stack I'll talk about what how we did failovers. Basically, how did we achieve the the capability of doing a failover using githubs, because that's not as easy as we thought it was and then some of the key changes to get speed and get more confident and remove complexity, um and then I'll talk just about what we are planning on doing on the road ahead.

A

So it was a journey. I tried to separate dungeon and dragons and work so you know, um but basically, casper and bian did an amazing job, transforming the company from working on a monolith to microservices, which was a great help when doing failures. We also transitioned from using deployment pipelines to using a gedox way of working, and that is basically key because you don't want to trigger 1400 pipelines when you do a failover.

A

Yes, so our tech stack, we use kubernetes, no big, surprise there. I wanted to make a joke about something else, but I don't want to do that. We use git ups, we have a mono ribbon. Just you know, there's different strategies, but it's a monorail. We use flux one at this time. When we did these things. We have now migrated to flux 2, which I could do a whole talk about, which is really awesome.

A

And yes, we did pay aws and s3 bucket full of money to manage our databases.

A

You could call it cheating, we call it convenience, it's really nice and we use rabbitmq to decouple our services, not just in space but also in time. So that's also a great help when you do failovers, if you want to do it without downtime, at least, and we use external dns to control our endpoints in the dns and to shift the routing weights in route 53.

A

We are primarily in aws, but we are also in google and in asher or azure or yeah.

A

Then we made some stuff ourselves. We made something called shuttle and it's kind of hard to explain what it is in very little time, but it's basically a thin layer. We put between the developers and our systems, so we decouple our services in kits from the from jenkins, for instance, and also the way that that the pipelines works and all that stuff it's a great tool. It helps us a lot. We can change stuff really quickly by using this it's open source.

A

We have release manager, it's a three-headed thing. uh One of the most important thing is the ham sea tail. It's not the meat. uh The first chimpanzee in space was called ham. So now you know we get the question a lot. uh So let's look at the first generation failover. So this is what we this is, what gave us the capability, and then we could start work on that right.

A

um You we would have a git ups repo with all our manifest. We had a cluster pointing flux to it and we had users, it's not to scale just so you know what we did was we created a branch because there are things in the new cluster that should be slightly different right, like cluster name, routing weights and all that stuff, and this is, I think, the way many people do it. This is the way we did it in the beginning.

A

We would then create some edits here, like the cluster name and stuff. We would then spin up another kubernetes cluster point flux to it: spin up all the services, and then that was fine. No traffic was routed to it. Yet we would then federate the two clusters.

A

Two-Way federation is needed unless you want to throw away some messages um and then we made some edits in both of these repos or sorry, not replace branches, and that would then shift the the rounding weight on the annotations that external dns uses. So we now have maybe 80 20 or something like that, um and then we can shift it again and again until we actually don't use the old cluster anymore, then we can remove the services and once we're sure that every service is is down and gone.

A

We can then remove the federation and we can actually remove the old cluster and then the fun part comes where we need to merge this branch into the main branch and stuff like that and point flux to that.

A

That was complex and- and it sounds it looks simple, but when you're in the situation on incident mode, this is not the complexity you really want, but we had the capability. There's a lot of work went into this, so it's really awesome so challenges a lot of merge complexity in the github repo.

A

It is not a nice experience. New deployments would stale during this exercise, because we would branch and only new deployments would or releases would go into the main branch, but they would eventually get back.

A

Most people felt a lot of people felt uncomfortable doing this because of the complexity and all this and it's it just didn't, seem like in the spirit of get ups it. It should be more smooth right. It feels like a hack, so to speak. Some of the observations um most of the edits was the cluster name. So there's a lot of things in the cluster that needed that barcode right.

A

It's still not a pet, it's still a cattle, but we need the barcode so that, for instance, fluent bit would know to send the cluster name with it, because when you have two clusters that are identical, you would like to know which cluster did. This locked line came come from right and there's a lot of other stuff like aws aim authenticator, the external dns annotations.

A

Would need to be added as well each time we would shift as well, because they couldn't be the same because then you would have the same routing weights on on both environments right so going forward to second generation.

A

We created two new controllers, because that's how you solve things today, we created one called cluster identity controller. It has. um It has some strategies to find the cluster name and then create what you call it config maps in each namespace with the cluster name, and then the applications can use it. We also created something called routing weight: controller to control the routing weight annotations that which is then used by external dns.

A

Yes, so the cluster identity. This is an example of a cluster identity and conflict map. What it looks like- and you can see you have the cluster name here in this case from 2022, and you can then use that in your application.

A

This is what the routing weight would look like the cr, where you specify the annotations that you need to put on increases with the right cluster name, and you also have the cluster name, because it'll only react to cluster or routing weight objects that has its own cluster name. And this way we can have multiple routing weight objects in the same git repository in the same environment, but it'll only react to the ones belonging to this cluster.

A

So that's the way we fought branching, it has a dry run mode, which is nice in the beginning, at least so yeah. This is how we do it, and this is what an ingress would look like when the writing weight controller is done, it would add the annotations and then route 53 would then connect to externally external dns would connect to route 53 and make the adjustments we use shuttle to manipulate these. uh It's actually a repo just like any other micro service, and we have shuttle where we can run.

A

These commands like delete, routing weight, add round and weight adjust running weight, and this is what it looks like before. We do. The failover we can see in the yaml we have one cluster with 100.

A

What we do is we have this extra repository. The old cluster is pointing to our main branch. We create a new cluster, we spin up flux, it points to the github repository everything spins up and we then have no traffic yet to the new cluster.

A

We, let's see if this works yeah we federate the rabbit mq and then we run the shuttle run, add routing weight. It will then create the entries.

A

We release this as any other service, and we now can add rounding weights to our new cluster and then simply by running these commands, we can then gradually shift the traffic so adjust routing weight to 50, 50, 80, 20, 100 and so on.

A

Simply like releasing another application which is really nice workflow. This is what we're used to yes, and once that's done, uh you can see in the shuttle the the run weight, resource application so to speak. We can see. We now have two clusters and you can see the routing weights.

A

What we then do is we delete the rounding weight, release it and then no uh traffic is routed to it, and we can then start to remove flux and all the applications when everything's down. We can then remove the federation and we can actually remove the entire old cluster.

A

Yes, so this was much easier and much less complex. Some of the effort that went into this, we coded two new operators. So that's also complexity, but it's code, it's much easier for us to handle. We only need to maintain this and not every time. We need to do a failover where we need to have as much cognitive overhead as possible.

A

We did 17 failover runs in roughly two months period, which was a lot, but it was a heavy investment, but I think it paid off quite well. Every iteration led to improvements. We also use spot instances in dev which led to a lot of great findings um yeah. Some of the results everyone in squad odyssey, which is the platform container runtime squad, can now do a failover.

A

The failure operation is down to five automated steps and the reason it's not one is because we need to check some stuff once in a while, no git up branching, which was awesome.

A

We didn't really like the branching, so getting rid of that was awesome and we went from spending a whole four hours doing a complete failure without downtime to actually 40 minutes, which I think is okay, so that was really awesome. Speeding things up some of the problems. Well, we have this identity controller that needs to have these strategies to find the cluster name, and that seems like a bit if we could only have like a name metered metadata thing in kubernetes we could remove an operator.

A

So if anyone is listening, yeah some of the goals well use, we want to use flux v2, we can use dependency, remove more complexity from our scripts. We want to migrate to cluster api. We want to migrate more terraform to to cross plane just to make everything behind a closed loop, yeah culture. What we did was the deming cycle plan to study act is the deming cycle, and we just did that again and again, and with that I would like to say thank you. They are open source you're, more than welcome to contribute.

A

Thank you so much for your time.