Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022, 1 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: How to Migrate 700 Kubernetes Clusters to Cluster API with Zero D... Tobias Giese & Sean Schneeweiss

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

How to Migrate 700 Kubernetes Clusters to Cluster API with Zero Downtime - Tobias Giese & Sean Schneeweiss, Mercedes-Benz Tech Innovation

Cluster API promises "to simplify provisioning, upgrading, and operating multiple Kubernetes clusters." Do you find it challenging to migrate your existing Kubernetes cluster provisioning to Cluster API? Would you like to benefit from all the features that Cluster API offers and manage your infrastructure the Kubernetes style? At Mercedes-Benz, we run and operate more than 700 Kubernetes clusters and 3,500 machines all over the world in on-premises OpenStack data centers. By migrating to Cluster API, we replaced our legacy provisioning, consisting of Terraform, custom self-written tools and Kubernetes operators. Expect valuable insights on what it takes to transfer production systems into the control of Cluster API with zero downtime and zero customer impact. Get to know the technical challenges of migrating, how they can be solved and how to extend Cluster API functionality to fit your needs.

A

All right uh thanks for joining this session after lunch, and then ah I'm really excited a lot of people in there in this room. I actually uh moderate a couple more sessions yesterday, but some room is not quite poor, so I'm really excited about this topic, and so my name is danielle. I'm a cncf ambassador, I'm I'm so happy to be here as a model at this session today and thanks for joining once again. So we're going to talk a little bit about our next topic is how to migrate.

A

700 kubernetes cluster, not 7, not 70.. It's a 700 equivalence cluster to cross the api with zero downtime. So I'm really happy for here and then please welcome our next great speaker, sean and tobias from mercedes-benz tech. Innovation. Please welcome the speakers.

B

Okay, thank you so hello, everyone and welcome to our talk on how to migrate, 700 kubernetes clusters to the cluster lifecycle management, software called cluster api. So today we will present to you how we replaced our legacy. Kubernetes fleet management with cluster api- formerly it was based on terraform and the most important part was zero downtime and no effort for the cluster users.

B

I'm sean schniewicz, a software engineer for mercedes-benz tech, innovation. Also, I'm a maintainer of cluster api provider openstack. This is my first kubecon and I'm so happy to meet all of you.

C

Hola hi, my name is subias gizer. I am also a software engineer for mercedes-benz tech, innovation, I'm also a maintainer for cluster api provider openstack and I'm working with kubernetes for five years now. So this technical presentation targets software and operating engineers and all who is into cluster management.

C

It's maybe a bit um a bit hard to learn in the beginning, but it's definitely worth the effort, so keep it on mercedes-benz tech. Innovation is a subsidiary of the german car manufacturer, mercedes-benz. It's the uh the headquarter is located in ulm and at mercedes-benz tech, innovation. We don't build cars, we build software and we are formerly known as daimler tss. Just in case. You want to check the git commit history or you already known us as timeless tss.

C

So yeah our platform team develops and operates a huge fleet management and we're operating 900 clusters all over the world in four data centers in atlanta, beijing, frankfurt and stuttgart, and just a note during the time we have submitted this talk, we only had 700 clusters, so 200 clusters more in half a year. That's not too bad, I think so. The agenda uh we first step in and set the stage just that everybody knows what we are talking about.

C

Then we will talk about the legacy provisioning architecture. Then, after this, the target picture with the migration to cluster api and in the end we will talk about the lessons learned and our next steps.

B

Okay, so we want to get to know you a bit better. um So please raise your hand if you know cluster api, who of you know, class api, okay, nice, who is using cluster api in production, still a few nice okay, who manages and doesn't have to be with plus api, who manages more than 10 clusters.

B

100 clusters: 1000 clusters: okay, oh okay! Quite some experience in this room um really like it so cluster api provides a central cluster management for the complete life cycle of a cluster.

B

So this includes creating deleting updating to a new kubernetes version and scaling so adding additional nodes to your cluster cluster api is a project maintained by the special kubernetes special interest group cluster life cycle and it perfectly works with multiple cloud providers such as aws, google, asia, openstack and many many more actually right now, there's a separate talk that is teaching how to create your own cloud provider plugin to plus api.

B

um So this feature of supporting cloud providers is why we chose cluster api, so we can not only offer private cloud with openstack, but also public cloud to our business partners.

B

Here you see the turtles that are about to start their journey, the cluster api tools that are about to start their journey. But before that I would like to clarify some basics. The key aspects of the title from our talk first off how to migrate.

B

So starting point for this is any existing cluster doesn't matter if it was created with some kind of automation or if you created it on your own. In our case, it's a legacy pipeline and then we want to manage the infrastructure and the cluster itself, the kubernetes dial. So normally, this is with custom resources and controllers.

B

So the architecture of cluster api allows an incremental transitioning path, so you don't have to do everything like at one point. You can just split it up and just adapt those parts that you really need.

B

If you take one of those three turtles and take a look at a single turtle, this stands for workload. Cluster. One of these workload clusters is what we usually give to our users. The workload cluster consists of a dedicated control, plane and multiple worker nodes.

B

And we want to migrate 700 of those workload clusters to be controlled by cluster api, our largest management cluster- that is the cluster that manages all the class api resources controls up to 200 of those workload clusters.

B

And when you look at the outer shell of the turtle, that is the infrastructure, the networking, the firewall, the router load balancer, and that also is managed by cluster api.

B

Now, when migrating one really important requirement for us was zero downtime, we have business, critical applications and downtime is just not acceptable for our users, so we definitely want to make sure that the kubernetes api and the application is available throughout the migration. This is not a greenfield approach where you get a new empty cluster and the users migrate from one to the other one. Now this is, we will not touch the user's workload.

B

This is really important.

B

So, as we now have clarified the basics to be us, would you like to tell us where we started from okay.

C

So this is the picture from our legacy cluster provisioning architecture, so we have a custom user interface where our users are able to order a cluster, and we have a custom api that triggers several jenkins pipelines and these pipelines are trigger some terraform pipelines for the cluster provisioning, so the infrastructure, control, plane, workouts, etc, etc.

C

If the cluster is provisioned successfully, we have ansible to deploy the runtime deployments. Runtime deployments in this case are, for example, the kubernetes core deployments like cni csi coding s whatever, and after this we will deploy the cluster add-ons within the bill like matrix exporters or custom controllers, etc.

C

So this is the legacy provisioning and, as you know where we are coming from, we will now take a look at the target picture. This is the target picture the user can order a cluster like before, but in the background we have some changes. The main change is that jenkins will trigger a custom binary that deploys resources into our cluster api management. Server and cluster api will reconcile the infrastructure, the control, planes etc, and after the cluster is provisioned, our custom tubular controller will hook into the correct time and deploy.

C

Then the runtime deployments and flux will deploy our cluster add-ons. That's it. Okay, perfect, let's start with the preparation, so we are using openstack as the infrastructure, layer, openstack or the cluster api provider. Openstack identifies the resources by its name, so we must change the resource names to the cluster api provider. Openstack name matching.

C

So this is the first step and can be done by a terraform, for example, just make sure that this is renamed correctly, for example in the cluster api provider, aws they're, using ids and not the names. So this is much easier. Just take a look at your provider, how there's done okay, so there are multiple ways to migrate to cluster api. We decided to do it in three steps, so the first step after the resources were renamed is to migrate the infrastructure that cluster api is able to reconcile.

C

We need the cluster resources and the infra cluster resource, in our case the openstack cluster.

C

This is the first step after this we can go further to the second step, we decided to use um the worker machines as the second step, because we have, uh during the run time more changes and we are can we. We can have cluster api to reconcile the machine deployments, etc. If a user changes the replicas or the image flavors it's whatever, and because of this we decided to do first, the machine deployment of the worker node migration, okay, the third step is the migration of the control plane.

C

This is also really critical, because if there is something false or whatever the lcd could have, data loss and your cluster will be broken, so keep care on this step.

C

Okay, let's move to the first step, the cluster and infra cluster migration. This is really really easy, because you can just use your info clusters back and use the from the legacy provisioning the specs and deploy just the resource you can. You can then deploy the cluster resource for the cluster api to reconcile and just reference the infocluster object and that's it. Your infrastructure will reconcile and cluster api is able to manage the infrastructure, perfect sean.

B

B

Okay, so we now migrated the infrastructure and we have continuous reconciliation by the cluster api controller, with metrics and status easily available available from our management. Cluster next step will be to migrate. The worker nodes, existing worker nodes and a node typically in cluster api is managed by a machine and therefore we will create those three resources, machine deployment, machine set and the machine itself, and also the provider-specific implementation. The openstack machine.

B

Now the problem is the the existing control plane, which is not part of the second step. It's part of the third step plus the api is not yet aware of this control plane and to overcome this problem, we will create a fake cube, adm control, plane, the cube, adm control, plane controls the control plane for class api, and this fake hub, adm control plane, of course needs a machine. So we would just create a dummy machine which is not combined with any infrastructures.

B

You just have to create the resource and then reference this and I'm sorry, but at the control plane label which identifies this as a control, plane machine and on the right side also create a dummy, cube, adm control, plane, and please be aware, we added the paused annotation to both of those resources which will prevent cluster api from doing anything on those resources.

B

And then the last step to get your fake cube, adm control plane is to patch the status fields and add the condition to the cluster and just yeah, say: okay, you're ready now for cluster api, and that's it. For this part now we can actually migrate the existing worker nodes to be represented by class api so for each worker, node in your cluster, we'll create an openstack machine and it is connected by the provider id provided in our case from openstack and this openstack machine. The name of the openstack machine is referenced in the machine.

B

The other fields, such as the bootstrap secret. They don't really need values that are correct. You can just create an empty secret put in the name and then we're good to go.

B

Okay, so we now have all workload nodes represented by an openstack machine and a machine next off we want to create a machine set. The machine set is an immutable abstraction over machines.

B

B

We have labels, specifically specifically the deployment name that should be added to all resources on this slide. Okay, so the machine deployment needs this label to identify those resources to be part of the machine deployment so make sure to add those too and now the machine deployment.

B

As you see, the name is the same as in the label. The deployment name should be created and match the replicas to the amount of workload nodes you have in your existing cluster. Also before creating the machine deployment, create the cube, adm, config template and the openstack machine template put it into the machine deployment at the kubernetes version. You are using and that's that's about it. So pretty easy. I guess now. If you have a look from the customer's perspective, what he will see it looks as follows.

B

So, on the top of the slide, you see the state when we migrated the workload clusters managed by class api. The name stays the same as we then perform a rolling update to a new kubernetes version.

B

New workload nodes will be added to your cluster and will have new node names as they were bootstrapped and provided by class api.

B

So would you what will you continue with step? Three ethan.

C

Of course, thank you sean. So after we have migrated our worker machines, we can come to the last step, the migration of the control plane and the cube adm control plane cluster api resource. So let's take a look at this.

C

First, we have to create a new cube atm control plane with its real data, so we can define here the version. The cube adm config spec, where the cluster join configuration is in it, the machine template with the infrastructure, reference, etc, and because why we are using a new cube, adm control plan is that cluster api denies some changes in the kcp specs with the with a web hook, and because of this we will create a new one, because it's the easiest way. We can create just a new name and that's it.

C

Okay, after we have created the cube adam control plane, we can now adopt all the legacy provision control, plane, machines, okay, so we just creating a secret and acute cuban config, like in the machine deployment step with a empty value. It's just a dummy and reference all for all legacy provision machines, a machine resource we can- or we have to add the provider id of the openstack instance just to reference the infrastructure itself.

C

Okay, after we have the machines, we can create the openstack machines. This is the resource for the cluster api provider openstack to be able to reconcile the virtual machines itself. So we have to here reference the machine we have created. We have to reference the cube, adm control plane. We have created and also sorry one step back. We have to pause the kcp as well, and also the machine just to make sure that the qb controller in the machine will reconcile at the correct time.

C

So we now have the openstack machines created and that's mostly it. We only have to change the cube adam control, plane's name in the cluster reference unpause, the cubed control, plane, unpause, the machines and that's it. You can see in the screenshot below that the new cube lm control plane, the new control plane from cluster api will join the cluster.

C

It's not ready yet, but it will get ready and after it's ready, cluster api will remove the legacy provision, control, plane and we are done so easy. Okay, after this step, we can clean up our dummy objects. That's you are that simple and we are done. We have migrated our first cluster and then our 700th cluster, and that's it. We are ready with our migration.

B

Okay, so how does it look? Are you still following like an easy one? Two three approach, I hope so um so now we want to share some best practices, some tips and tricks, and lessons learned with you to sum this up and the first one is not really cluster api, specific, it's a general recommendation.

B

So whenever we talk about zero downtime, we ensure that we can safely drain the nodes, and this is an action that has to be performed by the users. So we ask our users to add port disruption budgets to their deployments, so we can then safely drain their nodes and sometimes one of those pot disruption. Budgets is not working or even the upgrade. The kubernetes update is stuck due to one of those pdbs. Then what we do is we contact the customer.

B

Ask them to fix it, and then the upgrade will continue by itself through the class api, reconciliation.

B

The next two things are really nice functionalities. From the cluster api controller, the one is called the pre-drain hook or annotation, and the other pre-terminate hook or annotation. The pre-drain is just executed before draining a node.

B

We use this pre-drain annotation to add a no schedule chain to all nodes that will be anyway deleted during an update. So whenever a new, whenever a port gets rescheduled to another node, then it is mostly most of the time scheduled. Then to know that is a new one and not the old one, and this increases the update a bit.

B

The second one is the pre-terminate hook. This is executed directly before the node is deleted and we use this for mostly infrastructure related things such as volume attachment. So we remove the volumes and the second is to remove the node from the load balancer, so highly encourage you to to make use of that functionality.

B

The next one is nightly, builds and testing testing is key, so we highly emphasize and recommend to you. Please test the migration, but not only the migration itself, from legacy to class api managed clusters also test creating new clusters and maybe deleting or upgrading clusters. This really helped us to identify nasty bugs and issues so yeah. This is yeah. Testing is always important and the last thing is actually a problem we face. This is a caching problem.

B

The infrastructure is not always as stable as it should be. You probably know, and sometimes openstack machines failed or had like a status. That was error, something like that and most of the times we enforce the deletion of one of those openstack machines with maybe removing the finalizer. The problem is the controller that watches on those resources doesn't know that the resource was actually deleted because the cache is then invalid or has false data and to mitigate the problem.

B

We just restarted the ports, but luckily one of our colleagues created a pull request for the controller runtime project and fixed that small bug, but nasty bug. Okay, that's about it! um To be honest, where we are going next, thanks.

C

Sean okay with the migration to cluster api. It's not done our platform is under uh permanently con construction with feature improvements, bug fixing user feature, requests, etc. So the replacement of ansible with flux, cd is our next thing and also in progress.

C

Another thing is that during the initial planning of the migration cluster api features like cluster class or the runtime sdk machine head check, etc were just not there, so we are also going to plan the implementation of these features.

C

What we are really proud about is that we have uh developed a cluster api state matrix exporter, which is a matrix export of glassy api, and we are going to merge this code to the core cluster api repository and because of this, it's a donation to the cncf and a big. Thank you to all of you.

C

Last but not least, is that we're moving forward to also offer to our users public clouds because of cluster api. This is really easy. We can use another provider and that's it.

C

Yes, if you want to get involved, just ping us on slack or follow us on github and by the way, there's a cluster api, intro and deep dive. The next room in a few minutes from yuba rush and vince, so just join, and maybe you can learn something thanks and if you have questions now is the best time.

D

Hi hello is anderson here from brazil, I'd like to know the the time frame since the start of the planning until the last the migration of cluster number 700.

B

Okay, so the question was um the time frame from starting the planning and then migrating to 700 clusters. Right, yes, okay, okay I'll just answer um so initial planning was around I'd, say the end of 2020 and then the complete 2021 year was uh used for migrating those clusters, because, usually what we do is um we wait for kubernetes release that we anyway, where we anyway have to redeploy all the worker machines and use that time frame to also do the migration to cluster api.

B

So with maybe thought on the slides with 1 20, we migrated the work nodes and then with 121. We migrated the control planes.

B

So that's about it, but uh when actually migrating for 700 clusters, we took maybe some two weeks, maybe because you know doing batch batch updates to those workload and control plane machines.

C

There was also another question if we are doing the update batch, wise or cluster by cluster. um If we release a new kubernetes version, uh we have also migrated each step, so during the terror form renaming, we use the release for kubernetes release update to do this for the step, one for the infrastructure.

C

We have used the release, so it took four releases to do the complete steps to be.

B

Honest yeah, but when, uh when updating the clusters itself doesn't matter with class signal now with class api, um the class api controller supports the machine concurrency and the cluster concurrency and actually a concurrency for all objects, and this is usually set to 10 for us. So if we have 200 clusters in our largest management cluster, we'll just trigger all of those 200 at once. But class api has like a like this concurrency and we'll just work on 10 objects. At the same time,.

C

There's another question: one of the slides at the beginning showed that there were five platform teams, how many engineers in total, taking care of the platform in these 900 clusters. So in general, um as far as I know, we are a platform team or the five platform teams are 30 people or 40.

C

the core platform team for the cluster api migration.

C

I think we are about eight six to eight people, so the migration of the 700 clusters to cluster api have been done by eight people: cool, okay,.

C

Any further questions.

A

Any other questions: oh yeah. There we go.

E

I have a question about the initial provisioning, so you say you manage your clusters using cluster api, but what about management clusters right? So, let's imagine a situation when everything is burned down and you have to deploy the management cluster. How are you going to deploy the first one.

C

To provision a management cluster we're using kind, and then we are provisioning, a new cluster and we're doing a cluster cuttle move to move all the management resources to the new cluster.

C

So this is the initial check, act, problem step and we are doing backups with veliro, so we can do a restore with all its resources and then it's.

B

Done does that answer your question? Okay, nice, perfect.

F

The technique which we used while migrating the cluster is, is canary, blue, green or something else.

C

As far as I know, we have using canary updates, can we testing yep.

B

Henry, I would say thank you yeah. This is not not blue, green.

A

Okay, okay, so we gotta have more couple questions. Thank.

F

You hi um did you have been injured by any chance uh bare metal servers anywhere and a barometer workload or any plans for it on capo.

B

We are not, we are not using bare metal servers, um we have openstack is our lay. We talk to beneath its virtual machines, um yeah.

C

And the next step is public clouds like ezra, aws, google and so on. Yeah.

F

Okay, I was thinking about uh ironic on openstack yeah. Did you ever try it or no.

B

We haven't yet but definitely interesting yeah. Thank you. Sorry to tell.

A

C

One question in the back: okay.

G

I think I think you mentioned uh 200 clusters workload clusters per management clusters. Why there's a limit? Is there a limit for how many clusters can be managed by the management cluster? There's.

C

There's not really a limit, but we have multiple private cloud, regions or zones. Because of this we have per region of zone a separate management cluster, and this is the.

B

Reason so we we don't have more users on a single zone uh right now, but um I think it will. It will scale pretty well um until a certain point probably, but we haven't reached that point yet.

B

Okay, any more questions. Yeah, we got a one more here: okay,.

H

Hi um was this an unattended migration, and if, yes, you can decide the user when you want to migrate the cluster, it.

C

Was attended migration because we want to use upstream features from cluster api and what was the last question? Sorry? Have you heard it.

D

C

Okay, perfect thanks.

A

H

A

Any other question.

A

Okay, I think so we're done, and thanks for great presentation. Thank you so much. Thank you. Thank you.

B

Thank you, daniel and.

A

Then, uh thanks for attending once again and.

B