Cloud Native Computing Foundation Online Programs, 26 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: The many uses of Kubernetes cross cluster migration of persistent data - Ryan Kaw (DoK Day EU 2022)

Description

https://go.dok.community/slack
https://dok.community/

From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE)

Multiple clusters exist in most Kubernetes environments today, and number of clusters will increase overtime. The reasons for having multiple Kubernetes clusters are many, for example, overcoming scale limits, reducing complexity, geo separation, redundancy and having separate production, staging, and development environments. Once you have multiple K8S clusters, it can be useful to have the ability to easily move or duplicate workloads across these different clusters. Kubernetes does not have a native method to allow migration or duplication of workloads across clusters.

Fortunately, there are tools that provide this functionality. In this presentation we will explore the different uses cases for cross cluster migration, and what is involved, and how these migration tools work. We'll cover some popular uses cases, such as, Disaster Recovery, Test/Dev, and performance testing. Migration could entail moving the entire cluster, or individual workloads. The components that need to be moved would include configuration and resources stored in etcd, and persistent data residing on PVCs. We'll cover the uses cases and challenges for migration, and run through an example of using one of these migration tools.
-----

A

So um today, I'll be I'll, be discussing the many uses of kubernetes cross-cluster migration of persistent data. I think we all know why kubernetes is great, um but I also want to talk about what are the use cases for multiple kubernetes clusters. So, if you think about it from an operational standpoint, you might have one single cluster. You want to overcome some scaling limitations.

A

Maybe if you have a single cluster and that fails, you want to have redundancy. So that's another reason why you have multiple clusters from a devops perspective as well. You might have an environment for prod, an environment for staging an environment for for dev or even today. Now you might have kubernetes running on your data center and then you're being asked to leverage the cloud computing and resources out there too. So those are other cases of why you leverage multiple kubernetes clusters, as I go on through this.

A

The slide deck we'll also talk about data management, use cases, but also, more importantly, the challenges around taking that data and moving it to another cluster, because there's no native tools within kubernetes today to go and address that so cross-cluster data management use cases. uh One great example is maybe you're writing something on production today and you're running a certain kubernetes version and they just released the latest and greatest version uh to go out and- and you want to migrate and test that data as well to that new version.

A

uh So in that case, you might want to uh migrate, move that data to that to that new cluster and test it as as well or you might be from running kubernetes in aws today and from a financial standpoint, it's better to move it over into azure. How do you take that workload and and migrate over there? So these are just some of the examples of cross data management use cases as as well, but the challenge really is is around kubernetes is that these clusters are isolated.

A

So if you look at here, you might have a production environment, you have a staging environment and these clusters don't have any knowledge of each other whatsoever and they're both isolated in terms of storage and networking.

A

If we dig a little bit deeper down into the into the storage aspect of it, a lot of that important data sits on persistent volumes, pvs right and there's no native kubernetes infrastructure to copy clone or migrate. That persistent volume uh between between clusters as well and the other challenging thing is from a network perspective. So there's no network connectivity between the source and entire cluster other than publicly exposed services.

A

The persistent volume data is not typically exposed on the network for ingress or egress. So we really have to think about a way to transfer that data between those two clusters, while making sure that we don't overload the network and also understand the storage bandwidth as as well, because, if you're running in prod- and you start moving all that data, it could cause some some issues as as well and, as you think, about migrating and moving that data to a new cluster.

A

Do you really need to move the entire cluster right and in those cases you might you might want to, but you can also get pretty granular as well on what you want to move? Maybe it's just a specific name space, maybe a specific label, and you could just take maybe the etcd and the persistent volume so that application remains intact into into another into another cluster.

A

Some other aspects to take into consideration as well on the storage side is not all. Clusters are equal right, so you might be in this example over here I'm running aws eks on u.s west, where the persistent volume running on storage class ebs. While if I want to move it to the other region and it's running a different storage class gp2 and then this specific scenario on the left-hand side, it's using some csi drivers where, on the right-hand side doesn't have csi drivers enabled and it's using a different storage class as as well so.

A

Some tools that are available today, open source tools, valero and rustic, are, can can be used.

A

You can install them both on the respective cluster, so the source, and then the target is where you're going to want to restore it, to there's, usually a common s3 storage endpoint that both clusters can access, because that's going to be your backup target and then you're going to read from it to restore into that target cluster and obviously you're going to have to create that target cluster manually as as well and then restore that data in into it um cloud.

A

Casa io is another solution as well, so it's a software as a sas solution, uh which is installing lightweight agents onto your onto your cluster um and then it gives you visibility into uh all of your clusters within aws eks. It works outside of those realms as as well too, but it handles a lot of the challenges that are discussed around networking and disk issues because it has the ability to do throttling.

A

So you don't overload your production, environment and there's other nice features and functions such as role-based access control, so certain users can have access to specific workloads for cloning and migration and then also their deep integration with aws and eks.

A

Where they can go ahead and spin up clusters on the fly, so you don't have to actually create any standby standby clusters as well. So here's a high-level topology of the general workflow of how these migration tools will work. So if you start from left to right, you have your production on the left hand, side, maybe a qa on on the right hand, side. Essentially you install those migration tools or the agents on on each of those respective clusters.

A

It goes ahead reads that data from the cluster then goes ahead and takes a snapshot using csi drivers of the persistent volume and then moves that data over into number three, that little thing in the middle of the object, storage and then, when you go ahead and restore reads from that object, storage and then writes it back into that cluster, as well as restoring that data into that persistent volume on the target end as as well I'm a firm believer of pictures are a thousand words, so here's just a walk through and of restoring a cluster or namespace, rather within cloud casa, so go over here you can click on launch cloud formation stack, so it has deep integrations with aws.

A

After that point, it goes and detects all of the does an inventory of all your available clusters within within the kubernetes environment, and then all you have to do is simply gives you a cube, ctl command that you can just copy paste it into into each of your respective clusters, and then at that point it goes and detects all the various namespaces that you have.

A

You can select specific labels and in this example, I'm saying uh protect this test website over here and and snapshot it aspen and also move that data to to my object, storage as as well so from the resource standpoint, you have the ability to create a cluster on the fly for you or you can choose an existing cluster, so in this case I've chosen an existing cluster.

A

It also handles certain things as well, like changing storage classes. So I'm going from one storage class to another storage class. You can input that information into there.

A

So in this case it was going from ebs to gp2 and then also, I can add in specific settings like adding a prefix and a suffix as as well so when I go ahead- and I run through this, this work, uh this restore you- can go ahead and see on the bottom over over there that that name, space gets restored with the the suffixes and and the prefixes that have appended onto there as well as it's restoring onto that gb2 storage class right.

A

So um that's my presentation on on this topic, uh where I'll be at I'm at google. Actually so come by our booth as as well we'll be there if you want to check out our solution. Thank you. So much.