Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022, 1 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Transparent Live Migration of Services Between Kubernetes Cluster - Adam Janikowski & Jörg Schad

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Transparent Live Migration of Services Between Kubernetes Cluster - Adam Janikowski & Jörg Schad, ArangoDB

Operating a distributed database on a single Kubernetes cluster is interesting, but how about transparently migrating it from one cluster to another–potentially between different cloud providers– without impacting user workloads? Kubernetes has become the de facto default deployment for ArangoDB, a distributed Graph database. Consider for example ArangoDB Oasis, a managed Cloud Database service with over 200 deployments (aka highly available database clusters) across three major cloud providers and many regions. But outages, (Kubernetes) upgrades, resource considerations, and cost optimizations require the underlying infrastructure to be very dynamic including migration between Kubernetes cluster, datacenter, or even cloud providers. This talk provides insights into how Kube-Arango, the OSS operator for ArangoDB, supports live migration of distributed stateful applications without impact on users. Challenges in such migration include for example networking, DNS, and persistent data.

A

Welcome to kubecon again and uh also to everyone here and also people remote, very excited to have in-person events anymore. I really missed kubecon. I must say over the last years, uh welcome to our talk about transparent, live migration of services between kubernetes cluster across multi-cloud.

A

As you can already figure out, we tried to squeeze as many buzzwords in there as possible, but what is it really about? So it's about moving stateful services. So, in our case that's a distributed database between different kubernetes clusters between different regions and even between different cloud providers. This is a quite a challenge for us because it should be transparent for the user. So when we do that, we want our database users to actually not at least not lose the availability of the cluster.

A

They might have a small performance degradation, but from an operational perspective, this is super useful as you'll see for upgrading et cetera, managing a general managed service for databases. We are spoiled by choice. We had a number of options, we evaluated uh all of them. They require cooperation of multiple tools from kubernetes over our kubernetes operator.

A

Networking layer and also the database needs to be aware, so we'll talk a little bit about that and also try to give like general advice for different setups and then uh we'll try the challenge for everything: a live demo here on stage. Let's all hope this goes well. We this is my colleague.

B

Adam here hello, my name is adam. I joined the communities of my adventure with the kubernetes itself from the version 1.1 from this point on time. I worked as the devops, but I found that development is a little bit more interesting for me and from two and a half years, I'm developing the operator which is used for the anglodb.

B

From this point of time we went to the a lot, but I will pass it. Okay,.

A

No, I actually took the opposite choice, so I'm a little further away from development right now, which is very sad but uh cto currently at iran could be very happy to combine like my big, two passions cloud nativeness. So it was early on with this hype like back at mesosphere, uh building up also like community. So it's really great to see. Kubecon also growing, that much and then yeah the other passion database systems, uh also across different career stages and now yeah. I all get that like within one role, pretty cool.

A

Why are we here as a database company talking about at a yeah kubecon here? If we look at databases in 2022, we actually, even if we look at the landscape, we see it's like the first block in the cncf landscape. So we already see it's like quite important. If we walk around these across the sponsors, we also see a bunch of different database companies, and this is really like changed over the last years.

A

Being a database company nowadays usually means you are cloud native in some way or the other and interestingly- and this is something what I enjoy quite a lot. It also means you actually change a lot of things in how you built your database like five years ago. It used to be those servers in some basement, some fixed switch in the middle uh nowadays, it's basically a distributed stateful system on dynamic infrastructure, kubernetes telling us what to do aws telling us- please reschedule this or that.

A

So it's actually it's a very different game, but it also feels quite interesting to turn into a fully cloud native database and just from the customer side, uh probably like most of our users, so a wrangler to be as an open source project are leveraging also some cloud service or the other, and most of them also our kubernetes operator, and then we also have our customers, who also run on on the cloud on top of different options, we'll come to in just a second. As said, orangudb is an open core graph database.

A

So it's really a scalable graph database at its core, but it also supports other data models such as document. We have a full text search and retrieval on top imagine like a lucine inbuilt uh into the graph engine. uh We have a graph analysis on top based on google's prego framework and we also have a graph ml stack. But in this talk we actually want to focus on like the left upper corner of the circle, and this is about how can you run a rangodb or how can you distribute it?

A

You see on the right. This is just like some architecture diagram. I also don't want to go into too much detail. It's not the focus, but in short, a rangodb is a distributed system so kind of like you, don't want to just manage that it has different components, so it's not trivial to set up to maintain failover etc, and this is why we early on invested also in the uh cuba rango. It's called this is our kubernetes operator and then based on top.

A

We also have our managed service or asus building up, and I briefly want to introduce that, because that was one of the main drivers. We actually came up with the changes for this talk, but changes are also in the open source operator.

A

So a rangodb oasis is our fully managed cloud service, so automated deployments, failover scale up, etc. Yada yada, as you probably heard from all other vendors here, offering a managed service as well.

A

I think what makes us special we actually run across all the big cloud providers- aws google azure- and we do that by leveraging their managed kubernetes offerings. This is actually like. The naive assumption could be hey. It's all. Kubernetes is the abstraction layer. Interestingly, it's not quite that easy. There are still quite some differences in between, but I think this uh we are managing that pretty well right now, what he just the brief uh architecture piece, maybe to understand uh the uh motivation for this talk.

A

We have like one central control plane controlling oasis by itself, but then we have data cluster, so you can translate that to kubernetes clusters and they are spread across different cloud providers across different regions. So, right now uh we have uh like a very, very large number of those data clusters flying around uh up to hundreds of them.

A

So um this was kind of the motivation chris we were facing like some operational challenges. Of course, this is only the small subset which motivated this talk, there's a lot more but to just start out it's the updates of kubernetes and in particular, as we're using the managed kubernetes offerings. We are often forced to upgrade so especially google. uh They will tell us yeah, you have to update till then then, or we will do that for you and being a stateful distributed database.

A

We, of course we want to be in control when we upgrade and there are the first big challenges coming in, and this is about that an upgraded cluster is not the same as if I would spin up a new cluster with the same version.

A

So, for example, I think on google cloud, we started with like a 114, and we know that when we hit uh kubernetes 124, we won't be able to upgrade anymore just because the changes accumulated so far are too big and we'll actually have to deploy a new cluster, probably 124 around that, and this is also makes, of course, testing much harder, because if we test easy thing is to just bring up a new cluster and you don't even have a choice anymore, to bring up like some of the old clusters, and so this is one of the big operational challenges we are dealing is how are we upgrading our managed kubernetes clusters, and how can we also test whether it still works, because simply it's not exactly the same same environment?

A

Next challenge is migration between different regions, so I mean obvious thing is like an outage in a certain region or unavailability of resources in a in another region, a new region simply opening up and a customer saying hey. I want to move my deployment closer to where I am so reducing the latency between the client and cluster next thing is, and even if we already start by migrating between regions, we also would want to do the same between different cloud providers, and maybe the last migration option was, of course, it's also great.

A

We have a number of people in the community or also customers who are running their same self-deployed uh rangodb cluster on kubernetes and, of course, we also want to offer them an option to migrate into a rangodb cloud. So in short, our solution to that or what we want to achieve is a migration between different kubernetes clusters and, ideally also between different regions, providers, etc, etc.

A

Maybe just briefly, as we talked about this upgrade procedure, what we do as like on on a pretty standard routine is kind of like this two-dimensional update, either upgrading the kubernetes version of a particular cluster stepping up or updating the orangutp version from I don't know, 3.82 to 3.83, etc. So this is something we really do in like a daily operations level and it results in kind of the step function. But I said there's some limit to that.

A

So what we would actually like to do is not just upgrade the same kind of like still fixed cluster, but we would like to be upgrade to actually also move or upgrade in between different kubernetes clusters, and this could be back and forth. uh So in the end, we actually want to move from our two-dimensional upgrades to three-dimensional structure in which we can upgrade or move so with kubernetes cluster.

A

It's not always an upgrade, but basically move move on that z dimension as well, um so um the challenge is there and um therefore I would actually love to hand over to my colleague adam here, because he's actually the mastermind behind all this technical implementation of that. Thank you.

B

All right, so from this specification we went out to the requirements which was specified to satisfy the customers which we have. It is a little bit tricky to have classes which run, for example, one class around 100 deployments behind to make sure that all of the services are all of the time provided to the customer and the customer in the best effort shouldn't see even difference after the immigration during the immigration, there can be small difference. That's why maintenance windows are useful, but after the immigration there should be no difference on the cluster itself.

B

Also, we have to remember that database and client migration is not atomic operation. Remember when, for example, customer want to migrate from one region to another, he also probably needs to migrate his applications.

B

We need to have in mind that it will not happen at the same time, so there needs to be still backward compatibility where all the endpoints used by the customer still gonna be reachable for everyone uh outside also from the other side, while we are using dns to you to propagate or database to the customer, we need to have in mind that there are some propagation times of the dns. There are up to 24 hours in this scenario, even if we migrate from one place to another.

B

Of course, we will have different load balancers different ips behind that means that we need still to propagate the old service for at least 24 hours. Of course, configurable depends on the customer use case, and after that we can simply cut down their cluster about latency. Also latency should be as low as possible. We are using tcp, so each bump, in the latency time, between two servers increase the time of the queries for us.

B

A lot I can say and about the performance it shouldn't be affected in the matter that a customer can have decrease performance in the queries, but not getting timeouts on the query. Execution about the networking, also very important firewall for us was reliability. This is due to fact that customer is using the service all of the time. He cannot stop using it, so he should be able all of the time to reach data which is inside about other stuff, like security and reachability.

B

This was also very important stuff for us, but in principle data is data shouldn't be able to leak outside of the cluster. We are migrating data between different regions. That means that we need to go through the internet if we are going to cross the providers.

B

If we are in the same provider, we can go through the secure path inside then. This is not a problem and also there should be nice cooperation between all components used inside. It is not able to do this to do migration. Only from one side, all of the components needs to be aware, like dns load, balancers operators, even data centers needs to be aware that something is going on with. After this requirements, we came to four options which could we use to migrate.

B

First, one was the direct networking between two kubernetes clusters. This was an example of, for example, on the first stage we discussed about the vpn, where ports from the one cluster will be able to reach the pot in the second cluster to the port. Ip second one was the something which was most universal for us.

B

It was cube, cube, ctl, port forward functionality like this allowed us just to expose the api of the kubernetes open the tunnel and be able to simulate the traffic just pass the traffic between to the cloud clusters next, one which came also today, as my our mind, was the port internet exposure. We wanted to use their hot host port to be able to reach to be able to expose the ports through the host ip in case.

B

If you have uh cubelets exposed on the, if the nodes have access to the internet and they have assigned elastic ips, for example, and just go- go through this site on the last step, we proposed load balancer part. This is. This- was very this required a lot of the evaluation, because the mapping, the main goal of the load balancers of services in kubernetes, was to load balance the traffic between different pots. At the same stage.

B

We wanted to use it in a little bit different way to be able to map one pot to one port and be able to expose multiple ports through one service, for example, if you need to migrate three services. At the same time, you need to expose three different ports and you do not want to create three different loadable answers, because it did not make sense after proposal.

B

First pocs two options were picked to be implemented, so direct networking which I will describe in the next slide and service vote mapping based on the load balancer, which I will describe in two slides all right. So what works for us? For the first scenario, we picked the silum cluster mesh mechanism, which allowed us to interconnect two clusters which we had anywhere on the world in the demo, which I will present, it will be between europe and united states, and all of the ports were cross-reachable.

B

That means that we could reach the pod ip from cluster 2 from the cluster 1.. It was very useful for us, but required additional logic which allow us to manage the endpoints inside. This is due to fact that when you specify the service, you can provide the selector, but, as you know, selector is limited to one namespace about that we got limitations.

B

Podcadr cannot be conflicting because, if they've gone, a conflict, pos can get the same ips and even routing will not work properly, and it required one important precondition before you could do. It is that celium needs to be installed on the cluster, be the network layer on the cluster and also needs to be ready for the cluster mesh.

B

This was not always the case due to fact that, during her test, when we tried to change the network layer, it could provide interruption of the service due to fact that one of the pots were on the new networking schema. Other pots were on the old. That's why the second option was also important for us. It was service load balancer.

B

This was used. This could be used almost anywhere due to fact that we did not care in this scenario about port cidrs. We have just load balancer, which internally maps to some ips and in principle, from our point of view, we did not pay attention for it. We just needed ip of the load balancer range of the parts that was fine.

B

This worked on all meyer cloud providers without any adjustments from our site, and this could provide us functionality to migrate specific deployment, so specific customer deployments between the regions or even cloud providers about the limitation.

B

There was limitation about load, balancer implementation because, as far as as we know on managed kubernetes cluster, we have this load balancers in cloud. We have these load balancers already implemented. But what with bare metal, if you do not have anything which implements for your load balancer and about latency? There was issue just on the bare metals that, if you need to provide something, then this brings you additional latency in your connection on cloud provider, there was no huge impact.

B

It was just one two milliseconds connection additional latency and in principle it was just invisible for the use case. uh First scenario for the direct connection worked perfectly fine for the upgrades of the kubernetes clusters and second scenario works perfectly fine for migration of the cluster between regions or even continents, all right so from the network perspective. If we have this load balancer and direct connection, how to make sure that our service, which is tasteful, will be able to talk to the proper end point on other side.

B

This is due to fact that if you try to reach database, you have usually some of some access points. So, in our case coordinator, which needs to know where data is placed in this scenario, all of the services and expose for us, the endpoints on which they are listening and they are reachable- and we had to somehow make it transparent for cluster, which we are using.

B

That's why we used services to communicate to each bot, and in this scenario, when the bots were in the same name, space, we had service, which was called service. One was pointing to port which was behind with proper selectors, but in case, if we migrated the spot to other data center. This selector didn't work anymore. That's why we have wrote small.

B

We used this benefits of that you can use definite, define the endpoints in the manual way. So in principle we just use the same service. We just removed the selector and we defined the endpoint by our own. In the scenarios like described here with the direct networking, it was just pointing to the ip in the different cluster of port, which was already scheduled there and for the load balancing method. It was pointing to the one of the parts of the load balancer, which had one port behind on the second data cluster.

B

Okay, so how the immigration looks like, I would further describe what we got on the left side. We had got our operator with one service. On top of it, it was the load balancer service and three database instances behind. On the right side. We got azure cluster, where we first created the service itself in the service. We didn't provide the selector, so we specified that all of the endpoints which are in on the right side so in the second data center, are pointing to the instances which are still in the first data center.

B

Then doing migration with picked first member out of the first data center moved it to the second data center, and in this scenario we were able to edit the end points of the services to point not only to the instances which are in the same data center as the load balancer, but also to the other data center. This allowed us to have the failover recovery in case, if instance in azure, went down it's normal. That it can went down just normal restart.

B

There was still fall back to the instances which are in the which were in the aws. We were proceeding with that instance by instance, so we picked one instance from aws, moved it to azure and modified the inputs.

B

After all of them were already moved, service in the aws was was not pointing anymore to the any instance in the aws cloud. It was just pointing to the instances in azure operators left in azure aws, but in the next step it has been moved here. We have this propagation time which we need to get for the dns propagation, so we waited until there is no any more traffic through the aws and after this has been done, so everything has migrated. All dns propagation took place.

B

We finally closed the aws cluster and we were finally migrated to the azure without downtime in principle, and now we will go to the small demo, which will present the benefits, the benefits and the side effects of the migration.

B

uh On the slide we can see uh in the description we have on the left side of each line so eol or us we have defined where the cast client is placed, then on the second place, eu or us we have defined where the service load balancer is placed in scenario on the left top. We have information that we are contacting from a group to the cluster which is placed in the error. It's a load balancer, which is placed in the error on the right side. We have information, we are contacting from us to the.

B

U we see that huge increase in of ping due to that distance on the bottom left side, we see that this pink is almost two times higher than other. This is due to fact that our traffic first went to the us and then had to go back to the load balancer to the eu.

B

After that we see u.s u.s communication. It is exactly the same, almost the same thing as the communication between usu. Just this goal. The traffic needs to go through the same path, all right, just a second, I will here we have the live version of it.

B

I will go first to.

B

The migration this is comment which we provide to the is it oh, it's too small.

A

Can you, I think you can just control plus.

B

All right so here we have information which we need to provide to our operator to migrate the clusters. In this scenario, we define that in the context error, so all of the pots which are placed in europe. We want to migrate them to the second cluster, which is in our case us. I will issue this comment.

B

And on this on the side, we will see just.

B

On this side, we'll see how the endpoints change, as you can see, one by one coordinators on top, you can see the eu on bottom. You can see the us cluster and, as you can see, the end points for both clusters are changing all of the time in this scenario, in u.s, once the pot become ready in the us, the end points for the service load balancers, which is database and database ea changed itself to the new values which are only in the same data center which reduced ping for us.

B

This is this proceeds one by one. So, each time one pot is killed in one data center. Second port appears in the data center in u.s.

B

This is currently going home. The second one started now we will proceed. We proceed with the third one. We will see that third, one end point disappeared from the us just a second. It will just proceed and third boat will start in the us in a few seconds.

B

Just give me a second: it is about 33 sec. It should take about 33 seconds.

B

Yes, going to be, okay, third part has been killed.

B

And after third part will be started in the us, we'll see that all of the end points which were in the? U are pointing to the both in the u.s and from this point of time, our service is fully migrated and due to the fact that we use the services inside everything, went in the transparent way for service, which was in this namespace.

A

B

I'm going back to presentation.

B

And what we learned to this uh to this implementation phase and investigation space that what did not work for us.

B

We got a first idea about the port forward, but in principle it gave additional overhead in case of the days, and it just was not there a regal eligible solution for us, because it added like 40 milliseconds of the delay, even if we migrated within the same region also exposing ports on hostport via public ip didn't work due to the security, because our data needs to be protected and hidden in the private network, and we need to have the full security like lobby like the firewalls or security groups, to protect access to it.

B

We also had, on the first step plan about manually meant managing the end points during the migration, but the problem was the scale you can do it for one customer, but not for 200 customers in the regular period, and why did we came with two solutions? Instead of one?

B

This was due to the different kinds of the updates when we updated when we had to migrate whole cluster, it was easier for us to set up cluster mesh and do it for everything what is inside, but if we had to migrate just one cluster within the kubernetes cluster, it was just unnecessary to set up the service mesh. We could do it to the load balancer and do it just for this particular deployment for the future.

B

Our plan is to provide the operator in two namespaces right now, when we were having operator in hi, we were providing functionality which use the kubernetes api leader election, but it will not work when you are in two kubernetes clusters. We need to came with all better solutions which will allow us to make sure that at least one of them will be active. This is due to the fact what in case, if first data center will go down.

B

The second needs to still work and based on this migration, we came to conclusion that distributed clusters can be also idea. When you will come to the you will use three different regions to spin off your cluster to be able to handle the single region, fails; failure which might be very useful for customers which demand such a resiliency of their solution, all right, so yeah thanks.

A

Thanks very much feel free to drop by. If you have any questions about avocados about uh cloud, migrating services, graph databases or anything else, our boost is stand 72. I will leave pablo on two, but easy. Just follow the avocado or just drop by here thanks very much. Thank you.