Istio Community, 27 Aug 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Istio August Meetup/ Istio Multicluster for Zero Downtime Migration- by Zufar Dhiyaulhaq

Description

In this meetup Zufar Dhiyaulhaq talks about migration of workload, archiving disaster recovery, and implementing active-active in Kubernetes between cloud providers. In this talk, you will learn about the Istio multicluster pattern implemented on GoPay.

A

First of all, thank you for the opportunity from the istio community to let me share share this presentation. Let me introduce myself, I'm a zoofar, a cloud engineer from gocheck, and today we will discuss about uh how we using istio multicluster for zero data migration, and probably I can also share the active active. uh Where is that uh traffic between kubernetes cluster so yeah uh in this meetup?

A

I hope that we can learn some pattern uh about this uh setup and I also really uh wanted to see some feedback from the history community about about the set. So let me give your introduction.

A

From our past experience, I think yes, yes from our past experience, uh the migration is always painful, uh so, for example, between fertile machine to kubernetes or between kubernetes cluster, as we know that microservice is actually crit dependency, each other. For example, when a single surface is a migrated or down, it can actually affect multiple services.

A

A

The simple, the very simple migration step is: we actually need to create a gateway or ingress between the the old kubernetes one and the new kubernetes. That's one! That's exposed this, this service that need to be migrated. Let me give you an example.

A

So we have a kubernetes which is kubernetes cluster 0.1, and we have three services that communicating each other, that I mean creating dependencies so, for example, merchant service calling payment service and after that payment service is calling banking service.

A

When we migrated the payment service, the most simple way to do that is actually set up a gateway both for payment services and banking service.

A

It should look something like this, so merchant service will collocate me that gateway will be forwarded to the payment service and because payment service also communicating with banking banking service. It's also need to communicate to a gateway.

A

So this is actually very simple topology in reality, in our cluster or probably in in reality, we actually have more complicated, topology, multiple surface, calling each other that creating a dependency and it's really hard to migrate.

A

uh In our past experience, we actually migrated from fertile machine to to to donuts in 2016.

A

We actually implement a service discovery in application or in application level, which means that we need to refactor all of the services implement the service discovery in the code, and we actually create our own solution, which is console and xds, because in 2016 we we don't have no better solution for this one and this setup.

A

This immigration is actually still affecting us until today, because no one is actually managing console.

A

The console puzzle is very old version and no one, no one want to touch that uh the the running console no one want to upgrade and yeah, and it's affecting our day-to-day basis.

A

So I will introduce about copay. So go pay is uh like a money for go check. uh In golpe, we have 300 more than 300 micro services more than one. What is that? 115 million apo call ipa call each week and more than 3 000 deployment every week that actually spread into more than 10 kubernetes cluster.

A

So uh we are in progress actually to move along all of the workload to steel, uh to htms, from console solution that that uh that we showed before and actually this this moving progresses will be presented uh uh from my college. He will present about immigration about this. uh This story in kubecon, north america 2021, so I probably need to have in short introduction about steel multi-cluster.

A

I believe some people here is already know the concept of steel multicluster, but I, I probably will uh will explain some of it, so we actually already implement uh the old multicluster setup in 1.6, but luckily we, no one, no one or deaf or or or or system team use, use this concept, and uh so istio actually introduced a new concept for multi-cluster and in this concept we must understand, at least for for for object, which is single or multi-mesh single or multi-network, singular, multi-control, plane and and the thrust model.

A

So the first thing is: uh is a mess so to easily understand a mesh? We can say that every surface in mesh is is unique even in different custom. You can see that, for example, you have a surface b in cluster west, and you have surface b in cluster is both of this.

A

Service is actually referred to the same service. They they do. They do the same thing uh if the west and its cluster is actually share, share share the same as uh the uh the multi-message is actually opposite of the single mesh. So every surface uh in in this uh concept is different. So, for example, you have a surface b on what is that on cluster clusterwiz and surface b? Also in the password is, if we use uh multimesh, they refer to the different surfaces, so they, even even the name is, is same or similar.

A

They do. They do different thing.

A

A

Also, we have a network concept. uh The first thing is single network, so in single network setup, so every workload or port or deployment or yeah, I see it workload.

A

Every workload in this mesh can communicate each other directly without the gateway. So, for example, you have a bot uh or workload in net uh kubernetes a and what in kubernetes b, they can directly communicate each other without the gateway.

A

The multi-network is actually opposite of single network workload between kubernetes cluster in the network is they cannot communicate each other, so they need to communicate via gateway, but this concept have additional advantage like uh you can use overlapping id between kubernetes cluster. So you do not need to maintain the ip between kubernetes cluster. They can use overlapping ip and we can scale the network separately between uh this kubernetes cluster.

A

uh Actually, uh multi-network uh uh is a concept that I personally like, uh rather than a single network, so istio actually actually automate automatically handle the communication between this cluster via is west gateway. So you you have a specific way for for this communication and it's ensured the security since it and since it's automatically encrypted by mutual dls, so communication from service a to surface b is actually a via mutual dls. So you so you do not need to worry about it, so you also have a concept. uh It's. It is the control pane.

A

uh If you have a single mesh setup, uh you can set up the control plane in each cluster, for example in cluster west and cluster is or you only need to set up. You can set up the centralized control, pin in only a single cluster in this example is clustered os.

A

The drawback is the drawback with this setup is when the primary cluster is down, which is the in this example, is space cluster. We lost the control plane and we cannot update the side curve or gateway configuration.

A

uh There is also another concept which is multi-control plane in multi-control pen. You have, uh you will have a control plane in each cluster and the control plane is communicating each other uh via kubernetes api. To and after that, uh I prefer this because uh when, when something uh wrong with the other cluster, we can just make this, for example, is if something wrong with cluster one.

A

uh We can just make the cluster too independent, because uh the control pane is there yeah and we can actually remove the the mesh and we can run the cluster to it problem and the last concept is the truss model, so in multicluster you need to have the same truss model.

A

You need to have sorry the same trash between multiple clusters, so we can achieve that uh using the same road ca between the cluster itself and installing the intermediate ca in in cluster 1 and cluster 2..

A

So, as as far as I know, the prerequisites for for this new multicast series, we need to have it's better to have a dns proxy for simplicity, which is is enabled in easter one dot age and on kubernetes note we must take the note with topology and topology region and topology zone, I believe in mass in all of the cloud providers. They already automatically do this, but if you install the kubernetes in in, for example, using uh using your own setup, probably we need to set up the sorry.

A

We need to take the note with with with these things also, you will have additional tag which is sub zone. It is from steel.

A

If you use a multi-network mechanism, well, you can use israel's gateway and you must implement through ca between the control play.

A

uh I have some recommendation, so if you have a infrastructure component like logging or monitoring that it's actually only specific to this cluster you, you should not uh include this to the htms, because we set up a multi-cluster.

A

So we uh based on that concept, you mean uh the mesh, the control plane, the network. We use a single mesh with a multi-network which is multi-primary, uh sorry, a multi-control plane more, which is multi-primary, and we just also did different network.

A

We go with this because this is the easy, easy easiest way to implement the multicluster, because we do not have to think about the uh workload to workload communication between cluster right, so we do not need to set up vpn or something like that. We just use the object and multi-primaries control. Planes also ensure us that there is no dependencies between cluster itself.

A

uh Okay, let me give you some migration flow, so we have a two cluster. uh Let's say we have cluster 0.1 and console 0.2.

A

We want to make it a hello world, hello, world workload or services from cluster 0.1 to customer 0.2.

A

So the user, in this case is, is a user from from outside of the kubernetes, but you can also call the hello world directly from from inside the cuponets.

A

So the first thing that we can do is actually to set up the mesh itself right. We need to make sure that kubernetes, cluster 0.1 and cluster 0.2 is in the single mesh and they can communicate each other via student swiss schedule. After the mess is set up or the istio multi cluster is set up.

A

You can just directly install the sorry, install the hello world directly to the cluster 0.2 in this case is still public gateway or or the mass will automatically load balance the traffic between hello world in cluster 1 and hello world in cluster 2..

A

It's automatically do that and if we already confident to migrate, we can just delete the service in cluster 0.1 and automatically uh the traffic from gateway in cluster 0.1 will automatically forward it to the cluster zero pins in this state. If something happened, we can just uh install or roll back the the hello world in cluster 0.1.

A

If something happen. In this case, we can just scroll back right, but if we already confident to to migrate this one, we can just install the same gateway uh like the kubernetes cluster 0.1 and just redirect the user or the dns record to point to the gateway in cluster 0.2 and the migration is very easy. The user does not need to change a domain. There is no need to change anything and we can just duplicate the cluster 0.1 if, if all services is already located, so let me give you a demo.

A

So, okay, I will just okay. Let me control see this first and so wait a second okay. So we have a cluster which is, I have a cluster 0.1 and 0.2.

A

Let me see if this connects okay, so we have customer 0.1 and cluster 0.2.

A

And we have a bot that need to be migrated. For example, we want to make that example hello, world v1 and in kubernetes 0.2. Let me just delete this first, because it's already there actually so, okay. The first thing I need to do is: I will remove the the services.

A

Okay, so we will only have uh example, hello, world 31 in cluster 1, and we will have no example, hello, world v1 in class of 2. It's terminating so.

A

Let's scroll the the service, uh this is the ip of the public gateway in cluster 0.1.

A

So you see that it's going to p f c and something like that.

A

So let's say that we want to make it to cluster 0.2.

A

The first thing we need to do is we need to to install the example halo of hello world v1 in cluster 0.2. So let me just scale it up.

A

When it's scale it will automatically lock the range. Let's see it's still 1.2 running.

A

Yeah, it's starting to look blends to kth. I believe this is kth right. It's slot balance between kth and what is the six cn something like that and it's the traffic automatically load lots because of zero fans in this state we actually uh crawl from a gateway in paso 0.1.

A

If we already confident enough, we can just remove the class the the service, which is hello, world v1 and cluster 0.1. So let me just scale it down to zero.

A

And the traffic seems already redirected to the cluster 0.2 automatically so yep, and if we already confident enough, we can we can install the same gateway in cluster 0.2.

A

So you see, I'm also called to the same domain, but with the different id which is in this in this case is 5164.. The previous one is 50 to 5. Sorry 247 right when I call this yep.

A

It's working so we we curl this domain to the new to the new ready set to the new cluster and yep. We can just directly delete the gateway and change the the dns config. Sorry, the dns record to point only to the to the new gateway, something like that.

A

So yep sorry, uh I think I already demo this one. Actually I created a video but yeah, so we also can implement this thing for active and active uh flow. So, for example, uh in cubase 0.1 we have 100 percent traffic go to this cluster and we want, uh for example, 10 traffic or 15 traffic to go to the club kubernetes cluster 0.2.

A

We can achieve this automatically via sto sorry via still service mess, so we can configure percentage of via locality, load, balancer and yeah. We we have active and active kubernetes cluster.

A

We probably also can set up uh some uh dns record, which is a dns resolution based on the region. So if you more close to the cluster 0.1, you go to the cluster 0.1. If you were close to the cluster 0.2, you go to cluster 0.2, all right and still multi-cluster can also use for failover cluster level. So, for example, you implement ninety percent of traffic and ten percent of traffic.

A

uh Ninety percent to the cluster zero point one and 10 percent go to the cluster 0.2.

A

For example, when the kubernetes class 0.1 is thought completely down, we will have a downtime. Actually, if we do not do something in in in the user level like like, for example, implementing a health check in the dns record, or something like that, but the migration or or the fallback is pretty simple. We only need to create the gateway, the same gateway in cluster 0.2, and we just change the dns record to point to the cluster 0.2 and yeah the user can can can can access this service again.

A

uh From our site, we actually have feedback for for for this, for this uh istio multicaster, so actually uh the lot balance the lot balancer on cluster level is tricky.

A

So since we cannot not balance the traffic based on the kubernetes cluster, so currently the locality load balancer is only support based on the region zone and subzone, which is, uh I showed you it before, and.

A

And yeah we have a hard time to define the correct lot: balancer rule from the destination rule, for example, 90 percent go to the cluster one and then and 10 go to the cluster 0.2 yep. I think my percentage, my presentation is done. It's it's. It's probably just quick presentation for me.

A