Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2018 (Copenhagen), 4 May 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Multicluster Ingress Powered by Kubernetes Cluster Registry - Greg Harmon & Nikhil Jindal

Description

Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).

Multicluster Ingress Powered by Kubernetes Cluster Registry - Greg Harmon & Nikhil Jindal, Google (Advanced Skill Level)

Multicluster Ingress (MCI) is useful for a variety of scenarios: high-availability services, low-latency access, setting up hybrid multi-cloud and on-prem architectures, canarying clusters with new release versions and running experimental releases. In this talk, we will show how we can leverage the Cluster Registry to create multicluster ingresses with Kubernetes-style semantics. Using a controller against the Cluster Registry server, we can watch multicluster ingress resources and the list of clusters. This allows us to manage MCIs and automatically add/remove them from clusters as they are added/removed from the registry using label selection. We will also demo a sample application and show how users can make it resilient to cluster failures using multi cluster ingress.

About Nikhil
Nikhil Jindal (nikhiljindal@github) is a Software Engineer on the kubernetes team at Google. He had started on kubernetes by working on API machinery and is now focussed on multi cluster problems. Before getting the kubernetes bug, he was enjoying working on Google Maps.

About Greg
Greg Harmon (G-Harmon@github) is a Software Engineer on Google’s Kubernetes team. He’s currently contributing to the multicluster efforts. Before Kubernetes he worked on Google Compute Engine for 5 years, including the launch of live migration.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Hi everyone, I'm nikhil from google, and I'm here with my colleague Greg and we are going to talk about multi clustering is powered by kubernetes cluster registry. The lot of powerful words there so we'll try to break it down for you the way we are going to structure this talk is we are going to start with before diving into multi clustering else.

A

We will just give an overview of why people run multiple cluster and what does it mean running multiple clusters and then we'll go deeper into specifically multi cluster ingress and then we'll also talk about command-line tool. We have in the open source called cube.

A

Mc I will give you a short demo of that and how you can use that to create multi, clustering guesses and then would talk about cluster ages, 3 and how we can build controllers over a cluster registry and with that idea, how we can do multi cluster ingress and provide more magic about by using multi class angles using a controller with less registry and we'll do a demo at the end feel free to ask any questions as you have during the talk or in the end, we'll have some time for Q&A as well.

A

So there are lot of reasons why people learn multiple clusters. We've highlighted a few here. The most common one we hear is high availability people whine. Even if one cluster goes down, they want to be resilient to zone failures or cluster failures. Even if one cluster fails, they want to deploy their app to multiple clusters globally in US Europe Asia. Even if one new cluster goes down, they still want to be able to serve their customers, so they want high availability and that's a common reason to run multiple clusters. Another reason is to for scalability.

A

Sometimes one cluster is not enough. They want multiple clusters they want. Do. Customers want to avoid vendor lock-in. They want to have clusters on Prem on cloud different cloud providers. We easily be able to switch I across them. They want geo proximity, so they have global customers all through the world like in Asia Europe us they want clusters close to these customers, so they can serve them better, with least latency there. A lot of other reasons as well, but these are some of the common ones were here and like I, hope you connect with them.

A

Once people run multiple clusters, there are lot of problems they run into which they don't have with single clusters, for example, now they want to deploy their app across multiple clusters they've to ensure that they keep it in sync, in all the clusters, whenever they're rolling out a new version they've to roll it out across all, and how do you do rolling update?

A

Do you do it one cluster at a time or you'd like percentage-wise, you, roll and all of them, and then gradually roll it to like 15 percent 20 percent hundred percent across all of them together or one by one? How do you want to do it when you're rolling it back? You want to roll back across all clusters. So how do you do that easily and like build tools around that? How do you configure network resources?

A

So this is something we'll go deeper into on this talk like how do you configure your services and English so that customers are outed to your apps, which I already deployed in multiple clusters? So how do you do that? Now? You have multiple clusters. You want unified policies across all of them, like you want the same. Namespaces similar back rules, the same developers get access to all those clusters. You want to ensure the unified policies.

A

You want failure. Domain isolation like I, talked about high availability. That's one of the main reasons. So, even if one cluster goes down, it should not affect other cluster, so you want multiple clusters and you want to like roll the same version across all of them, but you still want independence across all of these clusters. They should, if there's an error in one, it should not affect any other clusters. Each cluster should be able to work independently without the other one.

A

So these are some of the common problems that people run into running Multi cluster, but- and there are a lot of solutions, we are working on it as part of segmenting cluster. So, but in this talk we are going to focus on multi clustering. This so, which was the second bullet point so a lot of times. Let people have a question of like what do you mean by Multi clustering?

A

It's like taking a step back and going to the basics, what we mean by multi class aliases and to explain that we'll start with what we mean by single cluster ingress ingress is kubernetes abstraction of the various load balancer providers that exists. So there is like GCP load, balancer, there's on Prem load balancer there other cloud provider load balancers and in kubernetes we have our abstraction called ingress. So users don't need to know about what GCP load balancer is or what they're on Prem load.

A

Balancer is the right and in respect and their ingress controllers then, which go and configure like Google cloud load, balancer or nginx load balancer on my load, balancer they're. All these ingress controllers that exist for various load, balancer providers, so what they do is they configure whatever the the implemented implementation load? Balancer is, in the end, users get an IP address which they can give to the users and users at that IP address and they're able to reach a particular service. So in this case like they want to reach service food.

A

So this is how they configure a single cluster load, balancer, extending the same concept to multi cluster load balancer. Now they have three clusters a B and C, and they are the same service who deployed deployed across all three clusters, and this will want to be able to write one in respect and be able to configure a load balancer which gives them one single IP address, which they can then go give to their customers and customers can hit that IP address and be routed to any of these three clusters.

A

So this is multi clustering. This. Why people want multi class saying? Is there are lot of advantages? The most common one here is they want? They want to do least latency based routing, so they want to have clusters where their customers are so in Europe. You know all across the world and they want to be able to have that single IP address, which then routes customers to the closest cluster.

A

So when customers from Europe hit that IP address, they should be routed to euro cluster and when customers from Asia hit that IP address, they should be routed to Asia cluster. So that is one of the very common use case. We hear some other use cases I, like I, said they want to be resilient to clusters going down so by default. Europe customers should be routed to Europe cluster, but if it's down they should still be routed to other clusters.

A

So, rather than like they're seeing photo force, they should still be routed and be served and which makes the application resilient to cluster is going down. And now you can build global services. Based on this architecture, you want scalability, you can handle a lot more requests. If you have the same app deployed across multiple clusters, if it was only in same single cluster, then you run into some scalability limits. You want, you might want to cannily clusters and you want to can renew kubernetes release versions or you want to carry your own deployment.

A

You want to do Bluegreen deployment. So by keeping multi cluster ingress, you have like a new kubernetes version comes up, you can switch 5% traffic to it and then gradually increase it to hundred percent or just try it out in your staging environment and then switch and rolling back is easy. So you cannon clusters using multi clustering.

A

This people want a bedside ups as well, people want they have their own data, centers and they're moving to cloud and well, there are across multiple clouds, so multi multi, color strangers helps enable them when they deploy their app across multiple cloud providers are on Prem. They can still get a single IP address for all of those apps. So these are some of the advantages people get with multi clustering. This.

A

So the problem at hand is people have an ingress spec and they have a list of clusters and then how do they configure a load balancer to get this single IP address that can now traffic to all the clusters and their various ways to do it? One of the way we do it is using this cube. Mc is CLI which we've open-sourced and this works today, so you can try it out the way it works.

A

Is you run qmc I, create you give it a load, balancer name, you give it your English spec, which is your standard community, singular spec. You give it your cluster list, which could be your cube. Conflict cube. Config already has a list of clusters, so you can use that to give lists of clusters to cube mci and what it does. Is it configures. So today, talks to Google cloud load, balancer and configures Google cloud load, balancer for least latency mode, so I talked about what are the various ways you can use.

A

Multi color strangers the default mode way. It works is the least latency one. So it says Google cloud load balancer in such a way that users from like in this example, for example, Alice from California, would be routed to u.s. cluster and Bob from Berlin would be routed to euro cluster.

A

So I'm going to show a quick demo of how the CLI works and I hope it works.

A

So, for my demo, the setup I have is I, have two clusters, one in Asia and one in Europe, and this is my cube. Config file, so I'm just listing the contacts in my cube conflict so cube, conflict or MLF has two contexts: cluster Asia in cluster Europe and so I'm going to use this cube conflict to create my ingress, so cube, MCI has the simple create command and all I need to pass straight.

A

Is I, give a name here: I give nuke um CIA's own printer I, give it my ingress spec and my cube conflicts. So just those two things and if you go and configure Google cloud load balancer to send traffic to these two clusters.

A

So if any of you have tried configuring, google cloud load, balancer, there's a lot of steps you have to do. You have to configure and shade back-end service, fiber rule forwarding port. So this would go and do all of that it will put in when it's done with the hell check, with back-end service with fiber rule and hopefully at the end it will say success if it works correctly.

A

So once it's done, what we'll get is we'll get a single IP address that then we can hit from all across the world and be routed to different clusters based on our location. So we'll see if that works, it said success, so I can list my ingresses with queue MCI. So cue, MCI is a simple list command and then it lists. So you see we created our new ingress. The new cube MCI his own printer, and you see it's there for this demo. I'll also use this existing cube. Mci is one printer.

A

It takes some time to configure the load balancer and like just to save time. I'll use the existing ones that I already have so I'll use. The IP address of that existing MCI that I have I, can cube. Mcl, says a gate status command which gives you information about the IP address, and what clusters is it's better, so I'll use this IP address, 35, 0 1 and, let's see so I'm going to use this app. This thing called GOP ker. What it does is it hits that IP address from different parts of the world.

A

I hope it works I'm using the free version of this service. There's some latency, but hopefully it works. Oh it's going to hit the same IP address from Singapore, Brazil, Virginia, California and so I have two clusters Europe and Asia. So you can see when I hit it from Australia and California. It went to the Asia cluster, which was running in Tokyo when I hit the same from Brazil and hopefully from Island as well. It should go to Belgium cluster, hopefully when it loads.

A

So you can see all from all around the world. Users can hit the same. Ip address and they'll be routed to the right cluster. So that's the great thing we get with cube mci.

A

Okay, so we have this working so what I want to show next is like we've been able to make multi cluster english work with cube mci, but it's still a separate binary that you have to install and you have to configure it to create multi cluster ingresses I think we can do a lot better and that's where it cluster registry comes in and I'll invite drag to tell us about cluster legisla.

B

Alright thanks the hill yeah, so I'm Greg Hartmann, we're both on the multi cluster kubernetes team at Google, so I'm gonna talk about cluster registry, so motivation here is: when you have multiple clusters, you have to keep track of them and deal with a bunch of queue. Config files, maybe have a bunch of clusters in one queue: config or you have a handful of queue, config files.

B

Then let's say you add a cluster or you remove a cluster. Then you need to update those cube, config files, you're working on team. Then you need to get those updates to your teammates and the other devs. So maybe you have something like it's on NFS or maybe check it into get, but so there's kind of ad hoc solutions right now for dealing with multiple clusters.

B

If you're more advanced, then you have some homegrown tooling around how you deal with clusters and their queue configs. So what we're building is the cluster registry. It is a basically a list of clusters as well as associated metadata.

B

Each cluster can have labels, so you can have them as test staging products cetera, and these lets you do standard, kubernetes label selection, it's just part of the metadata field and because it's a standard, kubernetes api. That means you can use the exist tooling, you're used to using for workloads like an acute control- and this is kind of meant to be a building block- that you can build other developer tools on top of.

B

So here's a yeah mph file with cluster and cluster list objects. You see, it's basically got the name u.s. West 1a, it's got the endpoint, it's got the certificate for the endpoint I mean you see right now. It's in alpha we're working on bringing up to beta in the sig multi cluster, so that's happening pretty soon, I think.

B

So, when you have a cluster registry, it's pretty straightforward right. You have a cluster object. It lets you basically crud your clusters. So what? If we put it with a regular kubernetes api server, then you can have it with all your other resources. You have ingressive services and all the other standard resources. So this lets you build more complex use cases and lets you have your cluster list side by side with the rest of your abilities, resources.

B

So one of the big value-adds of cluster registry is running a controller with it. When you have a controller, you could watch the list of clusters. You can watch other resources in that API server and you're, probably going to want to watch these resources in each of those listed clusters.

B

So here in this diagram, we have the foo controller, it's watching the cluster list and foo resources, and it's replicating that food to subsets of clusters based on some label. Selector is not shown here. So you have replication is one good use. Another good use is logging and monitoring type things. So, if you have let's say you want to scrape a particular data value out of every cluster, you can aggregate that in a controller and be able to report that backhand sort of across all your clusters in one spot.

B

So for the rest of this or will focus on the MCI controller, this watches the cluster list, as well as the ingress resource in the Cates API server. So what it does is it basically ensures that all the clusters in the cluster registry are part of a load balancer, so it programs the load, balancer API and when a cluster is added to a cluster registry.

B

The controller sees this as a notification and updates the load balancer to say, hey, add in my new cluster to a load balancer and just like that you'd get spread to the next cluster. So, similarly, if you remove a cluster, the controller sees that and automatically updates the load balancer to remove that cluster. You also have the health checks in GCP load, balancer that wouldn't notice that cluster going away if it were to like just disappear yeah and the last thing is that watches ingress.

B

So if you need to upgrade your update, your ingress specification, then that can also get pushed, will also get pushed to each of those clusters. So you know over the CLI, you know it doesn't you don't have to invoke this manually eyes just always running in the background for you, so it makes it much more automated way and much more. The kubernetes style.

B

So nikhil is going to demo the controller we've put together.

A

Awesome so now we know how we can build this multi cluster English controller with listeria, history and, let's see how we can do it so I'm, going to show a demo and before I start I'd like to thank Rohit who built this demo. He works with us at Google and thanks for building this demo. Hopefully it works.

A

So for this demo the setup we have is I have a cue cuddle that is configured sorry. You guys can see this okay awesome. So my cube cuddle is configured to talk to cluster religiously. The setup I have is a vocalist registry which is running an aggregated mode. The second architecture that Drake showed so now using the same API endpoint I can list clusters as well and I can try normal kubernetes api resources as well. So in this case, I would try ingress. So first I listed my clusters.

A

I have three clusters: one in Asia, one in Europe, one in us at global presence, I'm using GK and so I will then create an ingress so just to show what my ingress looks like. This is standard kubernetes, even if you were doing a single cluster ingress. This is how it look like you give it an IP address. You give the service that you want to expose so in this case I'm exposing these zone printer service. So it's the same example I used in the earlier example. What this is own printer app does.

A

Is it just been steal, location when it's being served from so when, when I hit my IP address? The request goes to one particular cluster and as a response, I get the location of that cluster, that that service that that request go to Asia. You know us which cluster did it go to so we'll be able to verify that it's working correctly.

A

So this is my English and, as Greg mentioned, like one of the advantage of running this controller as cluster registers that I now don't need to download a different QM, CI binary and it I can just use standard cube, kubernetes concept, so I can just run cube, cuddle, create ingress and the multi clustering risk controller. That is running. That is watching this cluster industry fault list of clusters and list of ingesys.

A

So it will see or there's this ingress that has come up and I have three clusters, so I should then go and configure Google cloud load balancer to send traffic to all these three clusters.

A

So once the ingress is created and Google Cloud load, balancer is configured we'll, be able to hit that IP address from different locations and be able to see that we are routed correctly.

A

So once I've created, ingress, I can list the angles to make sure it's created and I'm using standard cube code will create commands.

A

So again, you'll see this new demos own printer angles that I just created but I, also in existing zone printer, which I'll use for this demo. So this is the IP address, 35 2, 0, 1, 67 42, that I'll use and what I'll do is.

A

So here I have two instances: GC instances, one in US and one in Europe. What I'm going to do is I will call that IP address from US and Europe and make sure that I'm sent to the right cluster. So let's say I call it from Europe, and here my request is being served from neitherlands, where my Europe cluster is and similarly, if I call the same IP address from us, it's being served from Iowa, where my u.s. cluster is so.

A

This is the same setup as we saw before like now at the single IP address that I can head from all over the globe and I'll be routed to the same IP address. But controller has some more advantages as well, and we want to show you those once I have this set up working. The advantage is that it's always watching the list of clusters and ingress and whatever changes I make. It will be able to observe that and update GCL be as per those specifications.

A

So let's say what, if I want to delete a cluster or like I, delete the app in that cluster. So, in this case, let's say: delete the US cluster. So unless my clusters I have those three clusters, I'll delete the US cluster. So what happens? Is that the multi-class, alias controller that is running observes that, oh, that this cluster is now deleted and I need to update the GCL B that was configured to send traffic to three clusters? It should now send it only to two clusters, like I, am running.

A

This cube cut' will get clusters command. The controller is also always running that command and watching that now, there's only two clusters. So what should eventually happen? Is this curl command that we were running? That was the request was being sent to Iowa? We will see that it starts detecting that that cluster is down, and now my request would be routed to other clusters and you can see how seamless this was like as a application developer or anyone who was maintaining the cygnus.

A

They didn't have to make any change like whenever the cluster list was changed. The controller automatically observed it, update that GCL B and to my customers it was all seamless like they didn't see for like us. Lester was down, but they didn't get to see that a cluster is down or they didn't get any photo for. Their request was still served and now they're being served from neitherlands in Tokyo, and eventually it will settle down to once all the BG be routed.

A

I, the propagation is done, they'll all be served from neitherlands so like this is amazing. Why cuz much didn't even get to see that the cluster is down and I didn't have to do anything? It was all automatic. It figured out and updated it to send traffic to other clusters, and this was for deleting clusters and same thing. Now you figure out the problem. You debugged it and you want to again add back the u.s. cluster and the same thing votes for adding back as well. You add new cluster or you clear.

A

The you in this case we'll create the US cluster again. So we list we have two clusters: I will create my US Lister again.

A

And now, if I list I get three clusters. Similarly, when the controller lists it really clusters and if I, if I go to my US instance, it will eventually start serving traffic from u.s. Lister. So when you observe it takes a while, when we deleted a cluster, it was instantly.

A

It starts started sending traffic to other clusters because, like that cluster was gone when a new cluster comes up, it still takes time to it make sure they held checks, passes for sufficient time, and then it verifies that it's stable and then it starts sending traffic to the new cluster.

A

Now this is not managing the twisters. This just manages the ingress and GCL B, so that DC will stop sending traffic to that cluster yeah. So I just deleted cluster from the cluster list like that cluster still exists. If I go to GK and list my clusters, it exists but I removed it from cluster registry, so that my ingress controller is watching cluster listing cluster history. It doesn't get that so you'll see eventually it started sending traffic again to Iowa back so as and as an application developer. I didn't have to do.

A

I didn't have to update my English spec or anything. It automatically happens so like this is the real power.

A

No, so the annotation was just to give it an IP address, so it needs an IP address to be able to send traffic to all clusters and in the standard, English spec, it's specified as an annotation.

A

So that's why I was using an annotation but there's no special annotation to tell that it's a global English like I, created it in my cluster registry and so many declare strangers, is watching all India's interest and it assumes that all ingresses that I created in class three have to be spayed to all the clusters that I registered with lister registry. If you wanted like a single genghis, then you directly just create in that cluster and that works as well.

A

Now this is just a prototype early version that we built for this demo. Rohit did.

A

A

Now it does that GCL be supports by default, that it held checks the apps directly and if and if a pod goes down or anything the service goes down, then it will stop sending traffic to them.

A

Is there one GL beeper, so you configure GCL B, there's an IP address by like load balancer and you configure, like forwarding, rule five, our back-end service, all the GLB resources that you need to configure.

A

Yeah, we are actually done with the demo and it's time for Q&A, so yeah.

A

So yeah, so it says the clusters that I'm running those are like standard GK clusters that already have English controllers running and but I do have a multi-class trading desk controller, also running in cluster registry, which is what configures G CL v.

A

It's yeah, it's running, bedless registry, so like I, have a place where I'm running my clusters. You see I'm running this molecular singles controller there as well and multi-class, saying as control of watch's cluster registry for a list of clusters and list of English's.

A

So for this demo, it's just like running, but if yeah, if you want to run it on a production service, yes, you then it has a che. You can run it on a communities cluster or you can a virtual set ups. Yes, and you should have.

A

So this particular demo configures Google cloud load balancer. Yes, we can write an implementation that instead configures engine X to do the same thing. Yeah.

C

In one of previous talks, there was idea to that this cluster Registry, together with some controllers, will be like a replacement for clustered Federation because it doesn't have some security stuff. But what I see here is more, it's more like just a copy of the same resource in different clusters. So you cannot really schedule like 50 90 percent on one cluster and.

D

C

Percent, so is: is there a Silla plan to do it replacement or maybe more advanced controllers are needed, or this multi cluster primitives are needed per every? What's.

A

Your yeah, so that's a great question and we from a lot of customers that they want, like more advanced load balancing words like in this case. There was just this where you send to all the clusters, and you do like least latency, with load balancing good. There are other load balancing modes like you said so, and yes, the API spec today does not support it.

A

So it's not possible for the user to specify it, but we are discussing with sig net work and the interest rate and there's discussion happening in signal to cluster as well about how to extend the API to be able to support more advanced use cases.

A

So you could use it like, like, like water, lynching, we want to build cluster, as you see as a building block that you can integrate with others as well and other tools that you have like. Maybe you integrated with your CI CD pipeline and you just want a list of clusters or you want some other resources, so it has like a CID as well, so you can install a CID and do your own custom like multi cluster, specific resources as well.

A

So you could do that, so it is flexible enough to be able to do all of that stuff. Yes, for this demo, we were just using the existing kubernetes api.

E

Hi great demo, but I have a question related to the cluster registry. How does it help? Let's say the process running in one cluster in cluster mode to access? Api object in the second cluster, like in terms of the security.

A

In like for this particular use case, and in this demo, we didn't have that requirement, and it's not, but.

E

Is it possible, does it help? Does it provide any mechanism or some some tools, No.

A

So another thing about Lester ageist is that it doesn't store the author information about each cluster. It stores the list of clusters and it has pointers to be able to point you to the right. Like credential store like it will tell you it's a GK cluster and then you can run g-cloud get credentials to be able to get credentials, but.

E

Then, for a controller, basically to watch for objects and different clusters, I mean you have to have our buck rules and the service accounts. No, it's how it's done. If you have a multiple clusters, yeah.

A

So, in this case, the multi ingress controller- that's running- does have access to all the clusters, so it has service accounts for those clusters. Okay, but each of those clusters don't have service account for, like other clusters,.

A

It's already open source, so it's on so we should also yeah. It's part of kubernetes slash cluster registry and you can go and try it out today and the API is alpha. We want to move it to beta, so these are some of the pointers. You can go, look at it and it's outside k, slash scale. So it's not tied to one kubernetes.

F

You have to populate the registry manually with all my clusters, so is there something like a cozy place to based all the population or self discovery of so.

A

It's a standard, kubernetes api. You can use q cuddle and that's what we were using in this demo, but it has like an arrest API, so you can write wrappers on top, probably like export your coupon figure into this cluster. Assist you automatically. So it has the kubernetes api. So that's the advantage. You can build all these tools on it. There's none right now, but it's easy to build them.

G

Regarding the bouncer, is it actually rooting for the traffic or is it ready directing I mean? Is it any benefit in terms of network latency that you, let's say, directo the asia customers to the asia Questor? If they all go through the wall bouncer anyway, they would get rooted. Let's say to the US for every request: yeah.

A

So that's the advantage we get with Google cloud like it has worldwide presence and what it uses is in any cast IP which is like propagated to true BGP routing to everywhere. So like Asia, customers are not actually coming to us, like they directly go to a checklist as via BGP routing that IP address is out. It is propagated to Asia and Europe NUS that same IP address. That's the advantage you get with GCP Network yeah.

A

Awesome are there any other questions.

D

Do you have any thoughts on policy-based routing, say I, don't know like a European user of a service who goes overseas say to America, and their data is located within Europe can't be exported to an American cluster, so their traffic always gets routed back to Europe yeah.

C

D

Would this be a thing that you would solve it this level, or would you push that into the application.

A

Yeah I I don't have a particular opinion about that because, like once, it comes to legal issues like this does try close least latency, but there's no guarantee- and if you have a legal issue, that that user shouldn't be routed out of Europe at all, and even if they go to us like they should still be routed to Europe. There's still some more logic. That's required to build. On top of this, it is possible to do it in your application as well. I would say I.

B

Would say it take a lot of smarts from the load balancer to be able to do that, because you'd have to have some sort of key or hash from the user to know where to route it to so that would take a lot of technology. I think to do.

A

You see I'll be dust about cookie waste outing, so that is something you could explode.