Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2019 (Barcelona), 23 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Reinventing Networking: A Deep Dive into Istio's Multicluster Gateways - Steve Dake, Independent

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Reinventing Networking: A Deep Dive into Istio's Multicluster Gateways - Steve Dake, Independent

Service mesh software is challenging the current networking state of the art. With Istio’s gateways technology, it is possible to connect large numbers of Kubernetes clusters. Connectivity between clouds enables cloud native applications to connect across cloud providers on the global Internet. In this session, attendees will receive: * An introduction to multicluster service mesh technology * A technical deep dive describing how crucial technical problems with multicluster are solved including: - Multicluster service discovery - Transparent multicluster connectivity - Secure connectivity over the insecure Internet - Multicluster routing * Detailed explanations of how CoreDNS and Envoy are consumed as building blocks Finally, the attendees will receive a short demonstration of an application running seamlessly on AWS, Azure, and GKE using Istio gateway multicluster technology.

https://sched.co/MPbL

A

Let's go ahead and get started, my name is Steven dake. This is a session about reinventing networking. A deep dive into is Joe's gateway, multi-class or gateways.

A

So I don't know if a lot of people know what an issue of service mesh is I'm gonna start with the basics. The first and foremost purpose of a service mesh is to connect services together when we connect services together, we use envoy that runs in a process next to another process, typically inject it into a pod. The surface mesh connects those pods to those those those sidecars together into a mesh and that's one of the fundamental properties of a new service mesh, so that connectivity is very important because that's how we connect everything together.

A

That's l7 connectivity versus l3, which is what typical networking is based upon.

A

So the next fundamental thing of a service mesh is security. If we look at how the Internet has evolved, there really hasn't been a lot of security built into the internet. The way sto works specifically and with our data plane of envoy, we use TLS for security, and we use pass through mutual TLS to provide multi cluster capabilities in security, and this is crucial because without security people are looking at your traffic. That's not good! When you surf the internet now we have to use TLS certificates when we access websites.

A

This solves that problem for all traffic.

A

So, of course, there's control controls, idea that we can control the traffic we can control where it goes. We can control what it does. I'm, not going to talk a lot about control in this Multi cluster, deep dive, because that is not the purpose of this session, but that is one of the cool features of a service mission. We don't lose this capability with multi cluster, it's something we get just naturally as part of using Sto and finally, there's observability reserve ability is key and one of the most valuable things that sto provides.

A

It provides us via prometheus. You can use prometheus with a variety of adapters. Essentially, it allows you to watch all the traffic in the network. What is doing what your requests are doing, what your responses are doing, what their timings are? Pretty much anything you can think of. We monitor this is huge. This is a very valuable thing. I'm, also not going to spend a lot of time on this and the steep dive, but this this is what askew is it fundamentally? This is what is provides.

A

So in the beginning there was a platform. There was probably the UNIX or the univac, but let's talk about the pdp-11, because I think that's what this is really focused about, so this individual Leonard Kleinrock, yes, he invented sort of invented ARPANET. There was a lot of people involved with this effort. He invented ARPANET with a lot of people on his team, he's a professor. He had a lot of students helping him and what he did. The first thing he did was he connected to wide area networks he connected UCLA and SR I.

A

So these are, these were PDP I, think 11th. They connected over wide area networks. Now that was one of the three equations or one of the three kind of vectors we needed to make networking a thing. So this individual here, oh sorry, the entire internet, 1969, pretty boring, there's four nodes for pdp-11.

A

Send the bill here: Bob, Metcalfe, I, think most people are familiar with Bob. He invented Ethernet in 1973, he created the land and there may have been another land protocols before 1973, but he really standardized a LAN. So this was idea that you could have a bunch of platforms connected locally in a local area, but there was no way to connect the LAN with a man that didn't exist. That was not a thing, so there's platforms and lands there's the ability to connect lands, and then this individual came along Jenny, Strazza czar.

A

We owe her a great date debt networking, because what she did is she invented the gateway. The Gateway is what a router is today. So when we think about gateways, we think about router's she's, a person that did the first implementation she's a person that made it happen now. What does this mean? This means we could connect lands with lands and we could connect multiple lands with multiple lands, and this is where the internet began.

A

So that was the evolution of networking. Let's talk about the reinvention of the networking with this Joe I'm not gonna talk. This is like a geek slide: I love, geek, slides because I'm a geek but I'm not going to talk a lot about this slide other than to say this is SEO in one environment. So if you have a cluster, this is one ISTE. Oh, this is what it looks like these are that can put some of the most of the components. The ability to connect is there, but it's just one cluster.

A

That's a problem. We need multiple clusters. So if we look at what we can do with that, this is another more complex slide.

A

I'm going to talk more about these slides in detail later, if we look at what we can do, this is a symmetric system, so we can have two of these together we can connect two of them together and by connecting two of them together, we've really built a gateway now you'll see here, there's a gateway, described gateway, operates much like the Gateway that was invented by the previous individuals now to to connections between two clusters. That's pretty cool, but really what we needed to get to is a simplicity, so I want to show.

A

First, the simplicity on tape, I've taken all stuff out, I just add the control plane, the application services which have sight cars in the gateways. The gateways connect the applications, the control, plane, controls the side, cars and then, if we take that model and we use symmetry again, we can connect three gateways together. We can continue this process on and on and on and on. We can have more than three gateways, so we can have a hundred gateways. We can have a thousand gateways and, with the same replicated control plane across all the systems.

A

This is how we're reinventing networking, but this wouldn't be very cool if it were insecure right, I mean if you have TCP protocol transferring over the network and there were no security built into it. That would be a huge problem. So, yes, do provide. Excuse me, sto, provides via MPLS security. Mpls is a protocol sort of based upon TLS it's beyond the scope of my talk, so I can't really get into the details of it, but MPLS is used to communicate within the cluster inter cluster and then pass through.

A

Mutual TLS is used to communicate between the gateways, so there we've got security and then magically. We pick up the other two cool features of the ISTE of service mesh or the other two capabilities. This is huge. This is why we want service mesh. This is why we want multi cloud, because we are reinventing networking.

A

Okay, so this is your architecture. I'll talk a little bit about this there's Citadel Citadel, you think the best way to think of Citadel is a certificate management service. It provides certificates to the rest of the service in Sto. Galley communicates configuration information by reading the Kubb API. It reads: service additions and removals and communicates that to pilot. Now you see on this diagram I also have it communicating to the gateways. This does not exist yet, but it will in the future.

A

This communication from galley to the gateways enables us to connect the cloud secure them. Talk more about this in the future. I just want to get the basics down the injector. Whenever you do a coop cuddle apply. What happens is that request hits the coop API that's sent to the injector. The injector says: okay, please inject this stuff on top of the on top of it. This stuff is a sidecar, the Envoy sidecar. It goes back to the coab API and the cuvette api instantiates, the the deployment and that's the model of how it works.

A

Pilot is responsible for calculating the state of the mesh. So you've got all these envoy side cars. Let's say you have 300 services in your mesh envoy or pilot calculates. The Envoy discovery service, the XDS services, which are also beyond the scope of this talk, XTS. The best way to think about that is a discovery protocol for all of the mesh and what pilot does is communicate with all those different side, cars in your mesh and make that happen. It connects that calculates it and connects everything together provides that connectivity.

A

Finally, there's a mixer I'm, not gonna talk about a lot about mixer mixer is a telemetry piece, the gateway the Gateway is just another on boy proxy is specially configured to enable a pass through MPLS. So this is a sto architecture.

A

This is symmetric. This is two clouds connected together. The clouds could be any cloud provider and within reason, their load balancers have to work properly. Sometimes it's a challenge. There's some other limitations. I'll get get to Charlie, so service discovery. What is service discovery? That's a good question.

A

So, first, in my mind, service discovery is two things: it's the ability of an application to discover another service via DNS locally. That's one thing and it's the ability to discover to convert a DNS name to an IP and it's the ability to do that. Conversion look remotely. So if you've got a remote cluster, it needs to be able to discover those remote services. This is what service discovery is so we took. We took a couple different approaches and there's probably more than what I'm going to represent here.

A

First approach we took, you probably saw John come up here. We mostly led this work. John mostly led the work of pilot and Kuby API communicating with each other, and what happened there is what we wanted to do. Is we wanted pilot to be able to read from multiple kubernetes services, so how that ended up working is we'd. Have one pilot we'd store all of the coop API credentials?

A

So all of your cube, config files, we'd store those in secrets in a secret registry in every cluster we'd, replicate that throughout all the clusters, and then we would access the coop API directly to do discovery. This provided endpoint discovery. It didn't necessarily provide service discovery in the way that we needed, but this was a good building block. This was a good key building block when the problems and challenges of this, though is it does require routing on your pod network, which is not really. How can we raise so this is an option.

A

This capability there's been other talks about service mesh is askew the service mesh here and I. Think people have talked about this specifically. So what came along in like 0.7, Avista or 0.8 was idea of a gateway. The idea of a gateway is essentially to write traffic in or out from a mesh and you route it in and out now, 0.8 wasn't quite ready. It was sort of worked, it didn't do. Mutual mutual TLS to my knowledge may be dead and I just didn't understand how to use it.

A

One point: I wasn't quite ready, didn't do discovery 1.1, which is available today for download it does discovery. It does security. Tls we've got passed through mutual TLS with gateways the documentation is there. It requires some difficulty there's some difficulty around configuration so we'll get into that. First of connecting the gateways so connecting the gateways. You have to have a shared route of trust, which means you need to be able to register certificates. You can create your certificates as you like.

A

The root certificate has to be very, very carefully handled, because if you lose one root certificate on a cluster you're, hundreds of clusters have been compromised, so you really have to take care of that root certificate and actually, in 1.2, we're increasing the security there by running Citadel in its own namespace. So people can't access that information at all, so it's completely locked down. So that's that's coming, but the clusters must have a shared root namespace. This is an example. I show here of how to create it using the power ball.

A

These are pre created, search. You can create your own they're, fairly straightforward, to create take about two hours to go through the documentation if you are dedicated, so here's the service entry boy, this thing looks like a mess. The best way to describe a service entry is a way to express a remote service in a cluster, so I want to know where remote services that's what a service entry is there's four parts to a service entry. There's the location, the mesh internal mesh, external mesh internal means as part of the internal mesh mesh external means.

A

It's like a virtual machine. There are metal that sort of thing. The next thing is the addresses, so the addresses are no local addresses I'm talking more about this in future slide. But, needless to say, the no local address needs to be unique per cluster, but it doesn't need to be unique per multi clusters. So you can you've got a lot of address space to work with here. You've got a slash, eight I believe in the one twenty seven network. Of course you have to use some of that for your hosts.

A

Now the cool thing about this host network is this: host local, local networking, local local network is that it does not leave the node. So we use it to map IPS to names for discovery.

A

The address is our gateway, so the gateway is just a load balancer in front of the service. What's called ingress gateway inside abyss do and finally, there's that port 1 5 4 4 3. Now what is that 1 5 4 4 3? That's our pass through mutual TLS, that's how we do mutual TLS between multiple clusters. So this is a challenge because we had to create one of these for every service in an opposing cluster, so Frank Bedouins, but-but-but dune, ski sorry Frank.

A

If you're out there came up with this idea of generating these, so he said well, you know, instead of because we have this information, why don't we just generate him and he wrote a nice blog post about that and he said at the end of the blog post. His conclusion was: we can generate this information on the fly. You don't need to configure this now I'm going to show it demo where I have configured this stuff manually. That's not how I would do it, but that's a way you could do it and.

A

Finally, there's core DNS, so accordion s, cool, really cool tech, really love it. What we do with Core DNS is we override or DNS? We put a stub in there, the stub disk translation from our local IP or from that node local IP to to a database. We keep track of that tracks. The service entries, so the service entries get mapped to no local IPs through that registration process of the service entry, and then we step out to a proxy of core DNS without coordinates, this system wouldn't work, it wouldn't work.

A

So core gayness is essential.

A

You could do this with cube DNS if you're on a really old version of kubernetes, but you still need our core DNS integration point without that it wouldn't work and it's just built into the system. The part part, that's not built into this system is setting up the setting up the proxy, which is what is shown here. This is all documented upstream. By the way, some people are free to read the documentation.

A

Now I'm really going on in here, I'm gonna demo, the hipster shop demo. Hopefully, I don't offend anybody because I'm not hipster, myself, I'm a geek as I mentioned. This is a screenshot from the hipster shop website, which is at the bottom of the screen. There I've reworked this demo to run across three clouds. So that's what that's was the original demo that runs across one cloud: cool thing about the hipster demo. As it's built, it's been integrated into ISTE already, so I don't have to do that work! That's!

A

Why uses easy, so I, reworked it a little bit. I've got something. Some services running on aks I've got some services running on e KS and I have even some services running on my bare metal machine, running metal lb in my home yeah my ear, oh and that's what I'm gonna demo so I'll dim with that now pray to the demo gods. Okay, so I want to show you a couple different things. First, I've got this goat Sh.

A

I type fast, with lots of mistakes. Basically what this does is. It applies the manifests that I've broken apart. I've taken the manifests from the book from from the demo from the hipster shop demo and there's ten or eleven different micro-services I've broken them up in into three and three and five and that's what this first step is I'm applying the services and then I applies it SES. The service entries for the various clouds, for is your DCP in my own system. So that's what this script does.

A

I'm gonna run it we can see like if I gotta get seal to seal. To is my command.

A

If we're going to cluster to it, just copies that keep config I know it's not the best thing in the world, but that's how I do it I'm like if I do a cute control, I get pods, there's no pods I'll get pause, there's no pods on cluster one either kyouko go get pods, there's no pause there either, because I'm, starting with the fresh system I, had installed this two on here I've been solid sto following the documentation.

A

Documentation is upstream so I'm gonna go back to CL to them and do my goat, SH and we'll let it go now. It's deploying the services. Then it will deploy the service entries in theory, and we see it's deploying the service centuries. Now now remember the service entries tell us about remote entries, so we need to deploy them on to clouds for every service. We have we're gonna automate that we're gonna fix that that's going to happen, but this is an example of how you can use multi cluster today and this works.

A

So if I do keep cuddle get service, yes, a system, a butterfly keyboard which Apple just recalled and then, if I, if I get sto ingress gateway, now giving me an IP to use. So that's my external IP, that's the this! That's a load, balancer I'm, just going to pull that up, I can cut and paste here.

A

A

Cool hipster shop. Now, let's go so: let's do some shopping. Remember is running across three clouds: pretty crazy, I'm really into high pi. You can ask my friends I'm, like a super high, fine, nerd I, see this vintage record player. I'm diggin it I'm, taking a look at pretty cool tonearm. Pretty cool cartridge probably needs to be replaced. Well, it's only 65 bucks and it still works. Can't complain about that. I'll. Add that to my cart and then I'm gonna browse for more products. I, don't really like shopping, but hey!

A

You know: let's go shopping, so vintage typewriter, as you can see, I need help with my typing. Maybe that'll help. This typewriter looks good in your living room, but it doesn't say it works. Yeah go ahead, go for it!

A

Now! I'm gonna check out I'm gonna place my order.

A

And the order is complete now this is take my word for it. This is using all eleven micro-services across three different clouds: This Is, Fundamental I've, never seen anybody do this doesn't mean it hasn't happened. I just haven't seen it. So if you have seen it cool, tell me about it later. I want to hear about it because I'd like to work with those sorts of folks. That's the demo now I want to talk about.

A

Thank you now, I want to talk about what we're gonna do next, what's the automation, what are we gonna do is so MCP just let me define that real quick. That's a mesh configuration protocol. Galley is responsible for delivering the mesh configuration protocol to pilot and it's going to deliver it to galley or sorry.

A

Pilot galley is also going to deliver that to the.

A

To the gateways yes gateways, yes, that's where it's going to deliver now. This is one cluster this. This is a problem because again we only have one cluster. How do we? What do we need to do? We? What's our next step? That's what I'm going to answer here, keep in mind this off forward-looking.

A

So the key here is: we need an envoy filter to make this happen. You know this thing Wow this this is. This is a piece of work envoy filter will allow you to rewrite rules inside your local cluster. Without this we couldn't do this work. This is another fundamental feature. Just like core DNS Anhui has a lot of cool features we use in this geo. This is one of them.

A

This allows us to have a galley name and replace it with galley, so we don't have to map galleys along with the gateways, because we only want one piece of information we want to configure and that one piece of information is a load balancer IP and the load. Balancer IP is only unknown to pilot once pilot. Has that information it can rock and roll. So.

A

I'm not gonna, get into details there. You can read the slides later, I uploaded them, but again symmetry at work. We've got the same thing now. Sds I didn't explain that SDS is a secret discovery service that provides security through G RPC, instead of through the typical method of loading secrets through the file system, which is slow and pokey. So we've we're moving away from that we're moving towards SDS.

A

So this this is symmetry but again two nodes.

A

This is a simplified view, so I can take out all of the complexity here and just talk about the stuff that matters galley its communicate with the Gateway over MCP. So we're gonna use a service entry to create a connection between the two galleys.

A

Now, if we do that, we can create a connection between the three galleys now remember. Galley is responsible for the distribution of configuration information that includes the service entries. So if we can do this between three clouds, we can do it between in clouds in theory or a very large number of clouds.

A

And how do we do this? We need what's called it. Then synthesis registry I'm gonna, try to say that three times fast later probably gonna fail. The synthesis synthesis ready, I'm filling already. The synthesis registry is responsible for synthesizing service in trees and envoy filters in the remote cluster in the opposing clusters. So we've got one and of course symmetry we've got two I really want to call this technology symmetric, multi cluster we've got 16 names on the list to call this stuff and it's being bike shadow deadly. That's all I can say about it.

A

Well, it's public, but it's being bike shed too heavily, so here we've got the sense that we've got to since that this yeah registries got two of them and now we've got three, and this is all made possible with envoy filter made possible with the service entry technology. It's part of SEO.

A

Now that's how we get automation now that galley ABC I talked about the need to rewrite the names, that's how we reference them. When we go out the cloud, but when we come in the cloud we want to reference them, miss Galli. We want to do Discovery's galley. So that's why we need that envoy, filter information. Now that we write galley a to galley that way we can reference it and then finally, here's my potential user interface I say potential, because it's actually bash script.

A

I wrote that pretends to be a user interface, but the basic idea is you say? Yes, you cuddle MC join and you specify the gateway address and sto figures it out through the technologies I've described so I.

A

Want to thank everybody for their time. I know your times really valuable. I really appreciate your time. I do have time for questions actually, so that's good I said save five minutes I'm happy to answer your questions. Typically, at this point, I would call up the developers of isseo on stage but I think we'd probably cross the stage, because there's much of us so I won't do that.

A

But if there's any questions I'm happy to answer, if I can there's fourteen workgroup leads I'm only one so there's 14, subject matter: experts I, don't know it all, but I'm happy to try and give it my best.

A

I'm sorry, my hearings, not good.

A

So the question was: do the PA's need to be in the same, addressing namespace and, and the answer is no, but if they, if you, if you have the same exact pod in the same namespace RDS will fail. So this is a problem we need to solve.

A

Rds would fail, so you can't do that, but to your question: yes, you can, that is, that is possible.

A

Absolutely that does work yes, yeah.

B

A

They can that's a really cool feature of this. You can share IP, pods or IP addresses in between you can reuse, IP addresses between the separate clusters.

B

A

Because the Gateway is the only thing that knows about those internal IP s, that's the cool technology behind that's a cool feature of this technology.

B

A

Don't yet I'd like to get there? Yes,.

A

Okay, there's a lot of questions. One of the questions was do does issue. Do bgp peering. The answer is no I'd like to get there. Second question was.

B

A

Ip address, thank you same IP addresses. Are they reusable across clusters and, yes, they are because the gateways isolate the pod networks between different clusters. So they are reusable. Was that? Was it two questions? Yes, okay, can we get a mic where's mine?

A

Yes, okay, I, sorry,.

A

High availability of the gateways- I, don't know not sure I mean you could have multiple gateways, so you could replicate the gateways, but high availability is stronger in this case, but it's unmeasured. So that's where we're at today with.

C

A

Could have multiple gateways, absolutely that would work yeah that would work. I don't have that working, but that could work yeah, yeah sure. Yes,.

D

A

The question is, if you have plans to look at lower levels below l7 and I believe the answer is maybe l6 and l5, but not really I mean really it's about l7 to us. So I think it's all about l7, it's not about l3, no five, six or kind of I kind of love them into l7. That's all the same to me, I.

D

A

Maybe if you, the question is: do you need defense? If you want defense in depth, do you need something to complement is: do you know it's to secure by default? It's built with security built-in and within with that in mind. So if you trust TLS, which we trust all the time, then the answer is you don't really need that, but you could get it with this deal as well. There is that capability so has that capability built up.

E

F

So my question is: I: looked at your demo, so you have, it seems you have created services and trades for different clusters. So my question is to use your demo, for example, if I have two clusters and well have two surveys, the other one has another two surveys, so it means I need to create two service entries in cluster a and it's then commit another. So it's entries in class the beads and make it connection why yeah.

A

That's exactly correct. Yes, okay,.

F

Thank you sure.

A

Any other questions: okay, there's some waving over here, I'm only taking the front I! Guess: I'm! Sorry, if you you know, move up and where you can catch me after everyone was I'm almost at a time. So.

E

For every ingress gateway, sorry, it's your ingress gateway that you.

F

E

Then you have n times n minus 1 connections right. So how do you solve these complexity? Because it's exponential right? So yes.

A

So the exponential complexity- that's a great question, I believe the way we saw that is through what the gentleman asked the question of: how do we do BGP pairing with this? How do we increase the peering model and we're working on this upstream? We are working on this. This is happening in two different ways. This some of this technology is called split horizon. It sort of does this, but not exactly split horizon yeah I know it says, that's what's called I think it's been renamed I, don't know, I can't keep track of the names.

A

There's the names change, no.

E

No yeah just I, wanted to confirm that it's actually I saw a lot of lines so yeah it's pointy, it gets. There are many connections going on and even if you do that from from a co CTL, but then under the hood, you will still need to map all of those right. So yeah.

A

That's that's a great question. We haven't measured the scalability of MCP, we don't know the answer to that question. We're gonna have to measure it to find out how well it scales I think it will scale pretty well just based upon house designs, G RPC protocol, very straightforward, very simple, but we don't know we don't have that. Thank you sure I did yeah.

C

Can you have multiple MPLS connections between two gateways, I.

A

Believe the answer is yes, yeah I believe the answer is yes, and that goes back to the high availability question. I haven't tested it so I'm, not 100% positive, but I believe the answer is yes.

A

More questions, okay, so is it for high availability or is it for service discovery across these different clusters? I mean what's the use case. What's the use case? That's great a great question. So, what's the use case I mean you could use multi cluster for a lot of things, you could separate information, segregate information for governmental reasons. Maybe the government says certain information has to go in this cloud. The certain information has go in this country. That's one use case.

A

There's like a list of 20 use cases I'm, not a marketing person, I'm an engineer: I developed the tech. So you asked me the use cases. I know, there's a whole bunch. I know the marketing. People have a whole bunch of stuff. They want in terms of use cases and we're making it happen. I can't answer all of the use cases, because I don't have them memorized anything else.

A

I didn't just get one. Yes,.

C

So if, for example, you have an on-prem cluster and you want GPUs so the most taking basic use case, that I saw these people expand the multi cluster with Walker node a GPU on on the cloud, so they can use East um like pretty easily to route traffic. There. That's.

A

That's a great use case today.

C

I waited for this feature in 101, nice.

A

Nice you're, a no doctor I, want to thank you as well the adopters we wouldn't be anywhere without him. Here. There's a question over here there any questions in the back that I'm missing.

A

Could the person that raised their hand and do it again.

B

Yeah is there a possibility to have traffic steering so I can decide, which is my preferred gateway to enter exit, this kind of multiple clouds and is there some global, lope lensing feature behind it? So I can say: please answered the traffic where your locally you're running it on three different cloud, maybe on East, Coast, West Coast and in the Asia and so I want. So if the users, maybe more locally from this kind of gateways and.

A

Tasks a question: the answer is yes, you can do that through the control features this Geo, it's not real intuitive how to do that with multi cluster I think we can make it into it at first. We need to get multi class are working really well first for that functionality, but that's just gonna come along for the ride. I'm gonna get that naturally.

A

So I think I'm out of time.

A

One more question: okay, so.

G

In the service entries that you created that allow you to talk to services across clusters, right, mm-hmm, I, assume you're, pointing them. The Gateway addresses there so that gate Reagan resolved to the respective services exactly.

A

That's right so.

G

Are you giving IP addresses of the Gateway or is there a way to resolve them with the DNS name, graphically yeah.

A

The way we do that is we specify that in a service entry, that was one of the four things I talked about in the service entry is that gateway I want to automate that because I think it's difficult, I think a lot of people think it's difficult to get that correct, but you can do that today. You can specify that and get this to work. This uses no advanced features of this, except for one.

G

A

Right, ok, good, ok, folks! Well, thank you. I appreciate your time.