Istio Networking Working Group, 30 Aug 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Istio Networking WG meeting - 2018-08-30

Description

- Discuss Envoy startup/readiness/health checking
- Discuss Secure Egress Traffic Control in Istio
- Discuss self-service ingress to multiple k8s clusters

A

Welcome to this your networking community meeting on this lovely Thursday. Today we have a full agenda, so we have a follow up for East.

A

Your network controller that was discussed I think about two meetings ago, will also talk about envoy, startup readiness, health checking, and there are two more proposals and discussions secure our egress traffic control in East, as well as self surface ingress to multiple kubernetes clusters, so I think we'll do it in this order, and it would be great if well allocate about like 10 to 15 minutes for each topic, but no more we'll try to stay within those limits. So rough island- and you are you online.

A

For the first topic, the pod network, controller, okay,.

B

A

We'll probably changed your order, let me see it, Zack are you there.

A

No exactly didn't join you guys, I.

C

Think the back is on vacation he's.

A

On vacation, but.

C

I can I can summarize that issue if that is like double checked thing that he was asked. Oh yeah, so I think this was an old issue that I mean a PR and an old issue that we had, which was that by default, when we start in the the issues that like in a before what he called pilot, could actually send envoys all the configuration.

C

The health checks continue to fade because there is nothing it's an envoy that actually allows traffic to enter the odd or leave the port and one option that I actually had earlier was that when the system starts like a default, configuration that we can have for every envoy is to allow all inbound traffic into the into the pod, so that all the you know, health checks and other stuff. Let us continue to pass and we don't have two players. Anything and you know so.

C

This would have you do it in such a way that nobody won't let any other outbound traffic, then only for.

B

Security, so MPLS will be completely gone so.

C

Yeah, it would yeah, so that was.

B

C

This is only for people who, like okay, with doing that in other words tightly, because we still haven't found a proper solution for it right, and so we.

B

Actually do have a solution that is I mean the problem used to be is that pilot was generating the config only after the endpoint was added to the service set, because pilot hooda knows that the board implements a service until it was added to the endpoints and I think we fixed it before 1 0 0 surface I see exactly.

D

B

Look at the endpoints that have not ready, so the probably should be gone. I mean we should no longer have the problems. That pilot cannot generate a config before readiness is.

C

Well, I think there was somebody who actually commented on the six days ago, which said that, like you know, they keep getting connection refused from the Postgres containers and yeah. So it's not just readiness, that's an issue here.

E

Yeah, it's you know, services will wake up, they'll, try and make a network connection and then fail and.

D

E

Of crash, or otherwise oh not not be built to retry their outgoing connections, I understand.

B

What they're saying so I mean we cannot have a solution that only works. If you disable security, I mean we want to be secure by default and it to be completely. You know, weird to say it want your service to work to disable security. So we need a different solution for sure.

B

But question is what would be acceptable if, if it's a problem is that the readiness check fails, because you start the application and who is not ready, then we need to consider how we can make sure the application doesn't start until your voice reindeer know. If it is, we.

C

This was a short solution, stopgap solution that I had, which was to see by default on my, which is be a pass-through proxy until pilot comes and overrides all the configuration. But that's not a it's, not a final solution by any means because, as you said, it's gonna be wrecked. Not so human.

D

C

Faltered, I, just wanna, let everything through but I think this was what Zack wanted to bring up in the in these two issues. This wasn't.

D

C

Time back and I yeah.

B

First, and if we don't find anything else, I mean one proposal: I mean I, I mention it in in the past. We have the unit container, so they need container I know is doing almost. Nothing is just setting up IP tables and it's running before the application. We could have so unit container to a bit more who a small request to pilot and download some initial configuration and then when the application starts, and we will have a configuration, maybe it otherwise.

D

C

Start a massive would start ice order.

B

C

That means we appreciate some storage between unit container and the other. That's.

B

Not completely unfeasible I mean we have a volume that is with the secret we have. We have volumes that are shared, I mean it's not really.

A

B

The file system- we're not old but we'll use, occasionally the other benefit for this release. Another p0 that we have for one one which Louie feels very strongly about is which is dynamic, setup of IP tables. Where we're right now we remember we had that workaround for the security program where IP tables is facing the container ports, and people have complained that they need to declare container ports and so forth. Instead of just us magically setting up the supports based on service.

B

With this approach with initializer making a call to pilot it can download the list of inbound works and the initial configuration the same time. So two birds, one stone of output- is called it's not very hard to do. I mean it's just basic, a pilot agent running in in each container. We making a call to pilot getting the config. We have secret. It may even wait for the secrets to be so. It's not. We don't have any startup program because no secrets so.

A

It will build the bootstrap config.

B

As well may build an initial config, you know preset configuration I, don't know, that's an idea and then five out crashes see to still have.

F

B

F

Corporation, it is they're designed are exploring the different options there trade-offs well.

B

There's a bugs were two different officers have been proposed and we discuss it several times, but we didn't get to a design took per se. I mean anyway, that's one alternative, I'm, just throwing so.

A

By the, when you say, dynamic setup of IP table you mean that we get a list of foreign.

B

Pilot and we use that to set up the IP team- yes right and that's our soprano container port. That's that's how how was the context where I initially.

D

B

We can explore it, let's, let's look for other alternatives, maybe, but the fundamental problem is that if application starts before M, who is ready, bad things happen so Zoni solution, I can see is to make sure him who is a different application. Stucks yeah.

A

D

C

And what do you think yeah yeah? That's true yeah attend the unit continuously worth exploring. We can actually see how far we go. I mean it should work for the normal cases, and maybe the key native case might have some extra startup latency, but I guess they have anyway. Gonna cash containers.

G

Should be cannabis for you or during its.

A

Supports orders.

F

And is not only yeah, that's.

H

Semantics of a readiness check around that pod is not part of the service to the readiness check is passing and maybe that's what needs to be or any other parts to to connect to that specific part through one boy right.

A

But then you would, you will need your app to change or like at least your specification for the absolute change to include a check on em boy right. So you would say your app is not ready until MgO is ready. So then you need to add something to the app and we we want to preserve the application binary and also the ml file unchanged, because we're not on premises of this do.

H

I that point I agree with I was just wondering it for any other, any other micro service. To connect to this particular one. The readiness check the passing should be.

B

A

You know what I think, maybe the the injector could add, readiness in like.

B

A

The application you know right.

B

But what we, what what aspect as I mentioned, is that the problem is the application when it starts it may try to connect to a service. I mean the first thing you're doing when you start will connect to the database and pre one or two whatever, and these are national database- fails because we don't have our bound cross. This your application will crash. Alexis is the next panic. I cannot connect to the database because the application when it starts it, expects we have networking and until a voice starts, you don't have network.

B

No, but so what is? Application starts.

E

Is not lost because if your container actually exits, then kubernetes will restart right.

I

J

I

E

Still kind of painful and annoying to have this thing where your containers in this restart loop until yeah.

A

B

A

B

A

To see the number of restarts being kept to zero right, because that indicates a healthy.

H

A

If you see too many restarts there, it's something fishy, I cemetery.

B

Start happening, we are kind of that for how we survive today, because you know yeah, it happens. That latest are concerns they get to stable state. It's not really. A p0 p1 cannot use this job.

A

Ok, so the other solution was like what George proposed to investigate with.

A

G

So y'all, not something an ambition, is already better than fortitude and.

B

For the for the other option, I believe it's already p0 to have the IP tables set up. It's somehow yeah I.

B

Mentioned about hitting a unit container be smarter.

A

I

A

So after a so yeah we have a minute. So does anybody want to add anything?

A

Okay, if not, let me see but volume, are you on line.

K

A

Great so then we can go to the second topic, which is the secure egress traffic control in East yo, and if you would like to share your document, that would be great. Yes,.

L

Can you see my thing yeah.

A

L

So I would like to present the requirements that we collected from several customers. The current solution and the current challenge that we have the requirements are the setting is we have micro services that have to exist external services, for example some legacy, databases or external web services, and we want to control this exist, a control and monitor so the requirements that we collected I think they are pretty simple, genera straight forward. Nothing fancy here, the requirements are the protocol to control is TLS is an eye. I guess this is the most popular protocol.

L

Every all the traffic is encrypted to live demo services. The second requirement is that it has to be transparent to the application, so the applications to continue sending requests to the external services without special configuration and I think this is a killer feature of Vista that you can do it. You can provide this transparent monitoring and control by injecting mic rejecting cycle process. Sorry, the third requirement is for to define policies and to enforce the policies. So there are two kinds of policies.

L

The first one is error, the whole mesh, so any micro series may exist. Starwood foo.com, a wildcard, the domain or a policy can be cursed. Hostname, for example, might be to the load bar dot home additional kind of policies if policies by source. So, for example, the user can specify microservice a makes s star dot, foo.com and microservice teammate set some specific post.

L

These are two examples of policies in requirement to provide the monitoring of with nine of all the access to external services, and the last requirement is to prevent tampering with the policies. Okay. So the assumption is that some of the application pods can be compromised and despite despite that, the policy enforcement still have to function correctly. So even if we have some compromised folder, we have to prevent the attackers from escape monitoring and pick which of the requirements.

B

The last one, obviously it's not something that we can support. We teach the way it is today.

A

B

Means that it's.

B

A valid thing to wish for okay, so.

L

Today we can provide this initiative and let me show how okay, okay.

M

L

So, first of all, I guess we'll all be with the question that side approaches cannot be trusted. So there are multiple ways for an application to bypass it. Cycler pops it one of them to get root access in the container, not in the port, not in no mood in the container only and then run is as a user with a variety of envoy and then the traffic there.

B

Are many others lizard yeah? That's what it's a long list of thing weighs an application can bypass proxies, always not given yeah.

L

Right so here we work with the assumption that only the eesti of components, the control plane and the gateways can be trusted, and this assumption has to be made valid by Easter provider or cloud provider.

L

They have to deploy Easter components and the gateways to dedicated nodes, the normal application, any applications in the application containers the nodes that have some access to these dedicated namespaces and to apply some strict security measures for these components. So this is the assumption, so you can take application pods. You cannot take the easier components, so it should. It has to be provided by cloud providers outside of Wistar.

B

L

So story, so do you mean this assumption cannot be provided by a cloud providers? You know it's not necessary. I mean I, don't think it's necessary to have absolute code. Okay, okay, an additional assumption is that one once compromised the attacker can avoid the sidecar proxy, as I've said, avoid the sidecar box interception and then the attacker can just you know. Lauren Eastern can read the code and it can run its own frequently on autopsy.

L

You can get the certificates of the pod and it can start sending fake mixed reports and the attacker will try to break with our policies and our goal is to still to provide secure in this control under assumption that some of the application pods are compromised. So let me describe how we can do it in my opinion, okay, so the first idea is to enforce control input and provide monitoring in some trusted component in this case in the egress gateway. So we direct all the ingress traffic through this gateway.

L

Okay, so any state is achieved by pilots and accounting tables to own voice. So we have sidecar own voice and we have the Envoy. At&Amp;T goes deeply and again, as I said, we have hardened security on the Ifrit Italy. This is the one step two. Is this ingress gateway component that we trust will send reports? Okay, we trust they will move. The fake will send reports to telemetry will send ship messages to policy. Okay, mixer and their monitoring will build ister formed and policies or seem to be default in the Eagles deeply.

L

This is step three.

N

L

So this is step 3 okay, so we have to prevent bypassing the Gateway, for example, by defining Network policies that all the Innes traffic that's the only interest. Rhotic that is allowed is the traffic that originates at theatres gateway. So no other traffic will be will be a lot, so we can use for that Network policies. We can use firewall, we we can, for example, tailor and allocation of public a piece to the internal nodes, and we can configure native address translation devices, okay and not to perform and negative.

G

Edges, translation, and so so so starting Arak I see what you're trying to do. I think I understand it. What I don't understand is why is this important for you.

B

For each key, oh yeah.

G

B

Mean some customs may have these requirements and they can probably do it already with firewalls and other things, I mean. Why should East your odd complexity and and and to support this you see there is it's that important for all customers. Yes,.

G

You describe the use cases, they'll be great.

L

Yes, this is the Nueske's. Actually, these are the acquirements, so the use case is pretty simple: we want to control access to the external services and we want to do it transparently, but.

G

I can imagine just saying that the external services should be responsible for their own security. I mean, in other words, worried more about security within the mesh and entering the mesh, and not worry so much about leaving the mesh. That's somebody else's problem and.

B

Also I mean yeah everything you see now. You know you score. You have a route, I mean. If you want to prevent outbound trap, you could put a firewall rule its resort program. You can say that nothing's from inside the mesh is allowed to route outside by simple firewall ruling in in the networking and then.

G

Yes, so but I mean I'm sure, there's probably valid use case behind this I'm. Just asking you to explain: why is it important to you or your company or your you know, okay,.

L

So the idea is that, once the attacker, the attackers will be able to code to compromise some of the application puts they try to, for example, perform a detector.

B

They're personally performed by people who don't exist, you mean it's, it's basically you're saying that hey is, you will not be able to do but stories I mean if.

G

You have a bunch of untrusted tenants within your service. Mesh I mean yes, this would be.

B

A

So I think I think like if I, if I understand correctly, madam, is proposing to always deploy the egress gateway, always.

B

A

Hear exactly how we can prevent those like.

L

Okay, so, first of all, let me introduce two more use cases: okay,.

A

L

Additional use case is that the attacker would try to just access. You know to steal data from the external services. Maybe the attacker will try to steal beta from inside the mesh and pass the data to to malicious sites okay to to their sites. Let's say the stolen data, the malware of the attackers, maybe we'll try to download updates for itself. So there are multiple I guess, use cases, okay, that you want to prevent uncontrolled unmonitored access from your cluster to the outside world and the additional use case in what additional good use case.

L

Sometimes you allow for user code to run in your cluster and you also control your user code makes s. Do.

G

You have this use case. Are you interested this.

L

Use case I have this use case and actually the connection I have a customer and for them this is a killer argument in favor of Easter. They want Easter for this in this control. Actually they want to deploy exactly for that, but maybe better documentation.

B

To explain that issue is not really solving this problem, because but.

G

This is proposal.

B

G

So far, I don't see any.

B

Still here, Manassas, some in your proposal is basically use IP, firewalls and up firewalls that exist and solve this problem, and there is no Easter required instead and mesh, because the micro services do not make easier for this to work. Yeah.

K

N

L

Sorry so actually the solution is to use the whiskey tree and to use.

D

B

L

Insert policy, that's all it's not totally possible, I mean if you put firewall rules there.

B

You can ask you to is doing exactly that. I am not sure what is different from what we can do today with egress gateway. You.

M

Know it seems to me as well that this is really just saying here's a pattern of using this. Do.

D

That you should.

M

Should follow and kind of document that showing that what what exactly is the hole that needs a change in this view, I guess is what I'm not saying here: okay,.

L

So good question: so when we present two challenges: okay, one is solved and one we have to solve: okay, okay, so this is the whole picture we prevent. You know we blocking any excess. That is not true. It could eat any. It was access. Okay, the third killing is to handle wild card domains. So suppose we have a policy that allows access to start good food on corn and they request sent to MongoDB one dot food code. So how can they give way forward it?

L

Because we have this limitation of envoy and only we can forward the quests only to predefined hosts and they are not known and in the case of the wild card to me, okay, the.

E

Wild card is in in the policy right the right one.

L

E

Get the full domain in the S&I alright.

L

So you get the yes, you get the hostname in this and I, but all we cannot send cannot forward the the request, bias and I. This is.

B

A problem: why not? Oh because because he doesn't have a you, cannot use a semi for of her, but that's exactly what 0 VPN is trying to address basic. You know: okay,.

I

Maybe maybe he's saying that there's no upstream cluster, that is called.

D

L

I

L

I

That's why I can't yes.

L

Although it has to know all the to come on, boy has.

L

B

We also demand cluster creation through the secure.

G

Additional test for s: no, no.

B

No it's somewhere where and it doesn't know it gets it out whatever destination.

D

B

He doesn't have a cluster already and then, and we lost by oh hey. Please give me notes references, it's it's. Definitely one of the top four requests for scalability and and I mean it's. It's clearly a good requirement for many other cases. So, okay,.

L

Here how we propose to solve it to deploy additional a sniper, ops II on the local host on the in there in the pod of of the Gateway? So this is not proposal, no for the standard, it Lewis deployment or a cart. Any easier provider can deploy with their own. It was it way, and in this case the provider can deploy this as my proxy. We, you know any Sun, I proxy. We tried inch next and it works okay. Is it the same as I? All the way through is there's no mutating a.

G

So say it again: sorry, is it the same s ni all the way through every hop or destroy, trying to.

B

Confuse here, I mean is the previous part, but why do you need a sidecar for first application understand the egress would need to take us anion and and resolve it, but and therefore it create a cluster for it. But why do you need a sidecar now ah the.

L

Sidecar is for transparent, directing routing of the traffic meeting is gateway, but the baculum know that it sends a request through digriz gateway. It just sends a request to the external service and the tables.

B

That means that the net or police I mean you are routing everything to egress gateway to the.

L

No Bravo, no by the sidecar envoy by ID.

E

Network policy doesn't doesn't give routing it just does and of allow deny. Yes, you still.

M

E

Up on boy, in order to redirect.

B

Right start to I mean you put the IP table. You know that either got your out start to is a gator I mean it's it standard the many time in the middle works order. You know how oh sure.

E

I mean you can do this in uniting tables, but, like you can do a lot of stuff that that envoy does for you in in any tables. You can do a balancing, an IP tip. What concerns.

I

Me more is this.

E

I

Go on constant, the.

B

Whole premise was the application can be compromised and- and you know, if application is compromised- you just not make me use the cycle just by the side. Oh.

E

Yeah the redirection.

C

E

Is in the mainline right like for in in when it's working correctly, you want the power consuming.

C

And then you just shut everything down if.

E

It's not it's not redirected, then it just gets denied and never actually hits the network. But you still.

D

E

For when it's working correctly behaving itself again,.

L

What mister is a point so since everything is blocked, every trial I see what you're saying again his book.

L

Has no other choice but but to direct the traffic should being the decrease. The attack phase now somehow to cheat leave is easily because no other choice. So if you don't have the.

B

Sidecar or don't have city code to direct to Gateway, you will be blocked. I'd, like most enterprises require socks or some proxy tool to get out.

B

It will be provided by a sidecar instead of putting the export setting the socks proxy or whatever it.

L

Is they click bingo? So this is the transparent required? Okay. So now this is the challenge. The first children challenge is wildcard domains. Here it is solved by additional pop see. Now we have a different challenge. We want to enable policies by source, ok, so for that we have to use MPLS twister, ok, to hit the sauce principle. So now we have this Angeles tunnel. We have the the original que les with the original is my the English night, and we have this outer snipe, ok of the MPLS tunnel.

L

Now the problem that we have that the envoy of the of the Gateway sees the the outer is night of the tunnel. It doesn't see the lesson I units an I. So what happens? Is this envoy import to the mixer? The outer is my, but the routing is performed by the Englishman, so we have a security hole here. Okay, so mixer receives the out reasonable that while the traffic is sent according to the inner smile- and we want to report to the mixer, the actual hostname, the traffic will be sent.

L

Ok, so we have now to extract the inner smile. This is the only part that is missing currently in easter to support this news case and we considered two solutions.

L

First to it, what we call as an ID the filter to envoy- and this is my leader filter- will extract the inner asana and will send to the mixer the in resin. I so.

B

Can we have cannot? We cannot avoid the double encryption, sound yeah? Yes, it'd be nice. If you could are.

F

You sure that so.

B

Many designs that can work and then.

L

Yes, but because the stronger identity, the identity of the source in Easter is provided by in caelis.

L

B

L

You get this hosting support, but.

B

You are saying that you are going to do one and TLS 21, so let me rephrase Kenzie's be implemented in a way that doesn't add a lot of value in a plug-in or some ways that doesn't create a lot of complexity and easier because understand. This is a wonderful, your skills but latency problems. We already have scalability problems. We have a lot of other issues that we need to resolve before he can take this kind of sounds very complicated to me. I don't know yeah.

L

But the whole issue is pretty simple. Okay, you need this small filter to extract the inner s 9 to side by your double encrypting yeah. That is, this is done in Easter. Today can be done today. No.

G

No but its power, everything as we like to avoid the double encryption and that's making more complicated. So what.

B

Exactly do you gain by this um so application. So let's say if there is no cycle invoice for the application, because application could you know be aware that there is? There is a egress gateway, mm-hmm socks for example, or whatever we are doing today and just make the direct connection with an outer SNI and I mean who do it, for example, just implement the socks protocol and and which, which has authentication or some some of the other. You know it should be connect, for example, h-2b connect.

B

You can authenticate the cooler with tokens or whatever and- and you can still proxy requests.

O

Don't have HTTP protocol right database- oh my god, opaque TCP protocols.

B

Is a big is actually proxy. It's doing, connect.

O

Let's assuming the client speaking HTTP, but.

C

That's the egress gateway on why speaking issue TPN can receive any arbitrary traffic and then send it to anything enters plain. Tcp, yeah.

C

E

Application is initiating TLS.

C

This is what we're using in on trying to do. Nan way for tunneling well be using the HTTP to connect external to connect in order to turn allah a bit rate traffic through through envoy, like you know, where's my sake well, WebSocket and so on between a cross envoys and the part of the proposal there.

C

That is, that cost in the saying is that, if using HTTP connect, and then we effectively going to be talking from citecar to on my over h2, which will bear, we carry all the metadata and all the headers and so on and so forth, with traffic comes in our way, unwraps it and then sends that binary payload to wherever it wants to go to.

B

So, in other words, gateway implements is to connect.

B

Proxying arbitrary TCP UDP token of other craft of the page to know if we just implements the standard IETF.

B

We solve the same problem, I mean yes,.

C

And we already way through doing that. Actually Alyssa from the Washington she's already started doing that work, and we do have a rough way of doing that for WebSockets over h2, but there's still some more work. That's required an NG HTTP to, after which we should be able to do Donald arbitrary stuff over H yeah.

A

So can can we maybe have a review for this design right because we don't want to dismiss like.

G

A

Without you know so,.

G

I think they were we're honoring the use case yeah.

A

G

A problem we solved I think the only constraint is the double encryption. Is that right? It.

B

Does make is the complexity, and you know some things that some things that nobody else is doing and it's it's not clear how complex it will be and how any kind of ability maybe quit with what with standard based solutions you may not. Yes, you may have applications and just use other proto right.

A

And we also, apparently, we have another like proposed solution for that. That's what we're I'm, not stopping.

A

B

A

The same thing, alright.

B

It's you know, there are other people on here producing. It is not.

A

Running out of time here and I want to give a chance.

B

If Ania me, no I think it's super important requirement, and and and we should prioritize to get that- that part may not demand classroom. It's the first part and I think that that we have no other choice but to implement this kind of on demand clusters. If the second part, you may need to discuss a bit more than so.

A

There is also a document right there at the slide, but.

B

A

Is also a document that Vadim sent so I think it's probably fair.

D

To discuss you all a bit.

A

Through the document and their comments- and we can have a follow up on that- but this is ID- let.

L

Me just summarize, so the only missing piece everything is implemented picture. Everything can be done today with Easter. It is a small part of reporting, the inner SMI, and we have these two solutions and I would like you to read the solutions in the document and provide your comments.

A

Ok, what if the application is.

B

Not doing SMI what the application is, starting that TCP I mean just like. We have this problem already of not getting visibility into the application if the application is starting already in TLS connection or TLS connection, assuming we solve that problem by direct integration or some other mechanism, if the sidecar would get clear text, would it avoid a double encryption, a.

L

I, don't think so so in as far as I understand, in order to verify the identity of the source, you have to use the certificates of the source, the ones that are rotating by Citadel, ok. So this is how you verify the source actually and also the source by its bounds.

L

G

Of the ways I mean, is there other ways to that's.

L

G

So the requirements we identify the source, but we could potentially.

L

A

C

Mechanism that doesn't require.

G

Double encryption, yeah.

C

L

Reporting both the inner sonic.

L

A

Ask are very short, question: I hope. What is that TCP proxy then that's running? Is that another like container running in ICP.

B

Proxies part of employers.

A

Ok, so it's not some other.

B

C

This is it. This is actually nginx. There's an IP CP proxy that they have eight four four trees, nginx right, the right one. So.

A

Basically, that also require introducing another.

A

A

G

So so one way we could do this is we could do some small piece of code just to get him unstuck and then do it the right way later. The ZZZ.

B

Piece of code is what we need for 0 VPN service and the other. One is what we need for Stella, because those.

D

B

Pieces are definitely on the many other lists. Alright,.

A

So I'll have to like go to the next topic. I'm sorry like to interrupt abruptly. Thank you, buddy and let's follow up on this offline and maybe with another meeting and I see we can recommend it. But okay, let's go offline because I saw Gabe is here and.

D

A

I'm not sure I found relies here also, but they would like to discuss of service ingress to multiple kubernetes clusters. Okay,.

B

D

Hi I'm here I'll.

A

D

D

Great so I'll just start with a brief overview of the use case, we're looking at and sort of why we're presenting it in this meeting. So basically, we have a proposal for a self-service TCP in restaurant to multiple kubernetes cluster, so we're particularly interested in the use case of external TCP connections, trying to reach an application on a kubernetes cluster when an application developer creates their creates their deployed application, they wanted to be able to be externally routable. We are also considering the use case of multiple clusters that are coming up.

D

So while there is a way that, on a per cluster basis, you can configure a load balancer and have your services externally routable. When you deploy many services on many different clusters, it becomes painful to manage external ratability. So we're looking to solve self-service ingress routing for multiple kubernetes clusters using.

C

D

Building on top of this do so, basically, we don't believe that this requires any changes to sto as exists today, but we wanted to see if this use case resonated with anybody if they've seen anything similar and had any general suggestions. So as you can see, no chord changes so we'll go down basically to the data plane design. So the idea is in order to have external route ability.

D

We wanted to deploy an additional routing cluster, it's what we call it and you would, as an operator, only have to configure a load balancer to forward to your routing cluster and then, of course, you would expose a node port to forward to the sto proxy. This via proxy would know where exactly the service is living and would be able to forward correctly to the node port. That's associated with that service in kubernetes.

B

Is this the Gateway that we have today I mean.

D

This would be the ingress gateway, ok,.

B

D

And so basically, how? Where and so all of this, the node port, the Gateway, config and and the correct forwarding would all be dynamically generated, based on seeing a specific annotation on the node port he's.

B

Not worried quiet because many many environments don't use noteworthy used or bonuses. Yes,.

D

So we believe that we can also support the load balancer case I, think for just like an initial POC, we're looking at requiring a user to create a node port service, but you could believe like it could be extended to also be certain for be dynamically configured for services of load balancer as well. Yeah.

O

I would say the use.

D

O

We're envisioning is on infrastructure that doesn't necessarily have automatic load. Balancer load, balancer provisioning from the cloud provider does.

B

Not work in this case I mean you get IP addresses for each node and you're out them to DNS round home you don't. How do you get.

O

Yes, we're assuming that there is route ability between the nodes, but not necessarily between the pods on the clusters.

B

O

Not using DNS here.

B

D

So the idea is you've configured your load, balancer or you've configured a DNS to for to your load, balancer for some like, let's say, hostname and port right and the load balancer will do port forwarding to your routing cluster.

A

All the other clusters.

G

And is the routing cluster doing like l3 or l4 or else Evan routing? Do you Fe look.

D

G

Http headers or anything like that to do.

D

Specifically, looking at the use case of like for.

G

D

And not just HTTP what.

G

About using an l3 load, balancer, and so those are out in cluster I mean something like katrana or something like that. Providers.

B

Have balancers so.

O

Just would require having a virtual IP for each back-end service. Is that right.

B

One of multiple IP addresses that's.

G

O

G

B

A

Require changes for what you said right, so it is a gunner from the category of like best practices on how to do this kind of deployments and, after so specific use cases. I think.

C

They're looking for feedback in terms of our lives like right- and this is I- mean a better way to visualize. The context is, if you imagine, cloud, foundry and kubernetes and like this is one example where a typical example is people in the cloud foundry. They would like to allocate a kubernetes cluster just to run stateful services, like my sequel or like some of these area. These days.

G

Always give me more expensive than doing it lvl 4. So why pay the cost to do this at l7 here if.

G

The Gateway so the.

B

G

That okay cut it so.

A

Here so the use cases where they have like the kubernetes cluster hundred cluster and they need.

B

To discuss in the past, we.

O

Haven't really gone.

B

O

Core part of this proposal.

A

B

Is that we run it or not? We cannot so when you see not what is kind of fine yeah.

D

No worries, so maybe we can just take a brief look at the changes that we're proposing to get that data plane into existence or what we're building on top of this do because again, we don't believe any changes are necessary.

D

So the basic idea is because you have multiple kubernetes clusters running each of your key Rene's service needs to register with the cluster registry, we're proposing adding sort of a new extension, a new controller on top called the route manager, that's watching the cluster registry and we and then can see all the services being created on all the clusters and based on that, it is able to see if a user creates a service that wants the external ear out a pole that we are creating an SEO config in the form of.

D

Of course, you know the Gateway virtual service.

J

D

And then also we're creating a new CRD of for external for mapping external port state. So the idea is that each service that you want each workload that you want to be externally routable will get its own unique port.

A

Question here, so is this the kubernetes cluster registry we're talking about because.

B

A

The services are available there.

D

Information from the cluster and then based, and that gives you access to watch the cube API server for each of the clusters.

B

Doesn't give you access to the clusters that the column I mean with the cursor I? Don't know if you, if you saw the designs that we have today and the implementation of for multicast initiative today, is a problem across the registry that it gives you an lists of registries and IP addresses. But you don't have any credentials to connect to the API servers and watch them like.

A

The secret signal.

B

Conditioning so what we do for music and I doing this proposal to kind of composite into a different direction from what we have today with multi cluster, because again.

G

B

Enough trouble understanding is echoing multicast that what.

G

Are you guys, you know it's not like closer to each other.

O

In fact, one of the things that came up the last time we talk about multi cluster was that external ports needed to be allocated manually in order to proxy through arbitrary TCP.

B

Different two different things here that what what Pilate is doing today is watching for some secrets and whenever the secrets are created, the secrets contain a cube config and they get. Those coupons. Digger creates a connection to the remote API server and then is watching the remote API server for services, endpoints and all other stuff.

D

Secret is that secret currently like manually created, yes.

B

The user, once a user adds instant according to cluster Registry, the user is studying a secret to to the you know, domain II case ever I say.

B

You have one pass that when you run pilot is running, you have a list of classes or footings and in class that registry you create this cube config and put it in series. Okay, so you communicate.

D

Directly with the cube API server of the cluster, that pilot is running on. That's.

B

What we are doing and the problem again in your design, is that you have no way to do it for the route manager to connect to the cube ApS to kubernetes api, because you don't have credentials. Cluster registered does not store credentials. So it's yours, yeah.

D

B

We know that you.

D

Would have to have a secret also generated in yes.

B

And we plan to use was the registry for multi cluster ambassador, just it will be in it at some point.

B

D

There's there would be another box here right, because the coop API is like the kubernetes api is to register with the cluster registry. We're envisioning. The cluster registry would run on the same cluster as the route manager. So, in addition to when I, add zit self on to the cluster registry, it could.

B

Also, make reading easier right.

D

B

I can sneak.

D

B

Fact here what do we stop here as a route manager which is watching the services is already implemented in Python? So pilot already is watching all the clusters for services and workloads, and it.

A

Doesn't it doesn't integrate that class? The register so I see.

J

It doesn't, it doesn't, implement, I mean it doesn't integrate with cluster registry, because the cluster registry was insufficient.

A

Actually so it has the same problem was the current multicast their solution with regards to getting perfect financials right, but other than that, the design the way it says it's.

B

What we have if we use the same, if we don't really implemented infants the wheels that we already invented. So besides the terms I mean is the second part and is the part that I think is the core of this proposal to have this creation of each tier, configs and external words. That's already something we are trying to do for for the 0dp and for the split ideas that we discussed last time.

B

O

Thing there was for arbitrary TCP services, port allocation on the outside had to be done manually by an operator, and that's what we're trying to address here with this external port CRT. That.

B

Means that's what it is. We said that initially will create manually, but eventually we want to do exactly the same thing. So if you happen to do it, wonderful I mean it's. It fits perfectly with.

O

We can scroll down and show that CRT now.

A

Integration with the.

B

Computable discussion first registry I think it's like not what is that heading yeah.

J

I don't see that is how we interval here, I mean if the cluster registry was killed. Also, we could adopt it more unilaterally, but let's I think it's dealing.

J

B

Be specific about boundary.

D

B

D

The use case that we're looking at is providing like right now and this proposal is providing external routing for kubernetes.

D

So we can see in the external port CRD. The information we're storing is the port that is going to be allocated to be listened on on the gateway and then the hostname that's mapped to on the virtual service to the service entry. And then we just want a unique cluster identifier as well, but.

B

This will be something that this needs to be internal expertise. Itself is not something that is implemented by history, something that you know: yeah Chloe's, kubernetes provider, I mean jakie years and so forth. We need to plan. Why not modify the existing service, because Easter is creating the service for the Gateway with a list of ports that are forwarded. You know the easterly egress service, which is type load balancer and entire internally has a list of 40, 80, 443 and a bunch of other ports.

B

That resource can be edited programmatically by pilot or whoever and when you need any existing kubernetes implementation will modify the role banister, because that's the specification.

O

So we have infrastructures where automatic load, balancing provisioning isn't really possible and, furthermore, we're provisioning, a separate, VIP or load balancer for each service is cost prohibitive. So what we're trying to achieve here is having a range of ports on a single external IP.

B

Maybe I was confusing so right now, East yo creates a single IP address, so we can locate the single IP address at a single load balancer, but the load balancer has a list of ports. What we are discussing here is anything or adding ports to the load balancer. So you have a service type load balancer and is inside you have a list of ports, and that was the most kubernetes implementations that we know.

A

C

Port range in order list of ports, they want like something from 30,000 30,000, which you can't do with an akuna service right.

B

You arrange what you can do one by one, so you can programmatically, add ports, I, know it's a menace to underage and and I think it's been open for two or three years. You cannot fold it for some cloud. If a particular cloud provider has this facility injury this resource, but we need a solution that also works on. You know: jke, bluemix and oversee other providers that don't have this external port facility and.

O

Proposals designed to be correct, gnostic, yeah.

B

Well, implement this proposal in you, you will need all all kubernetes provide that implements this external, pour specification well,.

D

This would be a CREP right that you could apply.

B

O

So I think you're misunderstanding. Something is the route manager, it's the one that is creating and responding to these CR DS right the right. This is simply a datastore mechanism for the route manager track of which external ports have already been allocated and which ones are still available for its allocated. We're not expecting the cloud provider to do anything with this external or.

B

Because we are getting back to not port, so you have the node port. Getting the traffic and use the route. Pager is a node work to to manage the I understand. Oh yeah,.

O

B

Your design is fundamentally relying on on north port and is not you working with load balancers. So it's not going to work. If you, if you have a service using load balancer. Oh you! If you choosing output, it's.

O

Assuming that there's connectivity from the routing cluster to the nodes of the users cluster, it's.

A

A bit different than the note or itself Burnett is I think he wants.

B

A

To the actual note, where the pods are.

B

A

Mean yeah, you can get either way, no turret, it's more like about exposing that.

B

Provider, like you, use that load balancer, so in JJ, for example, this will probably not work and I don't know for another. It may work on providers. Wear them not not. Ports are extremely.

A

If you use cluster type load balancer, in addition to that note for being open, you get an external IP from a pool right and you basically get off like a file world.

H

A

Allows your traffic to get to that particular node again, so you wouldn't.

D

Want to be hung up on like node per versus load, balancer I, think that you well maybe are just ending up, but I believe like like the main thing is that the sto, like the SEO ingress gateway, the proxy in the routing cluster, is able to somehow communicate with the worker that the service is running on right. So, yes,.

J

D

Looking at the node port, if you had a load balancer as long as again, this routing cluster can forward, should work right.

B

Commented today, if you create a gateway resource with with a particular world, the proxies a gateway proxy will automatically open support. So we don't probably need the second resource because it's already implemented so the moment, if you infuse load, balancer type, node port. The moment you create the gateways, the proxy will release an on-set port.

A

What do you mean, load, balancer type service type,.

B

Service I have not participant in the current configuration. If you use not port, that's already happening today,.

D

The Gateway is already being configured yes.

B

Again, gateways automatically opening support, so what exactly is is like sorry source doing in this case. Ok,.

D

I think in that case, if the Gateway is already creating a type, a service of type node port thing, you wouldn't need to generate one but again, I think that's a small teacher.

B

Install options but I.

A

I think we we shifted away from the initial goal of routing across multi clusters, so I'm not sure how you would know which no port is where.

B

A

Exactly so I think you still need the roti manager and in.

H

This case I guess the Gateway is in the routing cluster, where the virtual services are in per user cost, or is that a good way to think about this.

O

Yeah I think that's probably good way of summarizing it yeah. Thank you. I realize we're over time.

N

O

Can just leave some comments on it: okay,.

A

And like again, if you need to schedule a follow-up, please do so like at the agenda and we can know maybe.

B

The environments, because we woody clustered all this stuff, it's we we tend to discuss it there as well: okay, okay,.

O

Thanks Peter thank.

A

You very much underlying it and thanks to everybody for joining the meeting and participating and sorry for staying over time again. Okay, fine, fine, I think.