Istio Networking Working Group, 2 Aug 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Istio Networking WG meeting - 2018-08-02

Description

Agenda:
- Istio 1.0 is out!
- Discussing VirtualService Delegation RFE proposal
- Istio-pod-network-controller
- Scaling endpoints

A

All right, I think we can get started, hi everybody and welcome to this wonderful community meeting where we have a big item to celebrate, and this is the release of Easter 1-0 yeah. Yes, so it's time to cheer be happy and Frank I want to thank everybody for their contribution, because this is was. This was an amazing like community important collaboration and contributions from everybody in this working group and all the other working groups. So this is really amazing. Ok, so I hope everybody had some champagne or something to celebrate with it.

A

Champagne and beer and other things.

A

Please add your name to the attendees list. If you haven't done so and I guess we'll jump straight into the second item now with the agenda, which is the discussing virtual, serving service delegation proposal and that's near as near as are you in the meeting? Yes,.

B

A

Great, so can you please share the document or you would like me to share it. You.

B

Can share it. That's fine.

A

I think it's actually, if you share it and you want to scroll all so, it will be easier, maybe to us so.

B

I guess I need to be represented, then.

A

Yeah I think it's so from the meeting there is. There must be a button to present or something.

A

C

A

D

Yes, I think so: ok.

A

Good yeah and we'll have more items on the agenda, so we'll probably cover this for about 20 minutes. I. Think that's fair! Let me know if you need more time, then I will discuss the upcoming big work items and one of them is the scaling of the end points and costing will cover this and I see somebody else at the East, your quad network controller, so I think if we do 20 minutes 20- and maybe we have minutes in the end left for you for the last item- I think we'll see how it goes all.

B

Right I'm still having a problem sharing.

A

Sure it's actually here all.

B

Right all right: okay can I, should I begin yeah.

A

B

I'm neeraj I work for Aspen mesh I had initially sent out a question regarding how can one virtual service refer to other virtual services rather than just to destination clusters, as the current spec is so that we can have more composable resources which can allow different virtual services to be chained together, and it can give ownership semantics to various application teams which are developing their applications?

B

They can write routes for those services and other services can other virtual services can refer to them when they want that routing functionality, rather than incorporating all the routes in a single document. So I think after sending the dog I'd received after sending the email, I received some feedback from Louis and Shriram and I thought it might be worth writing an RFA and getting that discussion going again since when zero is out, so this is the dot which explains the various goals at the top level, if you scroll a little further down under.

B

So these are the challenges that I see in the current document. Basically, there are three challenges. The virtual service resource can continuously keep on growing. If you have a large number of nested sub routes for each sub path matching and additionally, if you have a lot of matches on the on a regex or on a wild card host again, your virtual service resource can become enormous and as it grows, the operational complexity is huge. So this talk, this RFP aims to solve that problem.

B

Additionally, there is a very weak ownership semantics in the current virtual service, spec I'm, hoping this design solves that issue too, and the last thing is in kubernetes. If you have name space based, ownerships or namespace based are back policies, at least in my mind. There is no clear way to have a similar behavior in coop in the current sto, what your service pack yeah alright. So with that I mean I'm. The document basically goes through a simple example to illustrate what the problem is.

B

The example is in this, where you have two virtual services for vs and bar vs. Phobias is for the food service in team, one namespace it's currently, because the virtual service route destination has to be a cluster. It has to refer to the host bar team to directly as as you're doing this, you are automatically skipping over all the routes which are defined in the bar vs, so in the current document in the current resource. One way to do that will be to merge all of them together, as I say in the table below.

B

If you can scroll down. Okay.

E

Explain a bit what yeah sure.

B

Yes, so if you want to incorporate all the route rules for bar vs virtual service in the foo, which will a foo vs virtual service, you would have to merge them manually. So this is the merge route where you give a less abstract.

F

Example of when this problem shows up Mme sorry go on okay, can you give a less abstract example like without foods and bars and bowser's like when? When does this problem show up, and and what are you trying to do this.

B

Problem shows up when you have one virtual service, which has a bunch of matches, wants to route to another destination. But there are there's another set of virtual service resources which have been defined for those destinations. But.

E

G

Roxbury is that you have you, have a single hostname and within a subsection of that hostnames namespace like a path, the ownership of all these services- and there may be many that mapped into that portion of the namespace are owned by one distinct organization than another sure. Right so, let's say your Google comrat Google has Gmail and you have I, don't know, play yes right and so google.com slash, Gmail is owned by a massive organization in google.com.

G

Slash play is owned by a massive organization and they don't really need to know about each other's can fake. Yes,.

A

It's the case of a model this, because you said you have two different services: do they point to the same destination service.

G

Is one virtual host, but two different namespaces of.

H

The problem is the virtual services, let you manage hosts, but many organizations manage pads under a host. Yes,.

B

E

Discussion of ingress as well and- and there was a very big debate about how far should we go with merging stuff, because this example, when we take header matching everything else, becomes very, very complicated, beau to implement and to understand by the user. If the use case is not always, you know, basically, you have a prefix I mean you have the domain name space and you want to also want to add the prefix to say that Gmail goes to some other domain and so forth.

E

That's perfectly reasonable and easy to understand, and you know the use case is clear, but when we go into, if it has a header, foo, okay, who owns a head therefore mean how do you know that that you need to merge or or or because.

I

It's it's right, so I mean.

G

This is one of the intrinsic problems with HTTP right I mean you can end up on these word permutation cases, because well, the URI structure is strictly hierarchically. Space people do tend to route on things that are outside of that, for instance, you say maybe, like I, have an experiment, flag and a header, and that goes to an.

I

Entire suite of services, and even if we have.

E

A limited list of single margin, I'm happy when it's get arbitraries and it's hard for everyone busy right well,.

H

My favorite, the talk about like there are certain classes of merging that we can do relatively easily. Thank you actually, I I'm, not convinced of the merging, is really what we want. Andrew he scroll back up to that first example. I actually think that it can be pretty nice if this just worked correctly, where, when I'm at one, where the left side routes over to the virtual service on the right side, exactly.

B

That's the aim or that's the aim of this proposal. What.

E

Does it mean busy I mean what does it mean if you, if you match a header.

G

So it has to be a dag right, it has to be directed exactly always. It cannot be resolved right. So.

H

You do the first match. You say you wind up at the second service, then you only apply second service rules and.

E

Is this is a point where, where I want to mention scalability- and there was this ordered list of matches me right.

I

G

Need all avoid to be able to do you.

I

See what we're now.

G

Saying is: there's a mrs. synthetic match construct which is yeah when its header, exact bar and whatever else is happening in the sub list.

H

So yeah that's effectively. We would have to merge the virtual services in pilot.

E

We can imagine in pretend is not clear how we can be. You know made to scaler, yes Quincy, and what do we want this? We would just move to approach the other side. I mean from my. You have Detroit host that sees too much and is a literal host. You need to apply a large number of rules because of the. If, if wearing a large organization, you may have prefix, mattress and so forth. This at some point needs to be optimized in scale to national breast.

G

You have at some point you have to take the ordered list and find set the size it. We are aware exactly place that which is possible.

J

It is personal, worry doesn't want to do it, that's just so much complexity for something that you can actually automate at the control plane. Our advocate ops level right, it's like I mean this. Is yes it's a column, but like imagine, you build something else on top of whatever we have. That actually offers is abstraction. Where you know you have a UI or something that says like here is a something delegated thing, and then somebody else comes and overrides it. They all go through. One combination.

G

J

Whatever we want in pilot, it.

G

Almost still does the dispatch a sort of list yeah no.

J

I agree but I'm saying that this I mean is rather than trying to make a hierarchical list with a non-void. This is something you can actually handle very easily with an additional layer on top not directly as a yeah Mel thing that interacts with pilot. But you know you have a build additional layer sticker database in and that thing automatically takes and much this thing and you.

G

Know is there's two different issues like I think we know how to make pilot like if you want to do a delegation thing to another declaration for the same domain, that's fine! We can make that work right, though the configuration aspect of it's not hard. Well, there's a separation will talk about the access management stuff and then there's all play that can do is flatten that down right into just an order list by basically treating everything as a conjunction. Yes,.

C

J

Doesn't actually I thought for the whole thing? The ordered list stuff is not really that easy because, like when you take a service, and you want to get the list of all virtual services it applied to it.

J

You are now going to have two or three matches, one which has a wildcards assist, Astrix or local and then another which has an exact specific match and then there's something else that comes in between and then you have to take all of them combined and merge them, and this is where the match conditions start getting complicated because you have a red X on top and then, like you know, somebody else has a specific part in the bottom. You don't have to do.

J

Is they get matching according to how on why would the extra gets much easier, miss good thing? That means you have to import JavaScript objects possible to this because I mean people probably have like you know some food comm / Department want / something else. I just want to delegate snack Department goes to this virtual service, where they can do whatever the hell they want and that's where all the follow. Ups, not English, which.

G

I mean sure, trying to explain to people that we're gonna reorder, something right or like that difficulty we're gonna have is invalidation right, because you're gonna have raba clusion right, but trying to tell people that doing anything other than order dispatch and then we'll say that somehow magically we can make order dispatch tree based dispatch sure, but we shouldn't present anything other than order dispatch in the interface. Otherwise, it's gonna be really hard for people longer. Simon, API, I.

B

Totally agree and that's why this is an order.

J

Dispatch right now still using the same virtual service, where, if either you start writing this, the first thing I would end up doing is like great prefix matches and then do it by route destination goes to some fubar service and then I'll tell him those those guys know that they're there chunk is under /qu bad fix and then from there they create additional prefixes and drug access and so on and so forth.

J

What we do instead is have a different cost, a completely different cost truck that that has same or limited semantics of virtual service, which simply says host match and delegations, and you know some features that we know can actually be changed. Like you know, the header matches can actually be all appended together, but nothing in nothing in traffic's, nothing else just host match and probably the source label match or not. Even so, they will match. I mean that.

H

Doesn't address the primary use case, which is path based routing right? That's.

J

Only at the lowest end of the chain I mean you can't do part by sorting on the top and X parrot to work in the bottom as well. Right so.

H

The primary user pain point, though, injecting.

E

Here is very easy: I mean put, percept was a part base, we can just extend things, little forced being just hostname, we can say host plus Pat, and then we can. We can treat host plus or it plus prefix us kind of.

E

Scalability is that you know doing trim I mean none over. That is the only option to scale and way to large number of which run, services and paths. Yes,.

G

We need envoy to get say.

E

We need to get better, but if we impose constraints from Zen saying that hey you need to do, reg exists and you came early opportunity for them to get better. So quite.

F

A question related to that scalability. Couldn't this be solved by just running multiple meshes on this name, cluster.

J

The complexity into something else right, I mean this is effect, I mean the simplest and the most dumbest way. If you give me like one week to solve this, I would just simply have a very simple, my sequel database in a simple UI, where somebody goes and enters the rules as a top level tool, and then somebody goes and enters a more specific, delegated rule and I would just use a bunch of scripts to like generate to spit out the complete rules.

J

The other ways of solving this is like, as I said, you can actually have delegation. You can actually push this to on why some pieces of it connect goes to unwind. So on whatever.

E

Which one is exactly what we are doing, I mean what.

J

I was trying to say, I mean so if there is a dire need for something. That's not like you know, you can just start with scripts and database and so on, but rather you want to create a full full-scale, API I think then create a separate constellated I mean like you, don't virtual service pand, whatever it is, some fubar virtual service. That's look at new API construct which has like set of all hosts that would match, and that's the only thing where the delegation is set and it has a very limited set of concepts.

J

It doesn't have everything a virtual services and we have to yield in subpar a week. We have to drop the pathways, not just stuff, and we simply have to do. We can only allow like Adam at just because these things will just simply get appended and.

H

J

Don't think the primary use cases pathways matches right, I mean primary use case is more like I'll just say: I wanted to find a top-level thing where I want to add a few set of headers for everything else and.

H

Not these cakes, yes, this.

B

Is a simple example to illustrate the point: I think.

G

There's a separate concern of like hey for all the services like in this namespace, whether it's a virtual host I, want to make sure that these headers are propagated. All right yeah, but.

J

That's not the primary use case here so.

G

A

Yes, I still come back to my question because I see that in both cases left and right here, the destination is Bart into service. Cluster local I think that.

G

A

The case when we have like a shared back and basically oh, is.

G

If you're saying you know, one virtual service is referring to another virtual service as a destination. Yes,.

H

Which doesn't work today, yeah.

A

Exactly which doesn't work today, right well, I'm.

E

Saying basically, you you combine the rules in recipes with a script.

E

A

Here by the way, so here we're missing the second host like in the proposal, so you have a virtual service and it only lists list full team.

E

A

Have this right.

G

J

Example is exactly.

G

What I would yes, one.

J

Example right we can even write that that merging thing like you know some that simple later, when somebody talks database fetches the thing and spits it out. We can even write it for end-users because, like we know how this thing works much better than anybody else. So we know the semantics and you can just simply expose a very, very simple, demas UI, which simply says here.

J

The top-level thing that you can configure add a second level things, and it has the final, the leaf of this thing and where you actually are now only things that we know can actually be just.

G

There's a separate concern or a separate topic here about Oh, there's, obviously the scalability stuff, and what we want to capability the system to be there's. You know how do we won't be able to do this, and one other thing that needs to be dealt with in general is how do operators clean ownership of DNS name spaces that are not implicit yeah? We don't have our back over hey. How does Costin claim ownership of start Google com right, mm-hmm,.

K

G

Know this works, for you know, subdividing a namespace of an already claimed host. Yes right, it doesn't allow me to claim ownership of the host itself. I agree, which is a fundamental issue that we also have to solve.

H

Yeah sorry yeah.

B

I totally agree with Lois, but I think that can be dealt separately rather than on this proposal. The idea here is, in my mind, at least the mental model is every service in your cluster. Ideally, you can think of it as a reverse proxy in front of it, which is doing all the routing for sure yeah and.

G

The issue I have, and do you think these need to be solved together to some degree? Okay, if we define an acting system that says this namespace owns, you know start out, you know yahoo.com, and then you want to delegate to a virtual service declaration in another namespace right and how does the a clean system work without declaration all right? What are the expectations around the ability to edit it? What kind of declaration must occur in the yeah and.

J

Priority inversion and all that stuff is in.

E

Real world I mean what what what what's the example be a real world? Is you have Zack owned by my company dot-com and that company will delegate to different services? You not have you know who team one delegate using the bar as as a host name, so.

G

That just means that, like this there's a common root right.

I

G

Principles like I would change this example. I. Don't think this example is the one that we want. It's not.

E

Is that so making genetics is.

G

Usually, as an owner and there're children, those that kind of peers, yeah.

B

And I agree I mean we can have an example where you have a common host name, which is where, in this example, it holds true, and you have four specific paths: delegate your routing to specific services, different services and then, as you have sub routes and paths matches within those. Your virtual service spec keeps on growing I. Think. Currently, that is a real problem where, if you have a lot of routes for a specific host, I think can I it very difficult to my that.

J

Is I agree, but what I'm trying to push here is what you're trying to compress it. I mean what you're trying to push in is the user experience in terms of like and how you're going to match and how you're going to like, let thing delegate and so on into one giant war of yeah meld flat at the a melody.

J

It is a into an API that is meant to be like more modular and like as a composability API, rather than doing that as pushing it down to the lowest start of API construct a second layer above it where you can occur, you can create exposure thing to the end users Marrakesh. They can submit their things and you have the logic.

J

Click merge and you can kick out things that say: hey you have a path match and you have a ragged Smasher is not allowed, and so you do that at second layer about that actually may keep things.

G

Really the issue is right: now you require people to learn to API is exactly with 80% functional overlap. Then.

J

You could still have it with same API and I mean as the the tool that he presents could still take two virtual services and could still basically like can have a like hash, include', a different virtual service, something about sort, and you can whole bunch of imagine it's. The study.

E

So, by which I mean we will security engineer will understand this I mean understand what does implications know.

J

They know by the security engineers are only gonna look at the final flat and virtual service and he is gonna basically.

E

G

Should correct that, ultimately right, there's a flattened, serve routes for house that can be inspected in tested, that's different from pipe. So this this is the solve an authoring problem. This is not a solve.

B

Operator can look at I think if an operator looks at what they have configured the smallest unit possible. It makes them easy to understand things and I agree that when it gets lowered to envoy, you still have a big list of routes which is difficult to debug, and we can't have a good solution in there. Well,.

G

For that, actually we need provides Kenema like standardized testing in validation. I.

J

Don't agree, but what he was saying is that you, you also create a flattened list from the virtual service itself, like for the model that you have. If you have a layer about like these two api.

H

J

A flattened one giant virtual service for every host, which says here, are the list of all drowse that I map. This is something that machine generated, so there's no chance of errors. The humor input things which is also gonna, be in the same syntax as virtual service will be merged by a layer about which hatched and spits it out. Then then, like in the security person, would actually inspect the final flattened virtual service that your layer is generating and the end users are actually end.

J

Users, slash the local operators, only gonna see his section and the words of it that he asked and he's just gonna, be like it.

G

Right so that that's the get off version in the bottom of the document. Yes,.

J

I mean so Shriram I understand what you're recommending, because the github thing is like you know, within the same thing, I'm actually like in having a version of version of different virtual services, and he know logically.

B

The same thing right I mean you are trying to lower the configuration from multiple sources. You write a merge bot or some board that will do it or the SEO control plane can do it where you get uniformity for everyone.

J

B

J

Control plane, so the main problem here is there's certainty in terms of like know. If there is a there's a bit of thing here, where you have to decipher what the user intent is and I got enemies, for example, if I specify a slash, Google and then somebody's ass Google slash one two three, there is an explicit intent here which which, with, if you do with the NIST you they effectively in term, not even interpret images guessing the whole thing and trying to like in a match and let combine.

J

Whereas if you do it at a layer about, then you can present user. Your new.

B

J

Shriram, when you do things to.

B

Layer above you do two things: either you take away some of the functionality which the layer below provided or the layer above is the exact same replica of the configuration and the API which is below because you want to provide all the functionality. So you haven't actually solved the problem. You just pushed it to a you.

J

Don't have to do a merge body, you can actually make it interactive by like making a merge mode. Actually ask the user. I said.

G

Whatever Shriram, it's like, it's extrinsic to the increase or structural surface.

E

Implemented we can discuss a petition after a griot requirements. I mean is the requirement to delegate and to impose some sort of structure and and how much flexibility do we.

I

Know that there is a requirement to enable partitioned authoring of names, that's clear for multiple reasons: I mean for security. Another reason. So, let's assume we get that in and.

E

There are many ways to implement it. Zero means that we delegate host plus prefix for short decisions, maybe some headers. So let's focus on the narrow cases that we can agree on. First right.

G

I think the the tricky actually thing to resolve is whether we want to make this a feature of the stack or a feature of some system that sits on top of the stack yeah. So.

A

Thing and explain what I understood it is in more like simple things. So are we trying to do some sort of viral article partitioning here where we have some set of route rolls right that applied to the like? Almost in the DNS, let's say: I go to google.com I have my top-level rules and then, depending on I, can go. Let's say to translate. I apply some other rules, but there's.

H

A

Now it's actually it's not really delegated, because they will be all in the same order right so.

G

A

It will go to the next layer.

G

Solve is not an infrastructure problem. It's an authoring problem. Yes,.

A

So, who writes? Which.

B

How do you manage ownership for this resources? Ondra, that's the thing.

H

I'm trying on it and then once.

B

You have the right ownership, you can build our back on top of it. If you don't have ownership. Our back is very difficult.

A

L

Already do this in Cloud Foundry and we I expect would continue to do it at a layer. Above we already support ownership of domains and subdomains, who don't support ownership of path prefixes for the complexities that have been discussed.

L

You know, another team could start taking some of your traffic unbeknownst to you and it sounds like it would be a lot simpler if we could just give sto a computed flat file and take care of our back ourselves.

G

Yes, this tension right where we have alternate configuration modes configuration systems but have their own compositional models. Their own are back models and.

H

He likes being essentially.

G

Virtual services infrastructure right just basically what versus you know what we might want to happen in kubernetes right, we're we're also trying to maybe solve an authoring problem and that's the tension. Yes,.

H

So, as far as like short term steps to proceed, I think Sriram is right and then, like a tool to prove this out, is a reasonable way to start to implement this right. It's like we clearly, we don't quite know what we need here. So I think that that would be a pretty lightweight way to iterate on ways that we could expose this and what that new API would look like without actually impacting Pilate right.

J

It should be a completely interactive tool when like when it actually looks at these conflicting paths and conflict in networks and so on. It actually presents the question: is the user before actually like doing anything? It's not an out of the ball asynchronous like admission control, pool or anything about sort it just it's purely an interactive pool, and we can actually we can ship it.

J

As one of you, you know we can have a simple UI order form and so on the just has it and we can disable it if they don't want, but we still solve that problem in some extent right. We.

E

See them either gain your pilot to to kind of at least enforce some I mean.

H

E

H

Concern as well right so like, but right now we don't even know what the model might quite be to do the delegation right. So we can prove that out. Then we can decompose.

G

Right do a prototype right, which exists outside of pilot yeah.

H

That does emerge yeah.

G

Work through the details of the merge and see if we want to intrinsic I that behavior in pilot and also make a separate decision about whether we wanna move down a CI or not precisely.

H

B

Still have to solve the our back problem for domains regardless.

G

B

Agree with you in the last statement, I mean do you. We should have a separate discussion around how we'll have correct our back semantics for domains. Well,.

E

Actually, if you look at the reverse, I mean if we start to talk about discussion for the mess and how we delegate and how we actually shape configurations that we want to reject. It will also simplify, because if we have some way to delegate at arbitrable, it would also fit into this. This delegation, because.

B

I, don't agree completely with that, because if we keep the same order and people are and the users are supposed to configure what your service resources directly, the virtual service resources will keep on becoming larger and larger and difficult to maintain. So.

L

Do we really think that users should be configuring virtual services directly? That was not what behaves.

J

M

L

Gonna be the place where users are expressing intent for kubernetes.

G

L

G

Gonna, do your back.

L

And computation.

G

Like most things right yeah, we have an infrastructure, all API on virtual service. It does a lot of things and it to some degree or other is usable directly today. So I do expect lots of users to keep using it yep. Now whether there should be higher level. Api is on top well, we've already seen on happen with key native and we're not gonna. Preclude that there's a separate issue about whether we want to keep making usability enhancements to virtual service Yeah right to support some of the things that need rise right.

G

We burn off in Sri, Ram and I have discussed a bit before I think we need to prototype it a little bit. Yeah I, don't have an objection to maybe doing a prototype like in a branch and doing it directly in pilot. We.

J

Could do it in a controversy? That's why we have it for this thing right, I,.

G

Mean if you either gonna pilot, do it or an sto controller? We're gonna have some other piece of code. Do it.

N

J

This code can I come in where the code dress is different. That I mean like we have all the libraries and validation stuff in pilot, but this thing is gonna have if you're gonna, prototype the merging and so on. There has to be some form of a database and whether we can use the same at sea, RDS or modify and write, and so on. We just keep that separate and then bring it in which is a much more easier path, faster, D plate, rather than.

N

Let's say for sake of argument: we assume get-ups.

G

Right so there's a file system with all your coop config in it right. It's just the directory structure right, which has all the namespaces mapped right. So this directory for namespace and all the resources belonging to namespace belong in yellow files in those directories. Right.

J

So yeah alright, in that case, then you could use these T of CTL as a experimental mode where you can actually try to do the merge and spit it out, and somebody could actually, like you know, inspected and on that's.

H

Probably yeah I mean there was always the plan right error. That was like wait. That was always what I imagined this here.

G

Control like it certainly.

H

G

A reasonable next step that seems like a perfectly valid thing to do now. How do we do that by like love to design some convention and.

E

Different approaches I mean this is one approach, but also to see how it feels to to go it with just you know: hostname solely delegation hosting plus part only if it's easier to understand for users, because.

E

They can use the tools already.

A

Confusing when you have like to choose between two simple things, we cannot like just give them everything and say: try all these and let me know which one is which of us. Let's.

G

Say is to control, apply a virtual service, but you know and it processes all the delegations yeah yep and we say it's experimental yep.

B

Okay, that's that's a great start. I mean we can figure out if there are some hairy corners, which we haven't, thought of and see the usability point of view. What makes sense yeah so.

E

Yeah I'll still appreciate if we also explores a bit wrong, I mean just just to have some way to specify this namespace is only allowed to support ecosystems. This the true you have to do that.

G

E

Matter what and if we do that again, we need to see how it ties into into into this delegation, because they are.

I

Related, yes, so we need a separate proposal for.

G

That, yes, we have to have it I, don't know any radish. If you want to volunteer for that or not, but.

B

We have thought about it. A bit I need to make sure I have the bandwidth in this point or something to work on it. So give me some time: I can come back on because.

G

I know the kubernetes community itself is also going to care about this because they have the same freaking problem. I recall.

L

Seeing a demo in a session at next on this somebody had wrote a cube control plugin that took file based a structure and did the computation.

G

And the plighted- oh yes, so Google has a product that kubernetes configuration astronomical. The name of it is which is basically a kind of automation around. This get ops principle that does policy composition and other types of things and I'll have the guy who work on that helped to review whatever we do around the or back stuff, and we can also talk about how the composition or delegation might work. This composition is really domain-specific, but we could make sure at least we are aligned in convention.

G

I'll send out a link to that the demo and the docs for that tool. I was going to mention that earlier, because they have fought through a lot of these types of issues in in a generic sense. Obviously, we have a domain specific issue. Yes,.

B

That will be great and I mean we have discussed on this a lot, but please free-to-air your comments in the dock, so I can.

G

Definitely I have just busy.

G

A

The main comment would be that I guess the delegation is not the piercing it's more of a hierarchical nature right. It's.

G

B

Have to be a it.

C

B

To be a dag, yes, and actually there was another name which we thought, which might be better, is forwarding that's what traditionally in load balancers you see when one virtual.

O

B

O

B

That term, to it, that's better more for when you have authorship and ownership it makes, is that an author of a virtual service wants to delegate to another which will service awesome.

G

Right now, yeah, the other work we're gonna have here is in some cases like a virtual service, may be bound to a gateway only.

H

G

The Gateway and it may not be the same gateway, but the other virtual service that.

E

H

They could delegate it down. Can't bang two gateways like a third, so.

B

There are few things that I've mentioned in the doc when you guys get a chance. Please read about these semantics like for the thing that just wants to be delegated to. We can even skip the hosts if you want, because the hosts of those are currently ignored. These are just for composition. The top-level parent is the one which gets to decide what the host name is. I thought.

A

We would also do this forwarding, based on the host like the the top-level, would be I. Don't know, maybe team one something you.

G

Might want to host to be an agreement, even though the initial.

A

G

While declaration on a wild-card in it yeah and.

A

Also, like it's easier to reason about things, if you see what the hosts released, so even if the traffic that gets there only has that host right when you like display it I mean.

E

I do is, they are also HTTP level. Constants I mean if you, if original host in the request is food team one are we going to forward with the original host on? Are you going to forward it with I.

H

Mean will do what exactly what the virtual service says if they want to rewrite the host. We have that.

P

B

H

Degree right, yeah.

B

I think, if you think of it in the sense that they are actual reverse proxies just sitting between them and you think about traffic flowing through them, I think we can go. We can simplify the how the resolution will work by making the traffic yeah.

H

The problem is cooking into that one time right because it gets to spend yeah I.

B

Agree and that's why the control plane should be resolved in making the set of diverse boxes as client proxies resolve to config, which is enforce of the client, but I do think that that's right mental models and think about this one.

A

Okay, anything more on this topic. We went way above the limit I imagine, but that is good because it was a very good discussion. So anybody wants to add something like super importance about this question.

L

Are we thinking only so far in the context of HTTP now, because.

J

L

Allocation and yeah.

J

What and quotas.

L

Gets really complex, we've done this in Cloud Foundry. We allow domains to be associated with one deployment of gateway, routers or another, and and the cardinality of pools where I guess in kubernetes land clusters and those routers and the allocation boards gets really fun. So I only see HTTP here, but it would be interesting as as the community thinks about I miss you TCP IP is.

G

A hierarchical, namespace to kind of know, guys a namespace. It's definitely a hierarchical combiner.

Q

This and I it's also very but yes, so so.

G

It can be done, ports are orthogonal which make things all, but.

E

What are very important, the stateful sets, which are we still it resolved for some points they.

L

Are back around in port ranges that you have it's complex? It's.

B

Really interesting in complex, but.

E

You need to do it, I mean port, at least for stateful I've been having dedicated workforce. Careful sense, I think it's it's a pretty urgent sense. It's kind of so any viable solution is shortened and.

L

For what it's worth most customers, we've talked to say that they're, the clients of their workloads that important on HTTP protocols don't support SNI.

E

That's okay, because we have a sidecar that can support this anion and kinesic this and I. So that's kind of the plan for since we are being much more mutuality, a race-neutral TLS also are the same ie. Yes,.

G

Smi is an issue at the edge for nominate.

E

G

But within the mesh, SMI is fine. Give me reference we're talking about at the edge.

E

P

E

Kind of Sony option expires innominate. It's really has.

L

Anybody thought about dedicated VIPs and spinning up routers on demand and how to avoid making a change to a load balancer in that case. Well,.

H

So those dips work today. The only problem is that there's a DNS plugin for your application to resolve a host name into the bib. No, no I want to avoid it at all cost DNS resolving to more than one IP yep.

H

But that's fundamentally, if applications are trying to address a bit via hostname, there must be dns the results that name.

L

H

Nothing about use horizontal.

L

Scalability of dedicated VIPs are you at.

G

The edge or in the mesh only think about the edge okay. So at some point- and this is again a gap- we need to think about how gateways declare their bindings to VIPs yep, because that's so well, we point out the DNS right is provisioned. Based on that.

Q

A

We have to get to the next topic. There was a change here, so I switched.

J

And because, like you know, we could discuss the endpoints thing, but they had asked for this thing to be discussed. So I thought.

R

So I wanted to propose an approach to remove the need of having the in it container in the pod in the issue. Pod I'm, not sure if I can have a presentation but I'm not sure I can share it.

J

If that helps the file, I think.

R

Let me see: okay, I think I've found a way yeah.

A

There is a button and you just present yeah.

R

A

Are you a nice Joanna's, okay meeting or no yeah, we see, can.

R

You see: okay, yes,.

A

R

The problem like I said the issue in it. The container needs to be privileged, or at least to have root and NetID mean capability, just opinion.

R

Sorry, just about me not rude. Okay. Now that mean it's still a lot, because it can obviously, with that you can join the match, but you can also a part you can also leave. The mesh can also modify the community as being set up and still the IP is of the of the note or the other pods. So.

J

So this is just an image container right, which is not even going to contain any application. I mean unless you write your own inert containers and so on. This is just gonna, it's transient, it runs and then it is gone, and this is not something that this is running like, while your application is running or like so on. Right.

J

I do agree that the problem is that you're giving developers the ability to run privileged pods, which is something that some, like you know, deployments just don't like, but as such these at this point on the security hole and so on, this is like the thing is set up once before any allocation.

R

I mean let me get into a little bit deeper in the into that, because I understand that this is the point of view of the maybe of the community. But let me ask you, then: let me add another piece of information. So there is an upcoming feature in in kubernetes, which is called pod security policy, which.

H

R

To specify what a pod can do certain it's a set of profile for the permissions that you give to the pod processes, and it's now in beta. This is something that, if you don't activate it, anyone can create a privileged spot and that's that's basically how it works out in how they any container works in in OpenShift, so I'm about, for that, so I've been using openshift.

R

Obviously, in a bishop, these features has always existed as this security contest constraint, sec and, as you probably know, in openshift, it doesn't work by default because the default SEC does not allow you to create privileged spots, and so you have to you have to add this line to your namespace. You have to run this command on your namespace. To basically give the default service account, privilege SEC. Therefore, the default service account can create privileged spots. This is like giving your developers root access to the to the nodes. Well,.

E

I suppose the contents are still contained. It's.

M

Only for the part, but so know.

R

That, once you do that, your developer can ask privileged access for the containers that are.

E

Ending the container set.

R

But you cannot stop so.

J

So let's say this is the case right then. The solution that you're proposing like that when I was looking at solution airy proposed like even if what a moment was like a parliament that doesn't. This is actually the this is actually an issue. Then the solution aliyou had four poles right, which was to actually use the controllers. To actually do this, I mean that would be in the table in the next slide. I guess yeah.

R

So let me so yeah, so my prediction is as soon as other kubernetes distribution will pick up on the pod security policy and will create a default security policies that don't allow pods to be privileged and therefore to have previous containers inside of them. This problem will become apparent also to the you know, more broader issue: community, ok,.

E

So I have two two two two two two pieces of data that I want to add here: I, don't know if you are aware of them. One is that we also support the proxy interception, which is more scalable than redirect, and the only way T proxy works is that if the sidecar itself has net admin privileges, because.

H

E

Was you a system course that requires not admin, so he.

H

E

Not a problem only for the unit container is also problem for the sidecar. It's not privileged is not true. It's just never been.

E

R

E

Also, I want to add that once invoice supports UDP T proxy is kind of the only way that work with you db2 culture UDP. So that's why it procs is important. We will not be able to support UD be captured unless we support T proxy.

R

Okay, I don't have a solution for that intercept.

A

This is such a big topic, so.

R

The proposed solution is to take away that piece of code from the init container and run it somewhere else and to run it in a demo set. So.

H

R

That is a set of pods that run in each node. The life cycle of this demonstrate is to check for new pods and then decide whether the pots belong to the mash. If it belongs to the machine, it's not initialized run those iptables commands and then in the annotated pod as initialized.

E

They won't set start a new pose new containers in support. What kind of permissions we have is a demo set? The.

R

Demonstrate is privileged.

R

Anything in the pod is just going to enter the pod name, space in particular space. So once you're in the network name space, you can run those IP tables. You know here's.

J

The problem, though, right so until the demons that is activated that pod can receive traffic from anybody. It can also send traffic to anybody so and.

E

No because no TV show lies, it can have initializer judge adjust, which so we can prevent that.

J

Is that for it like? Is there a guarantee that the pod will not receive in a traffic or send any traffic on.

E

J

The thing to do yes,.

E

R

Right now, I don't have that split synchronization, because just because we haven't developed I think it it's possible to create that. Well.

G

If you have an innate container that prevents the application container from starting until the event container is initially.

J

Penton I hear you in the unit containers running on a dyno.

N

G

So hot in it we'll try her to check but doing nothing. Basically, why are you basically asking the demon, sir provision me please I.

J

See: okay, then the unit container could act. Action right.

R

Now yeah, maybe you could check internally the IP table. I, don't know if you need any specific permission to just check.

J

The unique some sake right you.

R

J

R

There is some check we market the pods as with an annotation, to say that yeah so.

E

Are you sure that it is not possible to modify the border to run stuff in the photos? With with a specific image me like? We do a second injector, because that will that will Stoke the proxy proxy and and.

E

R

Now the point is in: if you, if you, if you run something in the pol you, you will inherit the sec of that pod. Yes,.

H

R

If you have said you don't want to run a privilege, privilege scold, then you can't but another another privilege. Pod can enter the name space or namespaces depending what you need to do and modify stuff. But then it's probably it's going to exit. You know, terminate that process.

J

R

Just you know, and.

J

R

The pot alone so.

J

You could actually also set up a deep proxy this way as well right, so I don't think this.

E

J

A transparent call to CV.

N

R

R

H

So the other side of this is this: BPF: remove the requirement for Ned admin, no.

J

H

J

A different way of doing the same thing so I still requires cutter.

E

Please check I mean if you are driving people to the east, us that's really bad I mean from security point of view. You should only have NEPAD min and nothing else, I mean giving full privileges huge well.

H

I mean to Chris's point: if you look at the we, we do mark pods as privileged today d-link, but.

J

The open ships folks the not for us market, faster Blake because it was not working an open ship. That's why we had for us. We just only need the net Adam. You don't need to the list so.

R

I can add, I can add on that by the way net admin is still too powerful, I think for most organization. But the reason why for open ship it still doesn't work, it's because say: Linux and I I cannot tie the know why exactly but Celine accessing the way there. So the only way to work around that is to have privileged pods.

E

Yeah we use this. You.

E

Know we need to think about the deep rocks you need to think about other things. Yeah.

O

Yeah I thought a way took a lot of effort to remove privilege to true to both the unique container and the proxy in the community.

O

G

Still too powerful but.

E

It's still not as powerful as be literature, so.

G

It's almost to be an arms race, Kinsella problem.

E

Here, you're, not getting that that we know the internode, you can't let that mean all your support, so you can modify that you address of support, but you cannot modify anything it outside of support, but.

R

Net, in my understanding, NetID mean allows you to modify the net or namespace. So basically you can start you can enter the database.

A

E

Bility we still get one yeah.

A

Just let's, let's say somebody said here just to be clear: okay, you can read a new, more privileged container and connect it to mentor namespace of your choice. Sorry.

E

On scalability issue, I want to quickly talk about the fact we did the load test with this car ability test and some custom users reported the same problem. 1000 said we saw two thousand endpoints and we start having problems and we discussed a medical experiment, and we know a quick fix to improve that by caching, the virtual services, the other interest used in infant regeneration and I, think we are discussing cutting the one-zero-one release in two three weeks with justice Peaks to get scalability to better numbers.

E

Ability fixes to address this, this.

J

Is what we just start doing the list stuff yeah.

E

Just stop doing the least and cache the results in the push, context and reuse. That totally because we identify the.

H

Cause is a yeah.

E

Me too, proto compression, we know how to solve it is just you know, is not a huge problem with regard ones. You know one cuts and some what is the ones your branch.

E

A

So we still have two more minutes because we haven't wrapped up on this and we don't have a conclusion actually so.

G

We're gonna, like I'm gonna, go talk to this security folks on the kubernetes, teaming about their plans for reducing privilege and get them to review the proposal. Yeah I'm, going to talk to people like Maya. You know she's at Def Con, so we will get some review because we need to go through around her review with this. Okay. It is an area that needs more focus and we need a holistic proposal that enterprises are going to accept.

G

You know I'm not sure that they they're they're enterprises that are already willing to do what we've done today yeah. So it's not clear that it's, but there are certain classes of enterprises that will want the app Simone, absolute least exposures and we'll have to try and find ways to give it and.

E

There are the benefits of having observed. Initially, then I mean I can see. Many other reveal simplify the deployment. It simplifies a lot of things.

E

A

Is already so, there is code in github.

H

R

Can already run it and it's tips is.

J

The reason is the reason we can't have unite I mean it still does not remove the net admin stuff, it still requires net admin, and this thing is effectively. Removing the anti in a container which is actually nice and I was I would even go as far as saying that, if this works- and this is me and me, then you could offer it as an option in the 101 and.

J

Somebody goes and tries into themselves if.

H

You want to pull down the the current home charts and deploy it, but.

A

Finally, like Louie, wants to discuss.

H

A

Problem well, there is already a github, so the question is.

E

A

E

Please oh yeah.

A

E

Least, we can try it, and here we go yeah.

J

Now because it's like, it's suddenly doesn't like hard yeah.

R

This thing we created a separate repo, okay, I.

J

Mean you could start this to be like to merge slowly by, like a point, yeah.

R

Yeah, if you want me to I, can start thinking about.

J

It more at PR, the discussion is going on on the other things we could I mean some of the content. The constructs here are still helpful. I mean.

G

The other things to consider error. Logically, if Envoy is doing all the routing for all the parts you don't need iptables programming for kubernetes at all. You don't have to do it right, so you could actually eliminate you.

J

Still need for services, no, you don't need what do.

O

You need our IP about you, capture, the traffic.

G

Theoretically, right you're, just saying all traffic from a pod goes back into the local Envoy and it does all the right way right, which is really all that the IP. They know that all IP tables were doing anyway, so.

J

I mean, but then we also we could, when they also have parameterizing in a little bit by looking at the configuration, the sidecar injectors, looking at some in Bonn ports and so on, and trying to like omit the ripeness and of iptables stuff. If that.

N

J

That then, yes, you're right I mean I. Can.

G

I

J

Right that kind of doesn't we.

E

Can look it on into fuzzy boundary key? We can work around, which is.

E

Q

Domain yeah yeah, yeah preneur has the public.

J

Set up the network namespaces on and need.

J

G

Then the other thing is people could offer variations of this like if psyllium wanted to provide their own e BPF version of this. Instead.

H

G

Tables or or we did right, the way you would do the upgrade is roll out well, oh.

J

G

Upgrades gonna be an interesting challenge.

J

Work like this upgrade to say it's just a node node level thing for every new part of the.

E

A

That's a very good idea. You know like to just if.

K

A

Burning, it can be like it doesn't have to wait weeks can.

G

We get a doc going, the community drive just maybe take what you already have in your presentation and just dump it into a dock and put it they community right. So we can start just piling on it and add some are not necessary solutions but add options in there and then I can get it reviewed internally by some folks who are knowledgeable in this space too. Ok,.

R

But yeah I'll do that. I'll find a wait, but.

C

E

Also work for something, because community is getting very hard to.

C

E

A

C

A

All right, good thanks, everybody and sorry for keeping you late.

G

A

Exactly by seeing takes.