Istio Networking Working Group, 16 Aug 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Istio Networking WG meeting - 2018-08-16

Description

Agenda:

-1.0.1 status and fixes
- Scalability
- Memory usage in Envoy
- Split Horizon EDS
- VirtualService related topics:
- Istio Hostname Ownership proposal
- Support merging of VirtualServices
- Follow up on VirtualServiceDelegation

A

Okay, I think we can get started hello. Everybody thanks for joining this, your networking community meeting. We have several items on the agenda and quite a lot of discussions. I suspect we will have around virtual services today, so I propose. We start with the less less debatable items so, which are the 1 0 1 status and maybe like Costin, would like to have 15 minutes to talk about the split horizon ideas proposal.

A

That's a very new thing, so I guess people will need some time to to digest it, and then we can cover the virtual services. Part I, hope that's, okay with everybody, especially those involved like Brian Louis Niraj into the virtual services, API changes so related to 1:01. There are a few things that we need to deliver as soon as possible. A couple of them are some security fixes, so we discovered that the privilege, through is turned on by default now injected sidecar.

A

So we need to actually turn this off for most of the platforms except openshift. So the proposal is to add a flag to be able to turn that on so the default would be the default. Will be a privilege false right, not in a guard, we will still have the net cup admin privilege sir two throws before so for that we had. We have the other proposals to be able to get rid of that, and the other like security issue is the fact that we don't strip.

A

The x4x forward is for headers when the Gateway is an inch proxy. So that means somebody can prove that header and well, because some bad things to happen and the proposal there is to the hand flag in the deployment of a gateway to specify if that gateway is an edge proxy which will basically turn into numb hops. Numb trusted hops equals zero in on voice so that whatever will do it will strip all the expertise for others.

A

If we do that, okay, so if you have any like questions or concerns about that, please go and github and add comments there. There are the two to open issues, I'm gonna actually maybe add them here, and the next is scalability, which is fairly big issue in itself. Okay, maybe I'll! Let you cross talk a bit about that.

B

More like 1:02, he should not only one zero one I, probably you. So we did a number of musicians, because pirate was using far too much memory, but the optimization went into the three most commonly used CR these, which is literal service. This nation will insert this pretty much everything else in in issue or the other see. Artists would also need to be optimized the same way.

B

The root problem was that the config generation was calling model far too often- and you know it was not cashing the results and we really need to cache, because there are a lot of people use a lot of deployments. So that's that something that will need to take some. You know engineering resources and and contributions from everyone, improve networking security environments and pretty much all areas, and we will need some way to end of ordinance weekend.

B

We can move ahead with this with this and and most likely will cut a10 to when we make enough progress, and we, if we have another large chunk of improvements and we keep going and gradually move to you know I, never casuals, all the resources we need to cache. We are currently testing only with 1,000 services and I. Think few thousand five thousand endpoints, which is below and again where we are trying to increase, is due to some reasonable number, which again needs to be determined.

B

What what we expect and 5,000 10,000 services and and if I remember correctly, we set 1 million as endpoints, but probably not in this in one of the only but yeah please jump in and send PR sir, or help with this effort, because it's so critical so question.

A

Do we have some like issues people can take on well.

B

Don't I don't think we need we have enough bugs that are impossible to finalize already. You know, 600 how many we have creating a bug for this year. D is probably not very productive, will spend more time creating and closing bugs, but if people want to file bugs and and to track what they are working on, that's wonderful, I'm, not I'm, not gonna, get into hits 51 bugs ya.

A

Know like maybe on list of things so that can be notified. It didn't come good. Yes,.

B

We can we can I, don't know how I mean it's. Everyone is familiar with what we are working security team has the policies that mix I mean every every team has pretty clear, set of CDs and optimizations and and and any other optimizations that you you can find it's it's it's working. That's and one thing that we are still waiting 101 on is curies is looking for memory usage memory because we are using too much memory for each cluster.

B

Any comments, shirome is, this terminology is also fellows spending a lot of time. Accessibility.

C

There will definitely be another round of optimizations, but what we have have now asked austin says goes up to thousand services. We certainly need to do some more that camel conversion business and really keep pushing, but I think they won't the main thing right now. The late start at time of pods- and it seems like there is a combination of like some memory issues, an envoy and then some form of try I mean there's some lock somewhere and pilot. That's actually sequence, utilizing everything, and that just means to be yeah.

B

Sorry pirata so Sabine fixes a fixer and voice startup time which went from 30 seconds to below a second due to the stupid, optimization in GCC or whatever words, and yes throttling. It's still something that really remains right there for that. We need to there some tiles that we need to document about how to optimize and how to tune is to offer for a large number of tourists fast, as the G.

D

A

Any any question related to this.

B

Well, looking for volunteers, that's amazing.

A

Scene, if you have concrete items, you can also write, send email, okay. So the next topic is the split horizon ideas. Do you want to share your document.

B

Click on an income, okay,.

B

So I just wanted to you know, let you know that's one of the proposals that we are discussing for networking as part of the big 0 d, TN and and non-flat network.

B

It's a very narrow proposal, and you know it's it's relatively simple to do in in pilot.

B

It's probably Porto, gonna lure, aligned, I, don't know with with the other propose and another words that cinnamon and exact and other people have done for to solve this problem. Let me know if the proposed are sounds acceptable. It's proposing simply to reach different results, different ideas, results based on color and to automate the setup of gateways. When you have multiple clusters, that's it yeah.

E

Three reasonable to me: actually: okay,.

B

Well, that's yeah.

E

And I think it's complimentary to a lot of the stuff that we're talking about like I. Think the idea of pilot being able to serve across clusters is valuable and orthogonal to the concerns around my class radius and and per cluster failures on the Mac and stuff yeah.

B

So just you know please take a look comment: we'll try to squeeze it into one, but one if people are interested in these subjects are welcome to help implement it.

B

It's not a huge probability, but it's again something very nice to happen, and we want to make progress on all this. A yeah.

B

Anyone right now I mean I. Always people want to make it a piece here, but yeah.

F

So constantly um I haven't read this in detail, but this is along the lines of the same type of thing. Robert and I have talked about so yeah I think this I mean from what you described here, looks great to read. It detail I.

B

Mean just wanted that I do not to you, know, kind of just want to make sure people are.

B

To make it clear that it seems very narrow and is not trying to solve all the problems in the world. I'm totally separate documents for the other problems of the world right.

A

And so was the question. So this is the proposal and are you gonna actually work on it or somebody wants to work on it? They can work on it. If.

B

Someone was gets to it before me. That's wonderful, I probably spends the next few weeks. Working on scalability and and again I said this is e1 not to calibrate is new zero yeah.

A

B

If, by the time, I finish it scalability is implemented, I'll be extremely happy. If not, we have to start working on it. But again it's not a lot of code, it's more more kind of getting on the same page with everyone else and and- and you know, agreeing to go this direction. Alright,.

A

And also like it will be testing with multi cluster right. So it's yes.

B

One of the motivations for this is I'm doing a lot of testing with using multi cluster set up I mean my laptop using the mesh expansion and avoid connecting to two to the central cluster, and it would help my testing and the scalability words that I'm saying that's kind of how I ended up.

A

So if there are no more like no more comments on the split horizon ideas, we can move on to the virtual service. So it's good like we have roughly 45 minutes, which is excellent, so Brian you sent a proposal about he Co is your host name ownership? Would you like to present and talk a bit about this? You can.

D

A

Think it's better if he presents because he can scroll through the document. All.

G

Right, let me let me try to present here yeah.

A

There is a button there should be I'll. Actually it made me stop sharing. Let's see no do yeah, we can see all.

G

Right, can you see the doc yeah all right cool? Yes, so there's a lot of comments on it, thanks for all that feedback, but the so the general overview is things like virtual service are using hostname as sort of shared resource. So you know, whenever you make a virtual service resource, it has side effects for the entire cluster. So if I, you know, you can have two different virtual servers: resources that top the same host name and there's no real clear ownership about who can create those resources via something like turbinates are back.

G

So this proposal is about sort of associating post name ownership and with namespaces and kubernetes, so that you can limit which virtual servers, which host names a virtual service resource, can refer to based on the host name that you're creating it in so that maps to community star back it's not about changing the side-effects of what that virtual service resource can do once it's accepted just about you know where you can create them, and then, of course you know every namespace as a default domain, so you can automatically create resources with that domain name.

G

But if you want to refer you to other domain names, you would have to create some sort of binding or to allow that hosting to be configured and so I've got a proposal here for this traffic binding. You know, name notwithstanding and.

G

You know whether we want to do just host names, or maybe some other resources, I think that's an open question, but before I sort of go into questions about the proposal specifically, is there any questions about the problem itself? No.

B

The problem I think I do understand is agreeing that it's a problem. Okay,.

H

Yeah, just one question: are you assuming this merging of virtual services? Kind of proposal is in place for this there's this end under that, so.

G

I wrote this not assuming that was in place and and because of that I only focus on host its name, a virtual service. There was emerging between different host names for virtual service, so that this makes sense. If the merging goes forward, I think we're gonna probably have to add half prefix to this Association as well, so that you can limit you know if you've got to a shared host name amongst different teams. You can limit which prefixes those teams can configure as well.

B

One question I have is: who is an authority that design which pase pase is warning whatever music? Is your system, those resources, realistic ratio system, or can anyone put this kind of resources in any namespace, not example, yeah yeah.

G

So I'm not making any opinions about that I'd assume there's some sort of network admin who gets to Divya public list names and since you can restrict which types of resources different users can create, you could restrict the ability to create these association resources to only whoever, you think is an admin and then likewise divvy out the ability to create virtual services in those host names to you know. Whatever teams you want.

B

Our back is wonderful, but very few people. You know really understanding that. It's not so easy to. You know kind of put a correct configuration where only you some specific users who are admins have access to a nice space of a team and I mean it's it's it's a bit risky, in my opinion, I suppose, with sayings, for example, that you need to put all the resources ability it would not control the ownership in to is your system and simplify this.

G

Yeah I, see you're, saying I, think I think it'd be a really good idea to for the default install if we go forward this to yeah, basically do that. Oh the allow that, in only it's your system, I think we can create some. Some are back roles to do that. So, in other words, come come out of the box with some good roles that allow what you want. You know maybe create a new global name namespace. It's.

C

Basically be what it is today, where anybody can define anything and, like you know it's up to you and if you shoot your own footage this, what isn't because otherwise it'll it will break backward compatibility with everything else, that's out there yeah then, as part of the pops guides, you should have a dock which says hey.

C

This is what you should do in production, if you have multi team system and or if you're like having multiple teams operating on the same is jamish and here's the stuff things that you have to do this way, the impedance I mean the surprise factor and sticker shock was much less when people actually go play with like. Oh, this is not working, that's not working, and so on yeah. The other thing you start feeling good and the other thing I want to yeah.

C

So that was for the time two constants point: there are multiple ways of solving column and see their solution of doing this.

C

On our backs, think this is nice, one and I'm saying we need to see how this evolves in terms of how people are actually trying to use and different system they're coming up with before we settle on one specific mode, saying this is what is table support so until then, I would like to see this be kept on a separate one that we can enable and even be happy to enable as part of the default install but not pullin. Anything into this to your core.

C

Until we see you know how things are like okay and then yeah.

E

They're kind of extend on that a little bit like I think tying a hostname to a namespace, really isn't what we want. What we actually want is an arbitration on the hostname itself right. So if you have some permission to modify this hostname, then resources that touch that hostname, you have permission to modify. That's really what we're trying to model associated with name space, because that's convenient in kubernetes, correct but I started. I said.

G

Yeah, this is um specifically to allow communities are back to enforce that hostname to be C resource, binding, yeah, yeah.

E

So I it feels like a it's like I understand that that's the practical side of it. It feels like bad modeling. It doesn't die tying host names or namespaces, and is it really the semantics, especially once you start to get into hostname delegation and things like prefixes on the host name are managed or belong to different services and those services belong to logically separate namespaces yeah.

A

I actually agree with Zack like it to meet, seems a bit unnatural and too restrictive to tie the host name with the namespace. You know really.

C

What I'm seeing is that's! This is basically what I was fine can't ID two guys have feeling that it's not natural and others feel no is like nice, but so this could effectively be enabled, as sort of an alpha extension that people had in the off skies that, let me just let people users go and try this out, and we see how they are reacting, how they are like you know, do this thing and see how they're using it, and then we can make a more informed decision in terms of like okay.

C

This actually makes me so this doesn't make more sense, because an actual field testing will give us a lot more information that.

A

If they scare, the problem is still there right. So the fact that anybody can use hosts in any way so I like I, like what how this a proposal attempts to solve that the existing problem right. So the question is: if we have different solutions, I mean this is great. It highlights the problem. We have a potential solution right. Do we have other solutions like when we do this differently of.

B

Course we have many solutions, for this, I mean it's not again. One is to instead of using a nice that is to type by identity, which is what normally is done in in our box. So you specifies that only specific service accounts are allowed to control the host name, whatever they want to create it, which may be done with just our Park I mean we just have our about rules that specify role binding and as part of the resource you specify the host name.

B

You bind that host name with with with an account, and you can enforce it in Pilot or in whatever I mean it doesn't have to be enforced by our bar controller. It can be enforced by I.

B

C

Think avoid Zack yeah.

E

The pilot enforcing it is that then it makes it in congruent with other communities are back right like you're a it breaks. How Curtiss API policy works? Well,.

C

It's a better option for the galley where like, if we have a nice dependency injection framework that in sort of a preprocessor when that configuration is being ingested, that's part of the not just that mission controller or, like you know, spell before it, ships to convicts to pilot this thing is actually gonna go and validate, and then it's it's gonna spit out the errors and what it ships to pilot is the finance of configurations.

C

Without having to we can use it, I mean.

I

If, if, if you really want to model, it did even kind of more specific bespoke, it's you can you can always do it in CI, CD or any of disks kind of.

B

Is not easy, yeah.

E

So, like I think it's a legitimate request to use kubernetes are back to control some of these SEO style permissions.

A

Cluster cluster wide are back permissions like. Why do we have to do it for name space? Because you can? You can actually define permissions that apply so.

B

Why we can, but that's a bit harder and you want to mess up too much it which caused the white okay.

A

So then, maybe what is wrong is the association with the namespace instead of the circle.

F

A

Just like replace name space in this proposal with the service account, we might be in a better position, but.

B

That's the question is the better position or is not a better or more complex proposition, because.

A

B

Is very clean and simple and I like this about this proposal, if.

G

You it's not clear to me: it's not clear to me how you would map a service account into these resources via communities are back that I think you have to do it by creating these resources in specific namespaces. Yes,.

E

So that yeah and that's really the problem with the right this is a koreans- are back modeling problem. That's why I said that originally right, it's.

F

E

We just can't actually model the resource that we care about to describe policy about it, because kubernetes doesn't let us.

F

Do that, but each do on a buck is aware that can.

E

And I think a galley plugin specifically so that we can enforce the the right semantics for people using the criminals. Api server is an excellent way to implement.

A

Should we maybe look at least you are back for this, then you.

B

Know we can cry.

C

Because that is being implemented at our my level and that's it's just flowing through a pilot into an order. At least you are becoming. We can use the same syntax, but.

E

Ya know the API, not the implementation, is.

C

Implemented on its own, in order for the confirmation stuff, but that might actually allows us to probably have more more body call a better system in some sense by we can listen and not necessarily have to shoehorn one and together, but we can actually be like okay for virtual services here, the I backslide, it can just be a first-class abstraction. I can actually.

E

Model our domain yeah yeah.

D

I second, that I think maybe this needs to be presented to the security team, also to see what their views on whether it's to our back could be if the API level could be reused in some ways to render these. If.

G

You are back today is dealing with which services can access, which other services is that correct yeah, because this is I'm trying to model here which configs are allowed into the system and I'm not sure. If the overlap there is beneficial. So.

E

We mean the the API so like the literal are back. They Co are back API objects. Yes,.

G

F

E

Oh for services, service communication, but also we could the goal you know we're looking at. Can we reuse it to describe this case as well? 35.

B

Means our batteries are not service to service our workload identity to service. As far as I know, right yeah, based on the service that His Holiness business, it is not yeah.

C

So Arabic may may not work all, but I think a domain-specific model, for this would actually be I mean. But it's good to start with this, and then we can definitely have at galley level a constant back which says who can define what. But then the question of? How do you assign roles and how do you enforce of verify those holes, especially because this is authored by humans? So you have to probably authorize them with something and LDAP sense to directories. That is a much more open thing.

E

Louise, oh, if you constrain the problem to kubernetes incriminate, our back there's already means to get an identity based on external stuff. Right GCP, you get identities. All you know. That's one of the convenient things by using cranes are back for this case, but.

B

Very specific in any systems that exist today for configuration, has a notion of a user authenticate, a user that is allowed to make changes.

E

B

Know either you know CIC system or real person, kubernetes.

E

Is taking care of this already for us in the form of like I, could coop control and it uses the correct principle. It correctly translate local credential into a kubernetes identity, which we wouldn't have to reinvent right.

B

Another point here is: it will be very useful to see how this maps to the other proposal 442 our service delegation, other things, because somehow they are very close related and you want users to have to deal with five different.

A

So I don't know like.

F

A

Would you like to like say something because there was quite a bit of feedback I'm.

G

So yeah there's quite a bit of feedback. I, don't know that I want to take the time right now to go through some of the intricacies of the proposal itself. But with regards to the delegation versus service merging I think that this works in both cases like I, said earlier. If we're gonna do merging I think this proposal should include probably also ports and path prefixes as things that you can associate with a namespace so that you can control down to that level.

G

I'm not sure it will have to go talk about merging and how fine-grained that can be, and whether I just.

C

Need simplistic or or.

G

Not for that I.

C

Think right now, the plan for merging is a simple concatenation and it's not more of a merging so where we just simply take like a virtual service and then combine two of them for the same host, where we just append all HTTP routes from one end to the other, and that's just it, and that alone would get us much far and with that proposal this would just be it. This has no relevance to that. Actually, because this is you know, the rules are apply for.

C

One virtual service will apply for the others as well, because this is only focusing on the host names and not on the contents of HTTP TCP, TLS or routes that you define inside a virtual service. Yeah.

G

So I guess what I'm gonna say about this is I'm gonna. Add pathway fixes to this doc. I didn't want to do it, while everybody's making so many comments, but we probably should go talk about the other ways of merging virtual services at this point. So I'm gonna up to this stuff, probably later today or two with those extra details. But let's go to the next topic. I think: okay,.

A

Okay, thank you. Thank you Brian sure. So um let me see so actually so the next topic that I have here super merging of virtual service, Louis and he's not yet here I can't I, wonder.

C

A

C

We can I can, when it's very simple in the common problem. Is that, like you know things like a native and then the ingress and so on, people just want to be able to define multiple virtual services for the same host. They don't have own interrogation they're.

C

Only anything else, I mean they effectively saying that they by themselves are partitioning the parts and so on, and they they can be guarantee that you know they would be so in that case, it's just a very simple matter of combining all of them into one giant resource and then shipping that to I mean within pilot before the code takes over and, like you know, generate in the config and simple examples being like. You know, when we've seen in the for those of you are looking at the mailing list.

C

Right people just define a gateway, and then somebody goes and defines a virtual service. They find the same virtual service again, but the different set of paths so I mean this one be able in thing applies for destination, but you know different people come and define, upsets a.

J

Quick question on that.

J

So do you all like this proposal because it handles the most common use cases which is, like you said, a common hostname pretty with non-overlapping prefigure paths. But do you want to then limit it so that you can't have rejects expressions in there.

C

So I think the the contact that we would offer as a starter is that if you supply to actual services with the same host- and you know, then we literally be concatenated one sort of HTTP rules from one end to the other. That's just it and then it's all up to you in terms of ordering them and making sure that there is no, like. You know, conflicting path, replicas that will resolve on each other. We.

E

Can be a little bit smarter than that.

C

E

Yeah here in the there's at least some simple ordering we can do so reg X's are totally a different problem, because those are that some. But if we, if we look at just specific path, prefixes and that kind of thing we certainly can wider than one directly.

C

For the user- and we could, the least we could do is- or at least the only thing we can do is make sure that the catch-all path is always in the end and yeah. Everything else is just basically the order that was specified, so it we.

E

Can always do longest to shortest maintaining the same relative order to get better matches so.

B

My proposal has been and and I feel pretty strongly about it, for performance reasons is to you know, have something where we apply. You know exact matches, first prefix based second or other ratings, whatever energy and the default, and the reason for that is that you can do you know very efficient implementation of you know harsh mobility for the exact pattern and the equivalent for for prefix, and you can sort the sounds and so force would be. Or is this load nothing?

B

You can do about it, but at this we can have a squabble interpretation of yeah.

J

So I, like constant what you're saying the only other simplification which you think about is maybe, when you have rage axes for the same hostname, because we can't merge them properly, does it make sense to reject the configuration at that point.

K

Don't know acting configuration is actually a simplification as far as the end user is concerned, as.

B

K

B

If we can restrict them to to prefix I mean we see, we first do prefix matches so each Iranian students I have a prefix match. I.

L

Mean it has lower prodigies.

B

And predicts matches sure.

K

B

Is you might as well as.

K

Arbitrary concatenation order and just tell them that's what we're doing.

K

A

If the validation is done in.

F

B

Series, you can get, you know Prometheus graph on as a hedge, they can see what is.

L

K

Order dependence yeah. That means that we are gonna, have inconsistent behavior. Almost by definition, we should just. We should just avoid that problem for now yeah. We can come back to it later when we have a better feedback system and we've done a little bit more user feedback around less. I I.

J

Agree I mean if we have CRV status fields and if we can solve the problem or whose, when you start up again, if the results are predictable, that's the best thing I think I can actually show ya.

B

Again, for in terms of feedback having another generative megafauna is probably more feedbacks, then, looking later, as you can see,.

A

You can actually write.

B

A

I, don't have to have graph on I am like very minimal user of pilot III. I would like to have them see.

F

Our disposition.

A

That's applicable also for other things. For instance, there are like customers creating virtual resources and they start sending traffic and they get five or three or four Oh forth, because the things are actually not ready. So we can even have in the virtual service something to say: that's not contritely.

K

A

K

In on okay, let's.

A

K

These things, because there's lots of use cases of at least four status, that.

F

K

Related to this scenario, so let's for the moment just pick an arbitrary merge and leave it at that, so we don't have to deal with the API concerns so.

C

That we can track that as a separate issue like.

F

C

L

You can arbitrary.

K

Merge policy right.

E

We just waved our hand yeah.

J

This is the convention. Okay,.

A

So, actually am I midnigh I have a thing about the date using the oldest of the state as a criteria.

A

B

There's no way yeah and we can place you.

E

Have it I'll have some backstop yeah.

B

Production wins, there's no question that, but.

A

Can we just say instead of like so when we have to come, fix right that are conflicting? We will always just reject the last configures. You know that we will apply kind of focus.

F

A

Result will be the oldest one, so the user, after all,.

K

A

Look into each dressers times them to understand.

K

K

A

Because that will help us in case we like the oldest one already got, was sent to and for it right. So we have so like running down Boykin fake, with T all the server's right in your hands alone. We detect the mismatch, something we don't like. We invalidate the newest research that no.

B

Lady rejection, yes, okay, delusion.

A

B

A

Yeah, so we have to document that if they want to get rid of the old dressers, they have to delete it explicitly. Yes,.

H

I think what it makes sense to try to combine the old, be one model of precedence filled with this, like maybe have a precedence field optional at the top of the virtual service. That could be used to say oh I'm, being rejected because I conflict, but I want to be in front of this other things. So so I think.

K

I think we could do that as a follow-on if we felt strong interfere, but that it was necessary.

C

The actual you know the suggestion model having one virtual service per host, but in the case- and you end up defining two or three and we just it so that you don't have the element of supplies but yeah.

E

Like I think, a lot of this I think actually a lot of the use cases here could be addressed by allowing you to create effective, lead. The delegation right at the top level I want to split apart a URI in two sets of logical services and then I want to handle each of those logical services separately. Well,.

F

E

Should be able to do that or by merging back into one resource exam.

F

H

As long as it's separate.

E

Right, like I, think.

H

The merging is really the way you want to do this use case. I added to the to the issue, I added a scenario office. I think this is the case I've heard of every country. One guy talked to sort of has this same question: they have an application with some. You know website new host.

H

They basically call and then it's all gates out to different services that are that are owned by different teams, so they want to put all, but basically the virtual services for each team for each micro service kind of separate, but they want to merge it.

H

So that I think that's the most common use case for why we even need this merging right, and that scenario seems to me that if the virginal services themselves could have some kind of global context, like thing that says, you know the you know, Canada, here's the match, context emerging or something like that, but for all the rules.

H

So then I don't in every single rule within the virtual sort of back to say you know, you know if this header is this, do this, but also you know and that the pass of this this or this you you have to put that in every rule, if you're going to merge it like that in some higher level place and then, if we don't want to have delegation and two different things and I'd like to see, maybe that as sort of a global context, then you can specify at the time thank.

A

You for talking about having a sort of a match, wrestlers, almost right, so something that is gonna, be shared among multiple, like virtual service rules right every.

H

Rule within this virtual service has a global context of match.

H

So I sort of get sort of a global match, yeah.

K

I, like delegations, does that, for you.

A

K

Delegation is attached to a match. Yes, yeah.

A

H

A

H

Requires you to actually have somebody separately own, the top-level virtual service that has all the paths in it I think people like to be able to he can specify their paths in their virtual service. So then.

K

H

K

No conflict Frank between, like that a match and a delegation and then the delegated thing, having basically suffixes and once the top-level match was right. So it's basically strict subsetting yeah.

H

But it's also it's that's all put inside of a single resource that the team owns as opposed to the delegate model. You have a virtual service that has the pass in at the top level and then separate virtual services for all the rest of the rules. That.

E

That top level thing is a shared resource. You can't.

J

Get exactly exactly.

J

Parind virtual service, which delegates out you have the same problem, yeah that, in my.

E

Opinion is a very simple situation: yeah.

J

I think capturing because when you have a top-level matcher and then multiple virtual services inherit that you still have to associate them somehow, if I.

H

J

Your proposal correctly, okay,.

H

So maybe it's a clue enough, instead of having a virtual service that has the matches in it. So there's there's two scenarios right: the the delegate model, which is the one where you have a virtual service resource, and in that you have a bunch of paths and they all delegate out to other virtual services. Somebody owns that it's a shared resource, then each of the teams that are implementing the different paths have their own virtual service and that's where all the other rules going.

H

So that's that's model, one model to which what I was proposing is instead of having this top-level resource within each of these team virtual services right in there there's a list of rules, a host that you could add a top-level inside of that resource sake. Oh here's! The matching that activates this rule about any of these goals, so I would say that /pha who and slash bar are my matches and then I can now have a list of rules and that's implicitly not ending goes off.

H

But the difference here is that the resource, the team that actually is implementing path, slash, slash bar, is putting that information into their own virtual service resource. So, instead of.

E

J

Basically, I think.

E

It's a feature of the system that they have to go into a global resource. They claim slash flowing, slash bar yeah.

K

I think it's it's a much better permissioning model as it is about the means of producing a unified wearing structure and.

B

Having the right people notifies the right configuration because they are different, we do and every team to claim and.

A

So there are two different personas here right: one is the network admin or the measures.

B

A

Yes and the other one is the actual like corner of the service.

B

Yes, since the owner, as I said, this should not be exposed to things that he should not break I mean it's. It's.

H

So you guys think, but it's really preferable to have a separate own or a high level pass for the application.

K

H

K

Of the implicit permission model yes and.

B

Part of the explicit permission. What else? Well, because the previous proposal that we discussed was the exact is that that will require anyway central resource that will specify which hostname is authorized by my home. That.

H

Was really the only issue, though, that prevented me from thinking the delegate.

F

H

Is better than merging Academy? Oh well, I! Don't his go to delegate so.

K

We have some very specific use cases around merging that mostly feed in from automated provisioning of facets of the virtual hosting system. One is kubernetes ingress itself right, because the way it's defined right, so we get user pressure from that and by implication, all the tooling that was built around kubernetes ingress as a mirja Balad 'l, I, let's encrypt certificate provisioning systems and then, thirdly, the expectation that we had set with customers previously, what Rev, ruling or and merging, which is now cause some customers to feel like?

K

We have left them high and dry, and they have CD systems in place that are producing artifacts that are emerging and they're wondering why we can't merge them for them.

H

So kind of handle all the cases in my mind, except with permission Eman other than this trying to separate out the permission, the ownership of the past kind of high-level, that's.

B

Important I mean commissioning and not making mistakes, and it's not only that you have technician, but you should not accidentally take over someone else's domain and working configuration because you put right.

K

So we have this dialogue going on right, I mean Brian, wrote up the the permissioning tackling model around hosts which were having active discussion around. We could extend that to also define things around URL name spacing, and then we wouldn't need delegation to manage permissioning at this I think there's still an open debate about how exactly we want to do that, but I think we're able to start enough television yeah.

E

We add a few minutes: yeah.

J

So Louis another thing would be so I think if there are more complicated cases where you want to do routing based on headers. Yes,.

L

J

The current implicit or convention based merging does not help you that's. Why I think these two can exist in conjunction rather and separately, so.

K

I tend to agree, I just want to get to the permissioning discussion, and it's entirely before we make a final decision. That's all and.

B

I want to raise again the point of scalability and performance, because not everything that looks nice on paper can be scaled and if we get in too far I mean again hey that's, maybe we can deal with time in efficient way. Reigate is certainly we cannot and arbitrary. Much is done. We cannot so that's a thing. We need to scale yeah.

J

I agree, I think when you are also doing delegation in that we can come up with some constraints, saying what we can merge and what we cannot yeah. That still applies.

B

In what we give up then- and let's leave the one weekend with more efficiently for future aggression and see if a user really needs them, then we try to find a way to merge them efficiently, but.

E

I think the one that's outlined in the issue is a very reasonable model for us to do for the first implementation, ponton regexes merged the things that we know. We can merge sensibly sensibly and call today that.

A

You'd like us to start with some incremental approach.

H

E

A

To start with, with a new implementation,.

A

C

The pudding would be the reduced number of like user complaints and mails and like, and if you find that I think drops to almost zero, then be more correct. Otherwise, it's like you know. If the more fancier things we do and if it just opens up a completely new kind of worms, it is gonna, be nightmare. So it's one and then.

K

So we can make some changes to how the merging occurs over time based on user feedback. Well,.

E

That's why it's so that's why I think it should be a tool like we should build a tool that does merging. We can implement different merging strategies.

K

Very different.

E

Expectations about that right.

K

A

And also like we're.

K

A

Be because I hope we will not merge the issue and stay because the CRD, not the issue, though well. We know.

I

A

This and just like how that's gonna be operational status again. We're not gonna like merge. The I.

L

Want to be in the.

B

L

B

If you the CIC DC's, then you would have us that saying that much that you.

K

Conserve use a tool, that's the responsibility, use it I think. The issue we're facing right now is that some people can't yeah.

B

K

B

Of course, those those service we need, we need to meet the needs that put into I think.

K

If the first phase is to implement the basic merge, whatever we agree on and.

B

The code would be using a.

C

Very careful in terms of like how the merging is happening. I means you doing any form of merging within pilot I. Don't want to take any specific like arbitrations, that we end up having to change, and then it becomes a backward compatibility nightmare. So instead, if we do the dumbass merging and we still actually solves it and then when, when we have galley in place- and we have a dependency injection framework, we can have arbitrary preprocessors that actually pulls in all the configurations and decides to sort on.

K

We do have a get-out-of-jail right. If we pick an arbitrary merge algorithm we have, we think, is reasonable and we've thrown some research into what we think is reasonable. Now we can always tell people who are still having trouble to go back to single resource right. They have an out showing it yeah and later on, when we figure out her acting model, and we want to have delegation if they want some different separation of concerns right. We can push them down that path, so I'm not overly concerned with implementing a merge in line.

K

It's gonna be part of our API, but when it is part of our API, we could but people can get a different behavior if they choose a different means of interacting with the API right to make that choice. Yeah.

E

Well, attackers Farren's point: that's why I want to be careful of what we put in pilot a tool, easy to change and doesn't require, doesn't mean that pilot implements some behavior, that we know that we will want to change when we add this behavior to pilot that will become part of our API. That is a publicly exposed feature. People will depend on specific orderings and we will break them when we change it, and so then we're in the exact same situation that we are now where users are complaining.

E

They won't use a tool to fix it, even though that would.

E

K

We pick something that we.

A

Write it in another tool.

B

I mean we have some.

B

You can use whatever you want.

A

A

B

A

Is somehow like yeah they can anybody can do that right now? Yes,.

L

A

Right but the question is: what do we support natively like in pilot right? We will have that algorithm and we said the exact match and so on.

F

Right and it'll be.

A

In pilot that, in the memory and I guess, we will have some, maybe Co, CTL command or something to be able to actually to dump the result. Even if it's not saved back in CRD, we need to be able to say your your rules.

J

A

To look at all the resources, let me just display the chat, D.

E

Yes, if you're using a tool.

A

E

A

E

Services that will must necessarily see all of resources of your market yeah.

A

Absolutely yeah.

E

My point is: I, don't want things to land in Pilot before we that are part of the API before we are confident that that's.

B

Too many things to handle, but at least it was the most conservative ones that we discussed don't seem unreasonable, so they are provably vanities ingress. So it's not really well.

E

J

So I think one concern that Zak is bringing up is correct, like once you do a convention based merging, even though we are not changing the API going forward, people will expect the same behavior.

K

It is clearly part of the API right. We will have a document, merge, behavior and say that that's the contract times, if you don't like that contract go, do something.

C

Else, but we could keep the convention very, very small by simplest yeah. You can need the services from two different virtual service and we'll push the way you catch all prefix. To the end, that's a we don't actually move the exactly.

K

I think we do have a difference of opinion. There I think leaving more refined matches to the beginning, yeah.

F

K

A better approach, it's a more stable, more durable approach, and that has been proven out in the industry. Your skill.

H

K

H

Also, a notation that you set to pick the pick. The algorithm.

K

They do that in the future, but I don't think here.

A

Like not make the most complicated things first, like just you know, offers something and no I mean.

C

Reason I'm saying is that you end up picking the mostly exact matches on top right. If somebody has a play exact match for for a perfect match for /vu, and they say like no, my service came first and I want to have /o prefix be matched right. We.

K

Tell you to go and great a unified resource or use delegations rooms for.

B

Your students at.

K

Some point: we have to document the kick-out right.

B

With the exact first prefix second, so.

A

Right now the scoober net is the prefix matching reg X I. Think if.

K

A

K

The nginx English controller it tries to do a specificity ordering like what I put in the issue is actually what the nginx English controller does for kubernetes. Yes,.

A

Because people might actually expect the same behavior, they have seen. There's.

K

A reason why, okay, this might be reason why they picked.

L

K

M

Is there a general rule for what kind of logic belongs in pilot versus what logic belongs in galley like? Is there a principle we're trying to follow when we decide that this stuff goes into pilot versus.

K

Code, if it's the main.

M

Specific to networking it belongs in pilot.

K

Yeah, if it's tied to the a clean model or resource ownership models, it probably belongs in.

F

A

Extrapolate and you have a hundred components and each of them brings their own domain or specific checks and validation to galaga. It will become like okay, physical.

K

Exercise here is that pilot will take configuration from multiple different systems right and we'll have to be responsible for producing some reasonable representation of the network from that merged view right. So we cannot Gally can't pick up concerns that Pilate would have to take to be able to do that to meet that requirement right.

M

But if the only plug points we have are in Gally like if you want to customize the merging, are we going to make that an extension point inside of Pilate? Remember pilot gets the API right. You can decide to merge before you.

K

Call mCP right: okay right! So that's gather you are not going, but it's still before Pilate I think.

A

Jason wanted to say something: yeah.

N

So, regarding what the Long's of Galilee I think, ultimately it's it's going to be more than just things around, acting because at some point, as we start to separate what components consume from what consumers right so having a notion of service config your workload to think there's going to be need to be some domain-specific transformations that are performed by galley before delivering to the components. I'm, not saying this merging thing belongs in that category. I just wanted to clarify that that it's not such cut the dry yet yeah.

K

I mean I think we're gonna, we're gonna, be iterating on this right, yeah, there's a tension there between rich or more unified resource semantics like service service config, which is a unification of many many concerns and merge across multiple configuration domains. Okay, that's that's something we have to work on well,.

C

Solve everything.

K

C

You need to keep in mind that the Hessian D routes- it's not just route, it has timeouts, it has fault injections, it has mirroring and redirects so and it has caused policy in append and remove headers and all our stuff. So the I mean yeah I guess we could just pivot everything based on just the mask of match condition alone. But then we have to do the same thing for TCP as well.

K

Sorry, what was that, in the context of sriram merging or in the mind except merging, so I guess yeah? We have to define merging rules for TCP WebSocket every other protocol right. The merging role could be order rate times time order, I think for TCP. We can have something similar in terms of prefix specificity.

K

Right TCP is a hierarchical namespace. Just like your eyes, you know exact tcp/ip matches come before psycho rain matches, more specific site arrange matches come before that specific site, arranged mattress, etcetera.

C

E

Yet again, I'm sorry what exactly I.

C

Mean sni has the same semantics I, like my.

K

Hierarchical names.

C

B

I'd years I mean so difficult to allocate IP.

F

B

And the idea is very: we.

K

Will use this would say exactly match would come before slapping.

B

K

Maybe and I mean they really are hierarchical right. So at least we know wider ranges are difficult to allocate it's not difficult to understand. They represent the hierarchy so.

B

If the world is moving through well, you know them and which.

K

Is why we don't agree mostly expect people not right side or Exantus.

B

K

We should know that was too much what we need to. We need to lay it out later, yeah, it's great and.

E

Like it mad this- and this is yet more reason- I want it to be a separate tool and not in pilot.

F

E

We had every protocol, we have to do this exercise and in CDP it's relatively straightforward other protocols. That is not necessarily true well so.

K

Today, anything that's tcp-based is probably easier than HTTP, because the matches are purely based on a hierarchical. We don't have any fixed matching at all. I can't reg X match an IP today, so it's actually easier. Websocket should be relatively straightforward. It's if we had a protocol that was purely key value based in its identifier, x' right, then we beat we'd be in a world of hurt right because.

E

We don't like these things to like Afghan. We have some notion of topics and that kind of thing, but like hey, we're, not that's, really guaranteed that there's these yeah.

K

People should go, create separate virtual services for Kafka by.

C

The well potatoes, though,.

D

C

Because I actually teepee match is not just path. It has like path URI at all,.

K

Right, if you want, then, if you want exact ordering, but if you can't guarantee that you have purchased your name matches, you got a move into a single resource or delegate model I.

C

Don't know that's fine, but I'm, saying they're merging. How do we do that? Because when we have multiple matches that be I mean they.

K

Don't times 9 bordering sriram, yeah.

J

As long as I think, we documented well explain what our convention is. It should work fine, even though.

C

Because the match is not it's not.

K

That's why we have time stamp is the catch-all is like at some point we're gonna be about. You know.

C

Well, no I'm, sorry, even with all the time stand, if you have like you've, three virtual services- and one of them has like four plus this and something else has like no port, and this all like they have I mean we have to do it's a multi-dimensional match that we have to do in pilot for every kind. We compute the map of virtual server, and then we actually serve it up to all the sidecar son. Okay I mean yeah, I. Think we'll know when we implemented this. So that's that was my or.

E

In this year again, why don't want to implement a pilot first, because we don't know- and so we actually get in to implement it, because he is correct right. We do have these complex match conditions so like we can handle paths relatively easily. We all agree on that. We can't handle the port matching the the in those things and make a there's, not an obvious ordering there.

E

A

Have to do the same caching, that pilot does your.

C

A

Will be end up being fairly complicated.

C

Gene stratum and tool is like people can change the tool before they redeploy the same yeah.

E

The goal is to be able to iterate more quickly and not have something that is part: a P I contract that we release and then is there and.

J

It always gives alright.

K

What we can do right when we have multi dimensional map to say you have path and port right it is. We say that the ordering for port is undefined right, so we very strict view restricted right now we can go and fix that later. If the customer raises an issue inside hey, when you merge, you should give precedence to something that has path and poor or something that only has path.

C

K

Because it's more specific right but I'm not sure.

E

What are we gonna do when that breaks? The other customer absolute says: hey you, you should do it this way, and now you changed it. You put the port in perfectly you.

C

E

C

Have Nick Bobby the.

K

Only question here is.

E

How far we lose right now, right and so I think that we can choose to not lose it all by punting to tooling, until the community gets consensus on what is reasonable.

K

Is never gonna get consensus, design, there's.

E

K

No way other than this meeting, where we're gonna pull their company into a room and get them to agree. But.

B

C

It should t be match condition. We have ports, we have headers, we have gateways, we have sauce labels, a URI, Authority method and scheme. Now you have to make an ordering based on, like you know, given to match conditions. How would you order them? I.

B

Think what we agreed is that that we first take the rules that have only exact matches and we are plugging them, no matter what, if they have something else, they are not I mean who has only an exact match. It's a quite first define.

C

B

B

Which is exactly what ingress ingress? That's, basically it it.

C

Is still an exact match to have port? We are an gateways and source labels.

K

C

We need to resolve.

K

This among ourselves yeah. We need to come to agreement if we're gonna implement this in pilot, what the matching precedence is we're not gonna do it here is enough time, so.

A

K

Collectively, invest in like trying to come up with a very precise set and see if we're happy with it very small.

F

User, it might be.

K

More complicated than we necessary want because some people are concerned about the fact that I'm concerned about specificity and people are concerned that a fee incremental ii add stuff to it later will break people. So it might be better to be more sophisticated now and be done as.

B

Long as it can be in the medina scalable.

A

So what do I suggest is we do we? We have the document, we have the issue so where, where should this discussion take.

C

On the issue on the issue.

A

Okay, so let's debate on the issues, and so is the document still so.

K

Shri Rama II, you and Zack raised a set of concerns by ordering I think are followed, so maybe just dump. The first thing to do is dump in what your concerns are with the the proposal, which doesn't take the other multi-dimensional aspects into account all right and then we can go through and propose something reasonable.

E

What's in the thing now, like I, think.

K

It's a reasonable but I think there might be some gaps that we need least to think about and that we should cover and.

B

For every match that we defined calm, which will be implemented in again weeks, oh one operations, I.

E

Mean in constant, we can always do post ordering stuff of the actual envoy config. If we're really concerned about ordering the actual match tree like we can always dude that doesn't have tablets.

B

B

A

I think I think we will not be able like to close completely on this issue. That's why we have to take it offline and that because we are already 10 minutes over. So thank you very much very good discussion and I think we did get to some agreement like on many of the topics we discussed today right. So it's just a matter of fine-tuning the details, maybe like the algorithm and everything. Okay, all right. Thank you. Bye thanks.