Kubernetes WG Policy, 27 Jun 2022

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: WG Network Policy API Bi-Weekly Meeting For 20220627

Description

WG Network Policy API Bi-Weekly Meeting For 20220627

A

Awesome hello: everybody today is june 27th 2022. This is a meeting of the sig network policy, subgroup, 2 uh sig network and today we're going to be discussing a lot about multi-cluster network policy. Just as a note, this is a kind of off date for us because we're working around some public holidays that have been cancelling our meetings that are bi-weekly so we're on an off week, um but yeah not too much to talk about today, except for uh multi-cluster network policy, so I will hand the floor over to sanjeev.

A

If you want to kind of kick that off, we've been talking about this in sig network a lot and we decided to move the conversation over to here.

B

Right. Excuse me: okay,.

C

Let me just share here.

A

Yeah, I'm giving you co-hosts. You should be good to go now.

C

This should be extra large, I suppose, okay, so um this is, uh you know more brainstorming on multi-cluster network policy, sorry with chrome sharing. Sometimes it just jumps slides, so you might have to bear with that.

C

Okay, you just move the cursor and it seems to jump slides so just watch out for that. So this is a topic which kind of straddles, sig networks, network policy and sig multi cluster, and we it so happens. We have representatives from all those and all of we've had at least one discussion in each of these groups. So far.

C

C

Okay, um I thought I'd just mention one thing right off the bat, which is that we don't exactly have agreed upon multi-cluster network model. Yet to the best of my understanding, correct me. If I'm missing something so one basic question would be do we first need to even define a reference model, yeah didn't.

B

You had a hand yeah. No, I wanted to agree with you. um We haven't really pinned that down. um There was a pr open against the caps and there was some feedback. I uh don't think we closed the pr or merged it or got all the feedback together. So um I know that we need to block this on that, but I do think it is definitely relevant and we definitely do need to describe uh what we expect to happen in a multi-network model.

C

Okay yeah, so we can continue in parallel, while recognizing that there's we may, we need to continue to level set each other on the exact model for which we're defining policy, and I think we need to continue to get more end user requirements.

C

um If there's some way of querying business, people about what the field is asking for in terms of multi-cluster networking and policy, is there a way that our working group can has representatives from the product marketing side that can feed in more requirements?.

C

Has that been done in any prior activity, because we would be somewhat speculating on what we think are reasonable requirements.

B

Yeah, we don't really have a I mean. Kubernetes cncf doesn't really have a product marketing wing. um What we can do is go to our respective field teams, which I assume you've already done, and we've certainly got some context from ours, but we can also go out to the broader community in the form of surveys or questionnaires which we've done in the past to you know moderately success, okay, yeah! Maybe that's a good idea. Maybe we'll uh initiate a small community poll on requirements.

B

It certainly had a material impact on the shape and form of gateway, for example. So it doesn't not work right.

C

Okay point taken, I think, maybe we'll initiate some community poll in in the coming uh day or two, so just two levels that everybody I'm sure most people know this, but just to level set at a very high level, there's basically two models: there's either a flat or a single network model. Sometimes it's called, which means that all the clusters, um every part in every cluster, can talk to every other part in every other cluster without going through any kind of matting operation, so in some sense they're all in a consistent writing routing domain.

C

So that's the flat or single network model and a multi-network model is where ips are not necessarily unique across the set of clusters we're talking about. So there is an adding involved, which is typically some kind of egress gateway or eagle snack function, combined with possibly an ingressnet function on the other cluster right. So there are these two models. One is you've got a bunch of clusters that can all freely communicate to each other.

C

All parts can communicate without netting and the other one is in which they have overlapping ips. So they need to have natting gateways and there could be other variations.

C

You could even have multiple such groups of clusters or cluster sets, and do we want policy to work across that, like a kind of a multi-mesh, or do we want policy to only work within a single mesh? Slash single cluster set.

C

One could potentially conceive of no models that need not be aligned with the mcs api which the sig multi-cluster team has been defining or we could sort of ensure we could maybe decide that. No, that is, that is the reference model which we want to operate within and then there's also control plane models which define the level of global information sharing.

C

You know like, should we assume for certain models or for all models that they have access to resource information from other clusters, so things like pod and service labels are noble to all clusters, and that could be whether how it's implemented could depend upon whether there's an agent that has cube config permissions to all the other clusters or there could be a management cluster.

C

But should we even assume that, because some of these policy options could depend upon whether you know about labels in other clusters and the istio community has done some a reasonable amount of work here, actually quite a bit more than the kubernetes community in terms of defining various kinds of multi-cluster models uh which are based along these lines? So there's a single network versus multi-network there's you know single primary versus remote primary and all these others. So can we should we maybe adopt those kinds of models?

C

So all of this relates to you know the modeling anyway,.

C

The initial draft of this document we tried to straddle both the single network and the multi-network model, but that comes with a cost. Then sometimes you have to work with the lowest common denominator, because you want to have a policy that works on both these models, whether you're netting or not.

C

Some other proprietary implementations, one example being the cilia mesh. They only address some models like, for example. They assume this only works for flat network models, so they can assume global knowledge sharing, they can assume part ip uniqueness and they can hence be more fine-grained.

C

So we have to decide whether we want to do that or we want to sort of do that as the first phase of an eventual model that covers all these cases, so um not sure how much time to spend on this network modeling thing any quick cards here before we talk about policy. I have a couple of pictures here as well. I might as well share those um so this is actually something that laura lorenz and I have been working on as part of a multi-cluster service.

C

Networking uh talk that we've been doing uh both on youtube and and other places, so we've kind of borrowed those pictures from there. And yes, this thing keeps bouncing okay again just for a quick reference. So in most of these projects, there's the notion of a cluster set or a mesh which is sort of.

C

Sorry, I should not move my mouse, I suppose, which is sort of the a multi-cluster domain. So on the left, you have mesh a which is a collection of clusters within that mesh. Slash cluster set, there's actually multiple networks right. So these networks are, as we defined earlier regions where within a network you don't need to not, and when you go in and out of a network you need to not so you see that there is the orange network and the green network.

C

There's two clusters, each in orange and green, so clusters in an orange network, don't need a gateway when talking to each other, but they do need in a gateway when talking to clusters in network into but they're all part of the same mesh. So this is a multi-network mesh right, slash cluster set, and they also sometimes need to talk to other meshes. So our cluster sets. So that's what b and c are they're effectively external clusters, measures and and and so on, and so terms like network and gateway cluster set mesh and multi mesh.

C

You know we should keep in mind and again when we're defining policy. Do we need it to define across all of these topologies and scenarios or, for starters, just focus within a mesh and then talk later about multi-mesh?

C

So any quick thoughts on this model or the overall architecture.

D

So I I had a question about a previous slide.

D

E

Yeah, it's a bit more basic, I'm still trying to catch up, so I mean we. I know we call it. Network policy is the is the goal to provide a security policy irrespective of the network model? I mean what I'm trying to get to is that, should it matter whether there is ipam overlap, I mean I'll be trying to say that part. One is able to talk to part two with respect to how they are able to communicate with each other, or is that communication path important for defining security policies?

E

I'm trying to see is there a model where nat is not it doesn't matter? If there's not or not, should we think of such a model where you know it doesn't matter.

C

So that is exactly the topic that is being listed here, which is that one could define a policy model that works with or without netting, and that is actually what we have assumed in the slides that we will be going into in the next few slides, but because you're trying to work the same api has to work in all these models. It.

C

Actually you have to sometimes go into the lowest common denominator, because you have to satisfy all the models, whereas it is possible to have certain kinds of api objects which only work in one particular model, in which case they can actually be more fine-grained tim. You want to add.

B

Something so I don't think we should build anything that can't possibly work in multiple models that that seems like a premature dead end, um but I also don't think we need to assume the lowest common denominator. What we need to find is that middle ground, where we describe what we really want in the richest possible terms that we believe is practically implementable in multiple ways um and that's a that's a mouthful. You should print on a t-shirt, but um uh what I, what I mean is like sanji.

B

We talked about this a little bit a couple weeks ago, right like if you were to do service account. I can imagine three or four different ways that you would transport service account information around, whether it's through control, plane or through data plane, um and to me that satisfies the sort of plausibly implementable, at least for now, until we go ahead and try to implement them right.

C

Yeah so question: uh does that answer your question.

E

I mean yeah yeah, I think yeah. I think it does. I I was just trying to say that um the I think it does answer my question. Yeah.

C

Right so we are trying to address all models, uh but they might be a bit of a middle ground because we don't want to always go to lowest common denominator. There's a little bit of grain gray area here and that's part of what we need to discuss.

C

Okay. But this is laying out the fact that there are proprietary solutions that only address some models and I've. Given one example here this silly multi-cluster mesh that assumes it's a flat network only model, and then it can actually assume global sharing of labels, and things like that which we cannot assume if there's an acting involved.

B

Hold on a second, though, um I'm not sure I buy the. I agree with the predicate, I'm not sure I agree with the conclusion um abstractly there could be a control plane that understands the different clusters and figures out how to translate from one to another at the edge they're not doing that, but their api could do that right. Nothing about the way, I'm not super familiar with their network policy api, but the way I understood it was nothing about the api says it. It can't be done that way. It's just not done that way.

B

C

I would ventu and I haven't- I need to dig into it more as well, but just one point I'd add is that if the api, for example, assumes that I can uniquely identify remote parts with with some label um that may not work based on whether pod ips are truly globally unique or not, or even labels are globally unique or not.

B

But like psyllium in particular, since we're picking on them, they they do something like they have an ipsec layer below and uh in some of the extra space in the headers there they're embedding an identifier which they can use to identify, not a unique pod, but a unique set of pods that satisfies the policy constraint right. So in theory, the other end of that ipsec layer could either have a unified um id space across all the networks or could map ids from one to another at the edges right right. So.

C

And a consequent implication of that is also is things like?

C

Are we assuming cni's running on these clusters.

C

That have a certain level of like, for example, question one would be. We want it to work. If each of the clusters is running a different cni right, we don't want the same. We don't want to have a requirement that all the clusters are running, the exactly the same cni plug-in.

C

So there are so the tunneling and things like that have implications for the capabilities of the cni.

B

And I think we need to be realistic, though, like in the past, we tried to uh make this assertion that you know.

B

Network policy was a completely distinct thing from the way services are implemented, which was a completely distinct thing, from the way that cni was implemented and that these things were composable and that they would just you can mix and match them with off-the-shelf parts from different vendors and the reality is it doesn't work very well right, like each vendor makes assumptions about what they're doing at the cni level in order to implement their policies right, like tigera names, their interfaces in a certain way and then writes their rules based on those interfaces, and they maintain some ip sets to make the policy layer, work and psyllium.

B

You know installs their ebpf programs, and so I don't think it's reasonable to say everything must work, mix and match with any combination of things it's great if they do. But I think that there is some responsibility in assembling a set of clusters that you're assembling components that actually do work together and that very often that will mean they run the same cni or it will mean that they stick something at the edge that converts from one language to another language right.

C

So these are all essentially aspects of a certain model. One would need to agree upon for multi-cluster policy and for this initial draft we kind of wanted to make it minimal assumptions. So there are no assumptions that the cni is the same or even has similar capability across two different clusters in the same multi-cluster group, nor that there is any tagging and we will come to that and also uh it addresses both the flat model and the multi-network model.

C

But it does limit the kinds of things you can do with policy which might be okay again. We if we get more end user requirements, we'll be able to confirm that.

F

I think what tim is saying, though, is that that's not making no assumptions, that's making an extremely strong assumption that that you know it's possible to mix and match cni plug-ins in a multi-cluster environment and and the no assumptions variation is they're all running the same plugin.

B

So I I'd like to express sort of the middle ground, which is um if we can come up with something that works well and allows either mix and match arbitrarily or some you know thoughtful mix and match great that's bonus points, but I'm not going to take that as a predicate for accepting a proposal.

B

If the only proposal we can come up with that is reasonable and satisfies all the other requirements is they have to be running the same network plug-in or they have to be running an adapter, I'm okay, with that. It's my second choice, but I'm okay with it.

B

C

We'll have to then agree on which are the must-have requirements which might determine the eventual list of restrictions.

C

Okay, let's keep going. I think these are all valid points and hopefully, as we iterate over, it it'll become a little bit clear. So again, this was sort of the architecture picture. For now do we tentatively agree that we're talking about policy within one cluster set mesh? We're not really talking about multi-mesh kinds of policies like this picture is alluding to uh or.

B

Maybe we want to come back to that later, but I'm I'm happy to come back to it, but my initial inclination is, we have enough to chew on by just solving it within a within a cluster set, um and the sorts of policies that you would typically want to express between cluster sets feels like an order of magnitude more um intentional, more cautious than you would do within right right. At least this is the way we've historically thought about it. Right.

C

Again, just we're just laying out the fact that this there's in general, an entire enterprise is never just one cluster set, so be aware of that. But for now we'll concentrate on one plus to say- and this is just another way of representing that a more generic picture, which is that you can have a cluster set of four clusters.

C

There are all these gateways again. We don't need to get into that. I kind of just borrowed this picture from our other presentation, um but then there are certain aspects of which are specific to like the mcs api, um whether we want to tie this directly to the mcc api or not, is also a factor.

C

uh So in this case, for example, cluster one and cluster two can communicate without nat, but cluster two cannot communicate with cluster three without not that's why they have these mcs gateways and should the policy work between sources and destinations, no matter whether they have to go through the gateway or not.

C

For now we are attempting to make it work across both, but maybe we'll have to find some kind of middle ground somewhere.

B

I I do feel like: if we can't solve this, then we haven't.

C

Really solved it, so you want it to work in all all all kinds of communication within this kind of a genetic picture like.

B

Within within a single sort of authority domain right, which we typically call the cluster set.

C

Okay, so it should work yeah, so it should work both with some subset of clusters sharing a network, some other subset of sharing another network, possibly some adding gateways to connect those subsets.

B

C

This is picking.

B

This is picking the most painful possible model, but that's the one we need to be aiming at. I mean somebody tell me if they disagree strongly, but I feel like this is going to happen.

A

I think this is this already.

B

A

Yeah, this is the harder problem to solve here right I mean it just constrains what set of features you can expose in such an api right, and I think you're going to get to that in a minute. Sanji, like you know where is enforced, and can you still identify the sender et cetera, et cetera,.

F

I guess I feel like at a certain point, you're going to have as many issues with wanting to set policy on connections going into non-kubernetes cluster non-kubernetes areas, whatever, as you will going into other clusters, so there's going to need to be some sort of you know api for, for that's not multi-cluster anyway.

A

That's a good point: a a new ip block, a smarter ip block is that kind of what you're saying dan.

F

Yeah I mean we're going to need to be able to set policy for talking to things that aren't kubernetes clusters, so maybe we don't need to solve every problem you could have with the kubernetes cluster like at a certain point. If it gets too complicated, you could just treat the other cluster like it was.

C

B

Network yeah, I want to be real careful that we don't um literally try to reinvent like istio here right like there. There are problems that we can say are are above what kubernetes is trying to solve. um We just need to figure out where that edge is and honestly, like. I'm, not super confident. I know where it is, but I feel like we'll know once we've crossed it and then we'll have to go back and figure out where it was.

C

Okay, so I'll skip this slide, but there's a bunch of text here about sort of the kinds of requirements um you can look at it later. Let's just go on and there's put some notes on the kinds of factors we need to think about like how much dependent on the mcs api um the trust model across clusters right the the uh and sort of the threat model that we are actually trying to protect.

C

Sorry um and um a few more other things there. Okay. So let's look at some pictures now about the use cases. So, first you again, we've gone through this in a couple of previous, um so I'll, try to maybe run through it a little fast but feel free to stop me, okay, so the first use case is um for now all these use cases we are assuming. We are trying to operate within the mcs api to the extent possible and we are trying to make it work both in the flat network and the multi-network model.

C

Okay, so that's! If, if we were to relax any of those assumptions, there could be more use cases we're not going into those for now. Okay, this is something that should work in both single network and flat network and multi-network and without changing the mcs.

C

So this one is, um I have two parts here: they have imported remote service foo from another cluster, but on cluster a we want to say part p1 gets to use this imported service, but p2 does not get to use the supported service today there is no direct way of doing that other than um having a cider based policy, which also doesn't quite work because, ideally um well. Cider based policies are.

C

At the pod level, right I mean one could sort of have an egress policy here which treats the cluster set ip of the remote as the destination ip. However, that implementation is likely to break in many cases, because in most cases, egress policies are applied in the cni after the service load. Balancing has already been done, so it's applied against from source pod to destination pod, whereas if we wanted to work in multi-network, we don't know the destination port ips, so we have only the cluster set ip to go by.

C

So so that's why this? If we want to address this use case, which seems like from our early discussions with some solutions- architects, they felt like this was a reasonable use case. Then this would require the new multi-cluster policy. Okay, you have.

G

A question sanji yeah go ahead yeah. What does it mean to not be able to access the service, given that the service is really just a loose abstraction over a group of endpoints right? Can they still access the underlying endpoints directly if they go? Look at the endpoint slice, for example, or what happens if there's multiple services, exposing a single workload.

C

um Maybe I didn't quite catch it, but I guess in in there should be no path for if, if, if the sorry this keeps bouncing, um if the admin of cluster a wants to absolutely ensure that part p1, so okay, let me step back so customary has imported this service x.

C

Clustering has explicitly imported this service. Okay, sorry, I'm!

C

Let me slow down here um in the current mcs model, imports are automatic, so anything that is exported is automatically imported by all other clusters and, furthermore, in the importing cluster every pod has access to every imported service.

C

This use case is to selectively prevent that.

G

Right, I guess my concern: is it's not even really a multi-cluster concern like if you were to say in single cluster network policy that you were going to add a policy for a service rather than for specific pods? Then you get into problematic behavior right because, for example, I don't actually have to send through a service ip to reach it. I can go and access the the uh pod. I use directly right. Well, you would have you.

A

G

Like you can say that you like do the indirection, but then what happens? If I have two services backing one pod and one's allowed and one's denied? What do you do right? This is an issue we hit early on in ego uh many years ago and kind of caused us to completely redo our api, because we initially had our authorization policy at a service level and that caused a number of issues like this. So we moved it to a workload level.

B

And sorry, if I can just jump on the tail of that um when we were first codifying network policy, we considered, should we make it service, oriented or selector oriented, and it was that same argument that moved us to not try to make it service oriented, and I totally get that there's an attractiveness to wanting to make it service oriented.

B

It is sort of more ergonomic um and I'm not even inherently against it, but it does have some real drawbacks that if we want to do if we want to make m.c serve mc policy based around services, we should probably make single cluster services policy first and then define them. In the same terms, sorry, I jumped the line. I think.

F

In the multi-cluster case, there aren't aren't we only exposing services between clusters and not individual bots correct so.

B

F

Necessarily like in.

B

A flat network model, the ips are there if I can find them, does this mean that the policy doesn't govern the access? If I managed to circumvent the policy like if I did a dns lookup before the policy- and I just happen to have the ip address in mind, then I can use it.

A

But there's no api that specs out communication between pods and different clusters, even if they're in the same cluster set like so yes in a flat network pod a would be able to and cluster one will be able to talk pod being cluster two, but isn't that out of scope of creating you know of this mcs network policy.

B

Sort of api I mean, if we're defining it in terms of services, then we can define whatever semantics we want to define and argue about whether those semantics are correct right. We don't have a services, oriented network policy so we're we are all automatically being uh sidetracked into arguing about that. Instead of the multicultural problem right so.

C

If we look at what is the functionality we want to achieve, we want pod, we want there. We want part p1 to not be able to talk either to the service whip or directly to the pod whips if it happened to be a flat network and part p, once somehow figured out the back end pod ip. Both those cases have to be blocked right. So this is just our starting point here.

C

What we need to first degree is whether this functionality needs needs to be addressed, and if we agree that the functionality needs to be addressed, then it needs to cover both the cases where you're talking to the service whip, which is what you would normally do in an mcs environment or whether it happens to be a flat network, and you have somehow backdoor figured out the back and forth ip and are trying to reach the port ip directly. Both those cases need to be blocked. If, if we agree that this functionality needs to be met,.

B

Regardless of which api, but I'm not actually sure I mean you're you're asserting, but what andrew or dan just said, I forget sorry, I paiged out which one of you said it one of you two said it uh was actually the opposite of that and like I'm, not sure that that's a um a given in a service oriented policy model.

A

And rewinding to like, if we did this in network policy, I I maybe I'm not even and tim had some more background on this when they first designed it. But if we had a service, selector or network policy like if you select a service, it means a workload, cannot reach that service's vip, so call it the dns name whatever. um If you want to stop both a pod talking to a service whip and a pod talking to those other backend pods, then in my mind you need to have two policies. You don't have one like.

A

You have a normal network policy blocking pod, a talking to pod b. You have another network policy blocking pod, a talking to the service bit. No, that's crazy!.

B

Why I agree? No, that's madness.

F

Policy should be defined on services and implemented on pods.

C

Fair enough, well, I agree with the first part, not necessarily the second part, so I mean one way we could say is that if a complete implementation of this api would require you to block traffic sent to the service whip or somehow sent to the back end, both it could be one api in the control plane, but the data plane has to block all paths, so the implementation could be a combination of blocking traffic to the service whip and blocking traffic to the back and forward whips.

C

But the api and the control plane looks just just like one thing like do not let pod p1 talk to imported service, foo right.

B

Well and uh let's not forget now, as soon as we bring services into the mix, I see dan winship rolling his eyes, even though his video's off um as soon as we put services in the mix, we have to talk about node ports, external ips, load, balancer, ips and all the other ways you can get into a service, and I'm not saying these are unsolvable right, I'm just saying they're not solved.

B

I I like what you're saying with the idea of one one api and the implementation has to decide whether that means service, vip and pod or just service fit, because only the implementation really knows that.

B

But I want to be careful that we don't waste the last 20 minutes that we've got arguing about service network policy uh here like if, if the goal is to actually make progress, I think we should at least put on the on the agenda. um The other models that aren't service oriented for exploration.

C

Yeah one thing to remember is that, because we again going back to the fact that we wanted this to work both on flat network and non-flat network models in a non-flat network model, the only thing that you have is the service web or the service dns name.

C

So you would either need to have a multi-cluster service policy or you know not at all and then just say we're only going to do it for flat network only and then it's just going to be.

B

I mean already, I see a bump there, which is we have customers. I know lots of people who are doing this sort of hybrid island where the vms all share uh network, but the pods in each cluster, don't, and so in theory, it's possible for a pod to egress its cluster and hit a node port on a different cluster right, which means you, then have to talk about multi-cluster network policy at that level or decide not to right. Because again, we don't have a spec for this because we have inspected.

B

um But it is a case.

C

That we have to consider yeah and then, as far as the other service types we will come back to that I didn't want to complicate. You know all of the note port and the service type load balance, and all that, let's for now think of these just as cluster set ips or cluster ips and we'll come back to node port as well. But for the moment to keep our discussion simple everything is every service is a cluster ip service and some of them are multi-cluster cluster ip services. So let's just kind of focus on that.

C

For that we'll come back to notepod.

B

Can I ask one one clarifying question: um do you envision this api governing exclusively multi-cluster access, or do you envision it covering in cluster and cross cluster yeah, so that is.

C

That is another thread that okay, because sig multi-cluster chose to make mcs api as opt-in and services are differentiated from multi-cluster services. If we follow that same architectural, I want to say paradigm, even though I don't like the word paradigm.

C

um Services are anything you do to a service does not automatically imply to a multi-cluster service, because you have to opt in specifically for a multi-cluster service, but we could choose not to follow that model, in which case a service is equivalent to a multi-cluster service which also, by the way happens to be what the hto control plane does and his two control plane just joins uh two services in two different clusters, which have the same name. Space and same it implicitly treats.

B

It as a multi-cluster service, speaking pedantically mcs, doesn't have a different concept of an evident externals or of a multi-cluster service. It has a concept of publishing a service to other clusters. It is still the same service.

B

I do believe papers, maybe.

C

There's some ambiguity in the definition, because I know of implementations which explicitly treat the single cluster service differently from multi-class service, for example. Well, we know that the application has to specifically do a dns query with a different fqdn to find out the the multi-cluster flavor of that service and there's a different fqdn.

B

For the uni cluster version of that same service, you're right, you're, right on that regard, but otherwise all of the semantics carry forward. Anyway, I don't wanna, I don't wanna rat hole on this like I know you have more preparedness, so these are.

C

All valuable threads that that need to be chased down at some point we're just going to keep things simple, so yeah, but one last point about this use case based on discussions with a couple of solutions architect. They they identified this use case as probably the most important one okay for from from the field perspective.

C

Okay, so if we wanted to do it, one way to do it, and if we wanted to keep multi-cluster services separate from services again, we could unify them as well. One one sample yaml would be you simply have another um destination select destination identifier, so you know as an example here a service import reference.

C

So it says that um this part can parts that match this spot selector can or maybe cannot talk to these imported services right. So the gamble would be fairly straightforward. It would be just a simple extension of the regular policy.

C

Okay, the second one now so far, we sorry. So this is the third one. Okay, the second one is um also an egress policy, but this is for a given source to be able to select which destinations which remote services he can talk to, and specifically, even if it's the same service.

C

This part here p2 can is sorry. This keeps jumping pod p2 is allowed to access service foo from cluster b, but not from cluster c. Okay.

C

um This is what we've heard in some discussions about wanting administrative or geographical constraints on where you're able to access a global service from so who here happens to be a global enterprise service, it's available in lots of clusters and regions, but the admin for enterprise cluster, a or even the devops for cluster. A wants application p2 to only access the service from cluster b, but not access that same service from cluster c.

C

So you see that blocking there we can debate whether this is valuable, not valuable. The one. This is certainly not an obvious use case, but it does come up in some discussions.

B

So I'm on record in cignet calls and I'll be on record again. I think when we talk about multi-cluster and cluster sets, we should really discourage expressions of individual clusters you're either talking about this cluster, the cluster that I'm in or you're talking about all the clusters as subjected to the idea of sameness as governed by the control plane, which is a whole lot of implementation. Freedom that I just threw in there.

C

H

It yeah just uh emphasizing like from signal type cluster. This is sort of the case that is like contradictory to um how we've wanted to position that um if these are, if these need to be addressed differently for some reason, then they're actually different services. They shouldn't be merged together, like one of them is whatever west fu and east foo, or one of them is super secure.

H

Cluster foo and the other one is friends with everybody, cluster fu, um so yeah, just I think, emphasizing what tim was saying, but just that this this case in particular, was one that, like sig multi-cluster, um the namespace position statement and just the way that we've described to the multi-cluster services and how they merge is a little inconsistent.

C

I hear you and I'm not disagreeing I'll. Just add one one point here about this. If you look at the multiple clusters within a cluster set sort of like namespaces within a single cluster or or even a single namespace, I mean today we have policy that blocks pod a in the same name, space from sending to part b in the same name space.

C

So one would think that within the same name space it should be the same trust domain normally right. So why do we even have policy blocking part a from sending to part b in the same name space?

C

I know that's not precisely the same analogy here, but if you look at extending that to a cluster set right, it's not just because a cluster set is a trusted domain doesn't mean that uh any to any par communication should be possible for the same name. Space within a cluster set.

C

Think of it like part, a being restricted from part b, even though they're running in the same name space. So this is sort of taking that step to a cluster set. But I hear you both- and I agree that there's some debate about this use case, but it did come up so as for completion.

B

I'm mentioning it yeah and look um so much of what we're talking about here is either sort of rooted in philosophy or in a small n number of customers that we've talked to. um So I I'm totally willing to be wrong and to reconsider my fundamental premises. uh I just want to be clear about that.

C

Yeah, so this one is this: is this one is in the gray area in terms of value? Okay, moving on to use case c now again, um because all these use cases we are trying to make work in all scenarios right flat network and multi-network.

C

We are necessarily sometimes going down to the lowest common denominator or or to a coarser grain level than we would like. So this is one this exact. This use case is one such example, so the first two use cases were egress policies right. The sending cluster was blocking certain traffic. This one is an ingress policy right, so the receiving cluster is dropping certain categories of traffic right, so cluster c is receiving both from cluster a and cluster b here, and it wants to drop as fine grained as possible.

C

Okay, however, because this is constrained to work both in single network and flat network, it has no visibility to the individual pods inside cluster a so. The only granularity it can do is either at the gateway level or the egress ip level right so because there is an mcs gateway here from cluster a netting all traffic before it gets to cluster c.

C

Cluster c has no visibility of the source port ips, so the only granularity it can do if it wants to work in every topology, is by sending cluster or sending gateway entity or even sending eagles gateway ipr entity. Okay. So if we want to have an egress policy without taking the next step of adding metadata into the packet which we'll talk about next, okay, so here so far, we have all the use. Cases are in theory, working with all common cni's of today right. We have not assumed any special cni capability.

C

If we do, we could do more right and we have not assumed that it's the same cni in all the uh locations right, because the traffic coming out of the cluster uh looks the same whether that cluster was using cni a versus cnib.

C

Under those constraints this this is the best we can do, because we have no visibility of the source part and we have no tags added.

C

So first we'll talk about that, and then we talk about the fact that what could we do if we did have tags?

C

Is this clear in terms of sort of? If you want to do the lowest common denominator, not assuming any cni awareness and not having the source pod ip, then the best we can do is the destination cluster drops based on the source gateway or the source gateway ip.

C

Is that clear?.

B

Any questions I hear what you're saying: I'm just not sure how useful it is um like we know we have customers who want to run in flat mode who don't want to send traffic between clusters through a gateway.

C

So you did, you say flat flat mode.

B

Yes, yeah or or partially flat mode where they run like islands of v4 and flat v6 right, so they get to solve both problems.

C

Right so, but that would require a policy that that either only works in flat mode or assumes that there is some tagging of data from source cluster to destination cluster.

C

So this goes back to you know a similar discussion that sig network had for network policy, which is like today's network policy. One of the questions for cider based policies was well. Ciders are interpreted differently, especially for not south traffic right, because if you have an english policy based on an ip cider, some cni's or some knots out gateways preserve the source ip. So they can do a source, ip based cider policy, but not all.

C

There was no agreed upon source ip preservation thing and that's why cider was removed from admin network policy, if you recall, but here, if we assume that this we can define a policy which only works. If all clusters have metadata tagging capability, then we can.

C

This use case can be made more fine-grained by referring to an opaque tag that is somehow carried from source cluster to destination cluster and that may be identifies the source service or source labels.

C

Sorry, I'm talking a little yeah.

D

E

Sorry, the question so just to clarify so sanjay this particular part. um The assumption is that we can somehow individually identify the pods that we want to restrict from cluster a by using some gateway, api or a collection of parts or something of the chart from cluster. That.

C

Is part of the debate? How much do we want to assume how much source granularity should we assume like? If we don't assume any granularity, then it is the cluster or egress ipv? If we assume tagging, then we can assume more fine, grained planner. So it's a I'm sort of teeing up these questions as how much? How much do we want to assume? Okay, I'm not sort of being opinionated on?

C

Yes, we should assume more fine grained or no. We should not so. But your point is you: you got the right uh interpretation, which is that um a receiving cluster at a minimum can identify the source sending cluster, and that is the lowest common denominator that works in all cases.

C

But if you have tags added to the packet coming from source cluster to destination cluster, then you can get much more fine-grained.

B

What I don't want to do is end up with an api where the user who writes this remember the network policy is fundamentally a application owner oriented api right. We haven't talked about multi-cluster admin network policy today.

B

This is fundamentally an application owner thing. If the application owner needs to be aware of the network topology in order to write their policy, then I think we missed the point right, so it's either lowest common denominator for everybody and that's all they ever get or it's.

B

This is the api that we want to express and we believe it is implementable and it's on the implementations to shoulder the burden of it right. In some sense, it's like the difference between ingress and gateway.

C

But extending that point about this is a application developer policy.

C

If the policy works depending upon the deployment infra but doesn't work on some other deployment intra, then the namespace user has is becoming aware of the deployment infrared he has to know. Is this application getting deployed in a flat multi-cluster or a non-flat multi-cluster? That's right! I would consider that a fail right. So that's why? If we have to go back to the lowest common denominator, you have to assume the minimum abilities and then the.

I

Issue is you have to assume minimum capabilities or require a smarter implementation.

C

um Yeah as you'll see when defining like this policy when it has to define the source.

C

You the api, if, if defining your source, requires you to be more more granular than a remote egress ip right.

C

Then you, you are implicitly putting into that application. The knowledge that this is working in such an environment, where no.

B

I'm putting on the implementations the responsibility to if you want to play in kubernetes, you've got to get smarter.

B

Like I I'm all for meeting um implementations where they are when where they are make sense, but I don't think we should make a lowest common denominator api on something, that's as important as this. That just gives all the implementations a pass and says you don't have to do anything we'll meet you it's okay. I think this is a place where we need.

B

We need to represent the users, the application developers here in this cloud native world and say this is how we think people should be able to express their policies, and we need the implementations to get good at this.

C

So summarize, what you're proposing here are you proposing an api that can do more than the lowest common denominator, but somehow defaults? If, if your topology doesn't support it.

B

I'm saying we define an api that is rich enough to satisfy what we need, considering that like um going back to the ingress gateway example like the difference in sophistication between those two is, is huge. I'm not talking about that. Like network policy is itself not a very sophisticated api. I don't want this to become a hyper-sophisticated api.

B

I don't think we need it, but I do think we need it to be able to express what users would naturally want to express in with the consideration that they shouldn't need to be aware of the network topology. So, let's rather than thinking about well, this is all the lowest implementation can possibly do so. That's it we're going to limit it to that. um Let's think about what we would really like for users to be able to express right um and sorry cindy.

B

I know this is your talk, but we've only got like five minutes left and I really wanted to throw out the alternative um that we had we had talked about so do you want to tie it off so uh yeah? So your very last slide here is simply.

C

Yes, why is this thing bouncing around? Okay, the the use case d was okay. You assume that this is truly. You have a policy that truly assumes flat network, and it's a do everything policy, anything that you can do in a single cluster. You can do in a multi-cluster that assumes the control plane has global knowledge that there is.

C

There is an agent running on each cluster that can query labels and everything from every other cluster sort of they're it's a tightly coupled uh almost like a single control plane and that can then do very, very fine-grained, multiness multi-cluster network policies. That is also by the way what cilium does, because they have actually a shared control plane with global knowledge. So that could be another thing um I'll stop here.

E

So I just wanted to clarify sunday. You mentioned that I mean to do this. We don't is. We need to really assume that this is a flat cluster. I mean it's not sorry flat network across clusters. I mean that is not.

C

It's either flat or at a minimum it assumes there's a globe controller. Centralized there is a control plane which has global knowledge. Yes, that's requirement. Number one and requirement number two: is that you've got tagging capability in the data plane that any kind of source classification can be conveyed to the destination cluster through some metadata?

C

Those two requirements are there, then you can do this.

E

Right and the control plane requirement. um I guess when you say global control plane um it's I mean it could be a mesh and whatnot, but it's just one option you mean right. I mean you have a cluster set, which means that you know all the control planes in the cluster set and not necessarily need to be like a hub and spoke thing right.

C

Correct it could be physically a single control plane or it could be just the control planes are peer-to-peer like each control. Plane has access to each other control, plane, yeah, right, okay, okay,.

B

Okay, tim, you, you wanted to add something go ahead. Please I mean I guess I wanted to talk mostly to your to your option d there, which is, I would like to explore and consider uh a model where we assume that the same network policy api works for a single cluster and multi-cluster there's no new api.

B

uh Maybe we extend that api to make it easier to cover right so like one of the things we had suggested was what, if we defined this in terms of service accounts right like would that make life easier? At least service accounts have the nice property that uh they don't change over time right for a given workload? um Is that a simplifying assumption?

B

Maybe we don't need it, but maybe it would help right and can we make the same api work for both cases so, like specifically uh focused on the multi-cluster sig work right, so every cluster exists in a cluster set, whether it's actually defined that way or whether it's just like a virtual cluster set. That happens to be exactly this one cluster like it does exist in a cluster set and within that cluster set.

B

There's some control plane, it might be in a cluster, it might be outside of a cluster that control plane knows which clusters are in the set right.

B

So so there is this global knowledge again, even if it's a single cluster that one cluster is its own control plane, and we have this idea of namespace sameness across the clusters in the control in the cluster set right and um I've been meaning to finish a pr to make sure that the multi-cluster spec is really clear on this, but like that control plane has the freedom to decide which clusters express which namespaces, so it doesn't have to be all of them.

B

It has there's some implementation freedom there um and then, when I write a policy write a network policy that says to this set of workloads from namespace fubar. When I say fubar, I really mean the cluster sets fubar right. If it's enabled right and the way I get that policy across my entire set. Is I copy the same policy into each cluster right?

B

I can do that by hand, or I can do that through an automated mechanism from a primary api or whatever, um but the the point here being that when you look at a single cluster and you write a policy and you look at a cluster set, the policies are should be roughly the same, and that's the I sort of ideal that I hold in my head of making this work and then at that point the application developer doesn't know whether the clusters are flat mode, networking or island or mixed mode or have gateways or how those gateways work or anything about it right.

B

They could be theoretically running on istio and istio could be sticking a sidecar proxy and capturing the traffic and doing whatever its own gateway mechanisms do to get the traffic across between the clusters, uh like all of those, are valid implementations, and uh then I would like to see like what doesn't work about that. What what breaks down right, the obviously the naive implementation in flat mode would be to just copy information from all clusters to all other clusters, um and you know what honestly. I bet that works for a large body of users.

B

It won't work at scale, but I bet for those you know 90 of kubernetes users who have less than 20 nodes in their cluster. I bet it would work or 20 less than 20 clusters. uh I bet it would work, but it also gives us a lot of freedom to implement creatively right and I want. I want to give implementers the opportunity to show their stuff.

C

So um I think we can definitely dig into more precisely defining this model, which would include things like: okay, um if you're defining a name space level policy but to implement what you were just describing.

C

Each cluster needs cube, config access to each other cluster.

B

um We are again no, not necessarily right. There is a control, plane, I'm hand waving a lot like. Actually I don't know if you can see it because your slides are up, but I'm actually waving my hands around, because there's a control plane and that control plane needs access, that control plane might be one cluster or it might be a cluster with a failover or it might be all of your clusters. That is an implementation choice that I don't feel equipped to decide here or it might exist outside of any of your clusters right.

B

Maybe you have a control cluster, whose only job is to run these control loops right. Even if.

C

It is, you know, the control plane that needs access to all the clusters, but it is on behalf of all namespaces right, which are all separate applications.

C

So we um we were defining this policy for a namespace scope, multi-cluster network policy and we're saying that there will be this one logical, central, logically, centralized control plane um for all name spaces.

C

I I think, there's there's a. We need to dig through the the trust model and- and things like that.

C

Okay, are you saying tim that you? You believe that if we do that, then that might be all we need for multi-cluster network policy. Is that sort of your.

B

Thesis, I'm I'm hoping and I'm willing to go. Well, I mean I see andrew laughing, but uh like this is the r part of r d right. um We we need to go tr prove that the simple thing doesn't work before we engage in the complex thing, um and uh I have this this nagging feeling that I think it does work um or that it can be made to work um the the uh sorry uh sanjiv I lost the. I lost the question, uh so it is.

C

Is the summary of what you're saying that, um let's maybe just do use case d and it it it might be adequate and- and that's all.

B

We need to do I mean yes, I would call it d, prime, which is what do we need to change about the network policy api to make d work.

A

And prove that it's possible- I mean, I guess it's that question of how much are we hand holding or thinking we're handling cni's in api development or how much are we generating something that we think solves all the needs and is possible, possibly is possible to implement um and then give it to the cni? It's very give it to the implementers, which I think it's better to do the latter, but.

C

But we do want this to work again in all those models: right, single flat network non-flat network.

A

C

You don't care.

A

Because you give the complexity to the to the people in collecting the cluster set right. Is that what you're saying tim like yeah.

B

Look if I have, if I have this island mode, multi-network thing, and I want to teach it about multi-cluster networking, a multi-cluster policy like maybe I have to encapsulate the packet when it leaves the cluster, because that's the only way I can carry information to the other cluster. Like that's a valid implementation choice. Yes, we're putting some hardship on the implementations, but it's better than putting the hardship on the users.

B

C

I think um I need to find the right way to work. My response.

C

We we don't, I think, the current network policy um api um does things that we we don't actually need. I mean it's it. Firstly, it's everything is brought to participate to power right um and I think that's not an ideal model in every case like, for example, if we wanted to implement multi-cluster network exactly like you know, today's single cluster policies, um it's basically a full cross product between every every part that matches the source selector and every part that matches the destination selector.

B

Right, well, that's why I'm saying if we need to consider things like service account right as a simplifying assumption, then then we could right.

B

um It at least reduces the combinatorics.

C

The service account doesn't we need to go into a lot more detail, because a service account is not actually carried in the packet, so some proprietary implementations of policy based on service account eventually map it back to the power source. Ip.

B

That's right, a really naive implementation here goes to each cluster and it says: okay, give me all the pods that match this service account for each for each cluster. For each policy give me the pods that match this service account in other clusters, compile an ip set of those ip addresses and install some ip tables rules right, and that sounds great when you're talking small numbers- and it will obviously blow up in your face when you're talking about big numbers right, but it's like demonstrably doable.

C

Okay, so let's come back to service account separately, but the other point, being I mean service account, is sort of one aspect of it.

C

Are you saying that we do not want to introduce service based policies right, whether in single cluster mode or multi-cluster mode? It's it's? It has to be essentially this part-to-part sort of policy which we.

B

Have today so, given an infinite number of uh ants, walking an infinite number of paths through the maze I'm happy to explore both paths, um but I don't know if you have an infinite number of ants, um I I don't um so. My inclination was, if we're going to prioritize.

B

I would put let's figure out if the existing network policy can be made to work first, because, like ergonomically, it feels better to me and I'm totally happy for people to argue with me right in parallel or in sequence, we could explore what would a service oriented policy api, look like both for single cluster and for multi-cluster right and if we can agree on what the semantics of that are and if it turns out that that's actually way easier to do. Multi-Cluster with okay, I'm cool with that.

B

But I don't want to go to service just because we didn't explore the traditional network policy, because I think when people think about network policy, they really are thinking of it in terms of workloads. They expect workload semantics at least right.

B

But what dan said earlier, you may you may define it in terms of the service, but you implement it in terms of the workload.

B

I think it was dan yeah. It was.

C

Dan, let's try to wrap this up. Y'all! Okay, so is the takeaway here that and definitely all points are perfectly valid and good points thanks everybody um if we can make something like use case d, work or d d, prime whatever, which is uh reuse. The existing network policy api figure out what it takes in terms of a quote: unquote: global control plane or how much do we need to define or not define how each cluster knows about each other cluster um and then just throw some.

C

Think about a sample implementation, because I I think if we, if we want to have all the functionalities of current network policy, we will necessarily need a cni that can send a metadata from source to destination.

B

uh Cluster, I agree with that. If, if I'm not setting the bar at must support everything that the current network policy api supports right, I'm I think it'll be okay.

B

If we said here's a subset of what of the regular network policy api that was known to work across clusters right, um I think that's an okay starting point. um In fact, I think it's worth doing, because I think revisiting network policy as an api is worthwhile anyway right, because we've learned a lot of lessons since then.

C

Okay and then what are the any key quick points on the the control plane sharing right, the the global like? Do we need to define that or do we just leave that as an implementation, like somehow the policy agents somehow figure out they have access to all of the clusters? First, I.

B

I'd like to uh I'd like to know that it with confidence that it is implementable um and to do that really means we must implement it at least one time right, but I feel like at small scale. This is not a very complicated program that I could write against. My local cube, config, just to say, like here, is a demonstration that this idea of a control plane works, but also. I think that if we look at like istio and the way istio works, we already kind of have a demonstration of it.

B

It's not exactly mapping and it doesn't consume this api. But its semantics, I think, are pretty similar right, which I count as actually a point in favor like. If this can be a step on the way towards something bigger and richer. Then that's a win right because I know I don't know you guys, but I talk to a lot of customers who want istio but are intimidated by the adoption curve of it and so giving somebody a step towards that.

B

Even if they decide not to go to istio, they go to link or d or something else like a small step is a good thing. Sorry, I stepped on someone else dan.

C

No you're right, in fact one note I added here, which is based on some feedback from some service mesh people, was that it would be useful to have it. This could be a way to reduce sidecar state as well, um where you know as a benefit to the service mesh policy.

C

uh The multi-cluster network policy, you know, could be a single instance of node level policy that the sidecar could look into, rather than replicating it into every sidecar proxy, so that there is, there is an avenue of.

B

Helping- and I should all I feel like- I should be also clear like if what we decide turns out to not be similar to what istio or other meshes do that's okay, too, but we should explore it.

C

Okay, so we'll uh what we'll do is we'll sort of chart out a rough um control plane sharing model like how? What is the minimum needed for global information sharing and some sample sort of take some current network policy samples and see what kind of information would need to be known.

D

C

Kind of make it work and it will require every cni to um to support some kind of additional metadata in encapsulation.

B

Well, only those that want to participate- and I should also say I- I don't think it's our purvey- to define the standard encapsulation like we're, not going to come in here and say we're using a geneve blah blah blah blah to make this work like we're, saying here's the api, with the semantic that we think makes sense to users and the implementations get to figure that out and if it makes sense for psyllium and calico to say, hey well, let's agree on an implementation so that we can speak across clusters.

B

That's cool and I would encourage that. But I don't think that's our job to require.

C

But at a minimum you need to know how much context can be sent from a source cluster to a destination cluster like? Is it a 32-bit worth of context? And how does that translate into the api? Because, like today's today's network policy, api doesn't have um a notion of a cluster id right. So if we say that we want to use the semantics exactly like today's network policy api but say I want to drop traffic coming from clusters with label a and namespaces with label b, that's a new semantic that is not there.

C

In today's network policy period, you you.

B

Have to have so that's true, there's no there's no cluster object, and so, if that does turn out to be a requirement- and I will fight hard against it. But if it does turn out to be a requirement, then we you're right. We need to answer: where does that information come from and how does it get sort of cooked down into something that a data plane knows how to use.

C

Okay, but in lack of that, the other answer would be. I want to drop traffic coming from pods with this source pods with this label, no matter where in the cluster said they decide and and that's it right.

B

I mean that sounds like a wonderfully simple expression if, if we could get it down to that, like how cool would that be? Hey people plug your plug. Your clusters into a cluster set implementation and network policy just works across clusters.

B

That sounds like a phenomenal kubecon talk, so the only dependence on.

C

The sick multi-cluster model here is that these are all part of the same cluster set, but we're not really using mcs here we're not talking about service imports and exports. Then that's right. I mean that's the way. I'm thinking about it right.

H

Yeah, the service question is kind of separate if service.

G

H

Aggregation for people to uh to to represent network policy, but it sounds like you want to start with the current network policy, which is poddupod and it sounds like. There are separate reasons why workloads like workload to workload is the way people want to.

B

An establishment I just feel like, I need to repeat like if somebody wants to explore what service policy would look like. Maybe it's an entirely distinct api from network policy like let's cut cut bait when we have to right as an entirely distinct api. What would the semantics of a service policy api look like and how do we handle those corner cases of multiple conflicting service policies?

B

That sounds like its own interesting.

A

Exploration that that was on the daca for v2 and the folks who were driving that are no longer around. So it's an open work item so that hopefully we can get some people involved to help out.

B

A

B

Not pre-deciding uh what works and what doesn't work here right. If I, if I knew the answer, I would just come out and tell you yeah.

A

I think it's a different problem for one and I think, like a lot of these questions on g, we can't answer until we go. Do like a bare implementation. Sort of you know does. Does this work? What information needs to go for the wire sort of poc right.

B

Yeah we, you learn 10 times as much by proof of concepting it than you will by thinking about it.

A

um Sure, on that too, I I would like to wrap it up. I know everyone's super busy and we're 18 minutes over so um but thanks so much for coming, everyone really appreciate it. It was some good conversation um yeah. We can keep doing this in this meeting slot. If we would like happy to happy to keep it running, um yeah.

B

Sanjeev are you gonna come? Are you gonna take away and come back with a sort of proposal d? Prime, or do you want us to go? Do that somebody else to go to let's? Let's do.

C

It together, whoever wants to get involved, we can, we can all collectively. However, we want to do it.

A

Yeah yang was really wanted to be involved, he's on maternity, paternity leave right now, so I know once he's back, he will definitely be around to help sanjeev because he was doing this for vmware. He was trying to answer some of these questions already yeah yeah, absolutely.

C

So yeah um anybody else interested in collaborating on this.

B

Just just reach out for sure: I'm I'm, you know I'm going to participate as best. I can.

F

So before we go um on the topic of implementing, rather than just discussing things tim, should we be waiting for you to re-review the uh admin network policy api pr or should we just merge it and get people to start implementing v1 alpha 1.? I was about to say that thanks.

B

Dan, you should plow ahead um now that caps are done, um I'm I'm going to be able to turn my time back to prs instead of caps, um but honestly, like we argued the api to death in the cap like plow ahead and let's see how it goes.

A

Okay, well then, dan, I will merge it and we can move forward sound good to everyone here. Sorry, we jumped topic, but going once going twice and and yeah tim, please still look at it. If you have some problems, um just open an issue. How long have you known me not that long.

B

I think you know me well enough that I am not gonna hold my tongue.

A

No one on this call does that's why it's a great community cool, okay, hey thanks! So much everyone, it's a productive meeting, thanks for coming on the odd week off, and uh we will see you all soon, hey.

C

I'll just have one one quick note regarding amount of work right, um I was actually able to prototype a service based policy like over a weekend like, but if you have to do this whole global sharing and it's it's actually going to be a non-trivial impact for all the cni's to implement this. So.

A

Well, let's wait and see: okay thanks everyone we'll talk to you later.

H