Antrea Community, 21 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Antrea Community Meeting 11/21/2022

Description

Antrea Community Meeting, November 21st 2022

A

Perfect so good afternoon, good morning, good evening or good whatever is in your time zone and welcome to the distance of the Andrea community meeting uh today is at least in this time zone is Tuesday, 20 November, the 22nd, and uh for today we have an update from China regarding the uh API changes for layer, 7 metropolitis, so Sean yeah you're a sharing the screen. So please go ahead.

B

Thank you, and thanks again, for attending this meeting and after receiving a lot of useful feedback in the last two Community meetings and after some more investigation and the discussion with changing I would like to share that.

B

We make a change to the level 7 policy API and now our proposal is to merge the the seven policy to the existing layer, 3 and therefore their policy, and we made this change because of several considerations, including that first, it is easier to manage all Security in one place if with a separated, seminal policy- and maybe we will need another cluster, scoped 70 policy user will have file apis to manage the security of their workloads. So it sounds a little bit complicated and second is after receiving many feedbacks about use cases.

B

In last in in the in the last community meeting, we found that if we have separate apis, it would be more it would. You would need more uh it would.

B

It would require the user to be more careful when they want to use the seven airports, especially when they want to lose their workload, need to use multiple protocols and if some of the protocols are not supported by the seven features yet, uh for example, with separate clsm API, because the near policies are enforced in different stage, so a traffic, the traffic must be first allowed by the three under their foreign policy. Then it must be allowed by The Seven Year proceed. But you know some Protocols are not supported by the 70 policy.

B

For example, my circle already some database protocol, but uh we will have to scope the traffic. We will need to define the scope of the traffic that will be handled by you know 70 policy, otherwise they will still be dropped by the seven after enforcement of layer 4.. So on. On the other side, the the user must guarantee that the traffic must also be allowed by level 4. So they need to consider.

B

They need to be well careful to make the the three under four under seven to work together and there has to be some redundancy okay for the for the laws and and the server they need.

B

It is more consistent with the assessing existing Solutions if we import the the seven protocol into the 340 policy in which investigated uh several existing solutions that supported our seven firewall, including NSX and cilium and Calico, although the behavioral behaviors are a little different, but the API design is very similar so to be consistent with existing solutions to and give user constant experience. So we are also following this style and after making this change, there has to be some a few limitations because of the a restriction of implementation.

B

For example, as you can see, from the new test charge of the network, we have added 1.7 protocol s failed to the existing, therefore law, and but it's it its meaning. It's a little different from others, for example, for this port and the protocols their relationship is all. It means that you could use ports to define the port. The traffic should match, or you could use broadcast, to define the other non-tcp and non-uvp protocols such as ismp and ijmp.

B

The the traffic will match it means either of them is, is matched the traffic will will take this action, but for the seven protocols, because it's it needs the level 4.

B

To scope traffic, it doesn't mean that their relationship- it is not all but- and it means the traffic- must first match the the ports field, and then it will be sending forward into public the last seven a well engine for protocol detection and the lower enforcements, so the post office there should be matched and because we support only allow list allow this term for now. So the with this field being said, the action can only be allowed at the moment. This limitation exists exists in other solutions that are supported.

B

R7 therefore say two because most of the the seven engine supports only white list or Blacklist, not mixed priority against style loss, and uh as long as the traffic matches the level 4 criteria, the traffic will either be allowed by the if they match their seven protocols, otherwise they will be dropped so any rule. After the last seven book, one will not be enforced if the traffic already matches earlier for the criteria. So this is the limitation that we we have to had.

B

uh We have how to have because uh the implementation registration, but it shouldn't affect the actual use case. For example, uh I I listed two use cases we received in the last community meeting the first he said for an implication. I know that my my workload only serves as a HTTP server. So what I want to do is I only want to let the clients to accept my workload using HTTP protocol, because this is the only one it's supposed to be used. So this is what this could be put agnostic.

B

Equally, we could Define such a rule to allow only HTTP request and the master match, with the hosted name and regardless of the transport protocol, of course, is TDP and you can solve the port being used. So this is one use case, and another use case is I know. My application uh well depend depend on uh some other services.

B

One is a database service radius Service as an example I wanted to allow this commit egress communication with this port and I I know it will also relies on and at the last API you external rest API, so I, I, I I know that it's a uh it will use the TTP 80 port or it could be any other portal, and it could even- and it doesn't not even not need to do a need to specify the port, but because of because the uh the the TTP this rule is before the layer 7 rule.

B

So the the database service traffic will first be allowed. Then all TCP traffic or otcp traffic with this port specified can only be http, and my my application may also need to access DNS service, so I I will also I. Can I could also specify uh another udv Port. This could also be moved behind this uh behind this level, seven and Rule.

B

Then we don't need to specify this therefore criteria, because all traffic, except except these two uh two, this two def or already defined rule- will be dropped if they are not http and but if you specify another level of follower after the layer, 7 and the layer 7 only in effect a specific Port traffic, you will need to Define another job, job law for other traffic that are not selected by this level, 7 rule.

B

So this is a use case of post-specific protocol detection. It will only inspect traffic to Port 80., so I think this API chain should meet the requirement of all use cases, and so there is already a PR that implemented this change and any comments uh welcome.

C

um Trent I'm I'm, really sorry, I joined the meeting late, so I I probably didn't catch the first part, um but if we merged this into the labor their full policies, does it mean that you know that right now the there are seven protocols or the layer? Seven rules are no longer uh in place at isolation.

B

Oh sorry, let me share back my screen. um Can you repeat your question.

C

Yeah so I remember, you know when they are 7B in this whole API. It's like kubernetes, now a policy where, if you specify there's seven protocol, then yeah anything selected will be implicitly isolated.

C

um So when you move this to the layer four, um how does the isolation work, though.

B

uh It it depends on whether you also set the level 4 port. If you let, let's take this two examples, uh if you don't specify the level 4 Port, it means that this is a port agnostic, protect, protect detection.

B

All in English traffic must match HTTP Protocol. No other traffic will be allowed, regardless of the port being used. You could use a port 80 to access HTTP service. You could use other Port as well, but regardless of the port being used, they must be HTTP request and if you want to only to uh say you, you know that your application will only run on a specific, listen to a specific Port. Then you could also Define the scope of the the the traffic that the last seven engines should inspect with.

B

In this example, it won't inspect the traffic to TCP 80 Port. No, no other traffic will be inspected by the uh the last seven engine. So this isolation happens to this port. Only if you use a portal, for example, a.

B

Eight eight thousand to use HTTP. Then it will be dropped by this rule instead of this rule, because it's not this lowest responsibility to inspect another port.

C

But but in the case of the first use case, when you have this specific Ingress rule, it means that, for whatever selected in the apply to which is hidden here um will have the implicit isolation effect. Basically,.

B

Yes, yes, yes, as is planned by the comment, I will also document this behavior that once uh because you didn't provide the seven earlier four Pro criteria.

B

If we we, if we don't have this, it means that all traffic will be allowed right, but if this is a pure level four, so actually the level criteria is used to scope, the traffic that will be inspected by the 7 engine, so this this lure already will handle all incoming traffic yeah under the isolation. Well, of course, we are happy because it will only be allowed once this match and uh if this seven protocol is not matched, they will be chopped.

B

If you don't want the isolation for all your incoming traffic, you should specify the there are four criteria. For example, you want the other 7 engine only to handle traffic to TCP 80. Then you should provide something like this.

C

Well: okay, but um I'm just I'm just worried about not only you know the the meaning of it, but also. How do we implement this like right now on the layer, 4 um layer, 4 case this would be, will be as is right and and right now you know if we, if we, when we introduce the L7 protocol, um I I do feel like this is a huge change in the semantic potential semantic meaning of the rule.

C

B

Yeah yeah I understand that. That's also why it was not closed proposed in this way in the beginning, but after more discussion, we found that if we, if even if we have a separate uh last Seminary policy, there has to be some limitation and there has the meaning of the women how to do some. Something in therefore, as well to make the work together and we will also need to search.

B

There are four criteria to scope, the traffic that will be handled by the 7 engine, so he is not the best, but it's. It sounds better than the separate.

D

B

I understand your concern, but better for my investigation that other solutions that use this style also has this limitation. Even they support, allow and deny job and they support the priority. But actually the priority does not even play a lot in the when the blue is seven yeah, but at least we in our in our limitation with their respect their priority, and it's just that we we will have a default deny when the traffic uh is already handed by the 7 engine.

C

Okay, let me um you know, because, because you're saying that you have a PR, uh let me first look at that and uh um I'll. If I have more comments, I'll, just post it there thanks.

B

Sure, thank you for your comments.

E

I have a small question in my understanding: I think the learn form rules is used to filter the traffic tools and there are seven engine and, and then, if we don't limit the large wall traffic, then all related traffic will be found in the 2007 engine and the only the only the rule defined by layer 7 will be allowed. Another other track bill will be dropped instead of my insta is my. Is that my understanding right.

B

Yes, yeah, it's correct.

E

B

Okay, if no other questions so after I think we could go to collect the topic thanks.

A

John no I I, don't have any other question as well. I mean I need to digest a little bit to the change that I'm making DPI I understand the rational and I mean it's very well understood.

A

um You know, I'm, not really considering the impact from from a implementation or from an implementation perspective, because in a way another will find a solution there. One aspect that uh probably we want to consider here is: what do we need to document for users? So is there any caveat that we need to document or it would be like just standard entry or network policies within addition, because from what you told me, I think there are some some aspects that users must be explicitly aware of right when they want to configure layer, 7, Network policies.

B

Yeah yeah: we will enhance the user experience in two ways. The first is: we will have validating webhook configuration to uh to guide the user to create a legal layer, 7 protocols, for example, when they specify their loss. There are seven protocol we will validate.

B

The action can only be allowed and we could also check the check check the meaning check the relationship between the level four or criteria under seven, for example, when it's HTTP I think it's not, it does not make sense to use to use UDP protocol to specify PDP, so we could also do such validation and and uh another second, is: we will document the limitation in the internal policy while using a separate section about level 7 and document the what it means when there are seven protocol is the specific specified along and what, if they want to limit.

B

The seven blue only applies to a field traffic to a specific port and the default job Behavior. We were document to them. Clearly.

A

Thanks thanks for the clarification so before we move to the next topic, is there any final question for Sean.

A

Three two one and now it's time to move to the next topic, which we don't have on the agenda. um Lan I don't know: do we have any update for uh policy only multi-cluster for today.

F

uh Yes, I think I will give another update for the for this topic.

A

F

Yeah, can you see my screen.

A

F

uh Hello, everyone I think last time, I gave an overview of current design about the mod class traffic. When we want to write in our policy only mode- and uh in this page it's actually a later recap about what we have in the in-cap mode.

F

So you know that's in in cap mode, when we have those, let's talk the in-class traffic, it will go through the tunnel right, even even it's in cluster traffic or even it's, a cross-cloud traffic or traffic will go to the tunnel interface and it's all controlled by our entry agent, our own children, the primary ceiling and the multi-class traffic. Well, one thing that I like to highlight is that the multi-class service, for example, here it's, uh we call its multi-class service and it has some endpoint.

F

The end point is the uh plus IP from other member clusters. For example, here you can see the service full. It has Part B the IP is 192, but it's not the endpoint of the multi-class endpoints. We will. The multicast endpoint will be the service class IP. So you will see that so when you are trying to access from the port a in the member Class A and the part A actually access this IP. This is IP is the class IP which is uh generated by the local member cluster a and it will be have.

F

It will have a new endpoint. This endpoint will be maintained by entry Mart cluster, the endpoints will be the class, so remote service cluster IP. So here is uh which I like to highlight before we move forward, and one thing as I mentioned: all traffic is in the tunnel and we have it in the income mode right. But now we like to you know, run those traffic in network policy only mode and in nail polish on mode, uh the entry is no longer the primary cni.

F

So, as you can see here, the assumption is no longer meet and it's no tunnel interface and it has no tunnel between nodes, even in cluster. So even we set up those tunnels between different member cluster. Here, like this to support a multi-class traffic, uh we still need a way to you know because we have a Gateway. It should class the member cluster. We need to make sure that all traffic goes to the Gateway first, then it goes out to other member cluster.

F

So we need a way to let any cross cluster traffic, which will, for example, here if you see that if the request is from part A, it need to go to the tunnel, but the network policy only mode, there's no tunnel here and the Pod IP is not controlled by our entry side, because it's not the primary CLI and also the uh the Pod access is actually controlled by primary cni.

F

If, let's take the E case as an example, uh the all the traffic will go through the were controlled by the Primacy eye, the corresponding roads will be set up, so the part actually will not go through the tunnel. uh One thing is, there is no Tunnel right and another thing that or uh those three roads will be added by the primary cni.

F

So we are not controlling the traffic anymore, so we need to resolve this Gap and the last last time when I showed the Gap and we found that we need a list to do those changes in the nail polish only modes. One thing that we definitely need a channel interface.

F

Of course it's for cross-cluster traffic, only for local cluster traffic. We don't want to impact them and another thing that we need to set up the roads between the Gateway and the general node. So we can make sure that all cross-class traffic can be forwarded from uh or rotated by, the Gateway. Only right and the next thing that which is uh a new funding that last time, I, missed I, think we also need to you know, update those containers, interface MTU, because we have the tunnel interface here and for any cross-class traffic.

F

We need to reduce them to you, so it cannot be dropped when there is a large package so for now, I think this is also another Trend. We need to make it make in the entry agent and actually, uh since last, community meeting uh I uh I think the first design, which is a little complicated, and we have a follow offline discussion uh with a few people and here actually comes up a new Option.

F

The option A is the one which I introduced the last time, which is trying to use a contract connection track module to. You know to save those uh channel uh Source tunnel IP to make sure the the getaway can can know uh which parts traffic should be forwarded back to the source node.

F

But after a few discussion we think that's, it might be simpler to just create a lower Swift, rewarding which you can consider that it's as part of the roads inside of the OBS, instead of Let It Go controlled by the primary cni rights we need to. We need to capture those traffic figure before it goes out and controlled by the prime ministeri. So we need to add the research forwarding rule for each part, I think the first option.

F

It's the original purpose is to trying to reduce the open, Flow rules, because you know if there are lots of Parts.

F

If we set up, then each part in the level straight forward, it might be a huge number, but after a few discussions we think that so one thing is that the CT module will introduce some performance issue or we say the latency and and actually in the narrow policy, only mode, it's usually running in the public cloud and the part number should be not so big. So actually, even we add a one rule per part, it's a it should be acceptable. So uh actually I do that.

F

I did a performance test later I will show the data. So here it's actually the option A, which I actually I, think I have uh show this last time, and uh the mem issue is that in the the or the difference that in the in-cap mode, let's go back to uh sorry in the in-cap mode. I think we have.

F

Oh sorry, let me go back to the okay in the income mode. You know between different nodes.

F

It's actually based on the part cider to determine which note it should be goes to right, but see in the narrow policy only mode we don't have search for such Rule and uh so in the option A and we need to set up those power roads between Gateway and the general nodes, and in my original design, I was trying to set up those pod roads only for those exported service, and so it can reduce the reduce the number of the open, Flow rule.

F

So here is the original proposal and uh and in the original proposal that we also use the CC label to save the The Source Tana IP of the request. You know that when we have a request package from General nodes and from a pod or general node to the Gateway, the Gateway, the reply package, we won't be able to know uh which, which node it should be forwarded back to so we use.

F

We use the city label to save the request package resource tunnel and after the reply package is back, we use a city label which we have saved to match the destination and and then we can set the destination tunnel.

F

These two are the two parts of the option: A and it's actually the one. It's only one goal is to set up the Pod roads in the gateway to make sure the Gateway know how those cross-cast traffic can be forwarded to the original or not. But after a few discussions, I think.

F

Maybe this only this row table is an uh it's a enough to set up those Polaroids, as you know that we just need the level 3 for boarding table and add one row for one part, even though we add all roads or issues for one part and every part slots is not running in the Gateway, it will have a corresponding role.

F

To tell that, for example, here is a pod Original Part, which we can say it's a clan pod, it's IP and when it's a parties, this match to this IP- and we will set the right tunnel here and the only thing we need to do is to watch the part event to set up this kind of rule. The difference between the option A is that the option A we might not need there might be 1B, so many important rules, but in this option b we may have one rule for one part.

F

So when the party's number is Big, then we will have a long list here, but after I think uh based on our experience, we think that's a the performance or the latency introduced by this rules number should be acceptable. So there are two options and I did some Performance Based on these two options. Actually you can see that see, option A or the the original design at and and the option b in the TCP type string type of test.

F

It has not much difference, I think, but for these two it's actually has a slight uh uh improvements. If we goes to the plan, B sorry the option b, of course, the option b- I only add one rule for the Pod IP. So it's actually no uh huge open, Flow rule list in the environment, so the number might be not so precise, but it can tell us that if we don't have the city label in the in the process, it may help on the latency and help to increase the transaction, a number yeah.

F

uh uh So we plan to go to the option b and uh currently the code is almost down and I'm still trying to do those uh a verification and during this verification, I think we still have one same remain here. It's you know that so we have a stretch for now policy on the development and after a few uh verification, I.

F

Think of when the end point of the multi-cluster is the local service class, the IP and the part, the client part is trying to go to the local service backend in the same cluster, the actually the product traffic. It won't be go to the tunnel traffic. If we didn't do anything regarding this part, the traffic will go through the Gateway.

F

uh uh The connection is okay, the service is accept, is accessible, but it's not just through the tunnel interface, so it will be a problem to support the stretch in our policy in this module and uh I'm thinking that it's maybe uh this is a gap between currency implementation and if we want to stretch this now policy, uh our manual policy only mode, and we need to do more things and do more investigation on these parts.

F

Yeah I think this is the uh the one problem remained uh I, don't know if any other uh team members have any suggestion on these parts or any opinions or question.

A

No thanks a lot for your presentation, Lan at least no question from my side, we'll need to find the solution for stretch, Network policy support because uh I still thinking as a user. It will. It feels rather awkward if we have a network policy only multi-cluster, and then we are not able to support stretch Network policy in that modality in that mode. So yeah, we'll uh we'll we'll need to figure out a solution for that that I don't know if a young angry isn't already have some idea.

F

Yeah I think yeah yeah. Do you have any uh comments on this? One I think that today, I actually talked to basic reason, and we may need to uh to more uh verification regarding this.

A

C

A

Don't have any.

C

No sorry, I don't have any uh comments on the now I probably needed to look at it a little.

F

Okay sure thanks.

D

Yes and uh yeah also since currently, um although the client and the endpoint of the service are on the same note, the the traffic are still going to the tunnel, so maybe we should create a situation that we can verify uh if if the traffic goes through the Gateway, uh what will happen.

F

uh I think if the traffic goes to the Gateway, the tunnel ID should be lost, I believe right.

D

Yeah sure yeah yeah, we saw it in the in the agility pattern so.

F

um Yeah, so I think we still need to guarantee that the traffic goes to the tunnel interface right.

D

F

D

As long as we are using the tunnel ID to store the label identity.

F

Yeah: okay, yeah, okay, got it thanks.

C

But I mean it could be, it could be. We could use some other um um designs like for for, like uh um like the the specific uh scenario uh we can, we can just if it, if it's just you know in cluster uh sorry, not in cluster, but on the same node, then maybe we can do something like uh we can.

C

um uh We can consider that case specifically and uh try to not use the um nearby density or what not to to enforce traffic, but that that could be discussed later. I guess.

F

Mm-Hmm, okay, yeah sure, I think there might be a little different than I expected at the very beginning on the nail polish, only mode and uh yeah I think the case is a little special and there might be two cases in the nail policy only mode one is that no, the any point is in the local node and as the client part and another case is uh the no the part, the The Sovereign policies on another note, maybe I think we need to verify both and uh yeah I'm, not sure actually from the nail policy only in perspective how we can resolve this kind of issue or we have to set them into the tunnel interface.

F

But if we do have some options, maybe we can have some discussion later. Yeah.

F

Okay, thanks yeah and uh so I think that's all for from my side.

A

Thank you thanks a lot, so uh we don't have any other topic scheduled for today. So I don't know if anyone wants to bring up something for discussion, if you want, please uh go ahead and now it's a time for open discussion.

A

Well, it's fair to say that probably there are no other topics for today. In that case, I would like to thank uh all the attendees for uh for attending through this meeting, in particular thank John and um Lan for presenting about layer, 7, Network policies and network policy only multi-cluster mode. So thanks again, uh thanks again for your presentations and I would like to wish everyone a good day a good afternoon or good night. Thanks for attending and I'll see you in two weeks time.

A

And for those Basin U.S Happy Thanksgiving.

A

Thank you. Bye-Bye bye have a good one.

D