Antrea Community, 19 Jul 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Antrea Community Meeting 07/19/2021

Description

Antrea Community Meeting, July 19th 2021

A

So good morning, good afternoon good evening and welcome to this instance of the anthria community meeting uh today is uh tuesday july the 20th or if you are in u.s it will be still monday july, 19th for you and uh for today we have only one topic on the agenda and uh we should have hong yang and uh sorry if I get your pronunciation of your name wrong, uh presenting about the andrea proxy design with tc, and this design should include support for both node port and services of type load balancer.

A

No other topic was uh booked for uh today's meeting. So I believe that we can start with this presentation about the entry proxy design so online. Please go ahead.

B

Okay, thanks sabato uh vitamin outside the.

A

A

um Are you okay? Yes, we can see the screen now. Okay,.

B

Hello guys this is a presentation about a new design of entry proxy in to support the node portal of answer and class fpl kubernetes node, and so far it's linked only now, but I think some parts of this design can be also applied to windows in the filter, okay, to outline motivation and current design and the new design the support support cases of the new dji.

B

First, the motivation, the the target of this design is to remove the kuby proxy. If, if the copy proxy is removed, a lot of cpu circles and memories can be sealed, we need we don't need to handle. A lot of applicable interest is that we may account a lot of os flows in current design. The service can be only supported fully when the client is from the port. If the client is from remote host, android proxy has to need the help of kubi proxy to redirect the service traffic into entry gateway.

B

In addition to get the better performance, if the client is probably from hosts remote hosts, the pro the performance of entry proxy is limited by android proxy by could be proxy. Now.

B

Okay, this is the current design and we can see that that, when the client is up is from the pod all service, all services can be supported without copy proxy. This kind of service. This kind of traffic can be handled inside the oes.

B

However, when the client is from remote or kubernetes nodes and the service where people must pass through cookie proxy to do the net, that's the root cause of good anti-proxy performance is limited by cookie proxy.

B

What here is a new design? Okay, I'll introduce the new design in two parts. The first part is the linux is how to redirect remote and localhost service traffic to actually gateway directly without passing through and to proxy, and how to make the response. Traffic back to the client and the new design used linux tc to handle the node power and load balance traffic from remote clients and for for non-power traffic from localhost link. Tc is used for load balance. Traffic from local hosts and new design use can leave roots to handle it.

B

The first part is all the issue of bypassing could be proxy without could be proxy. Entry proxy must handle the complete life circle of service traffic. I think this part can be.

B

Oh, oh sorry, and this part this the second part, and the second part is os pipeline. I think this part can be used in links as well as windows, and there are three main reasons: first, how to do smart for source traffic, the second one, how to handle the destination mac address of the response, traffic and the third one compatible with the lag city plan.

B

Okay, this will talk. This will be talked later.

B

I'll introduce the new design, the first part take notepad, for example, okay uh for request, uh notepad track notepad traffic from remote client. First, a filter will create another test to the field. Take it as an eagle example uh we'll attach the two eth0s in grass for every node pulsar is here we take so and one and that's the definition part by default. The filter will be will be put on train 0.. The handle id is generated by generator according to notepad ip address layer, 4 protocol and the node part number.

B

These four factors can ensure that hand id is unique.

B

Deleting the node service the handle id is used to index the filter and delete the filter, then what the filter do is that it will be redirected to entry gateways, egrass. Okay, we can see the tc command here.

B

And um pay attention that if the interface has more than one ip addresses the about the about command will be generated for every ap address, and we can see we can. We can also say that the it is h1 um this this command will will be also generated for each one as dch0.

B

Okay, the next one.

B

For request, notepad traffic from local host relax are some differences first to clients from localhost, and then the target notepad ipo addresses can be loopback ip one.

B

Ip address and the utsu ones app address filters should be created and attached to two blue bags. He grass not not ingress like eth, zero and the source mac of the packets should be modified to actually deduct and together with the mac address, otherwise, the package will be dropped by obs.

B

Okay, here here is the tc command. Okay. Next one.

B

Okay for response there, the request job is that here is the response like epi tables, pc field and linux. Tc supports hierarchical change.

B

We know that the only difference of the node port node pause is the port number to redirect the response traffic to interface wire. This request traffic is formed according to the source, ip source id of the traffic.

B

Some basic filters will be created to redirect the mice traffic to a target structure.

B

Note that if the source ip address and the destination ip address are the same need to indicate that this request traffic is from from loopback.

B

The filters would have should higher priority, otherwise it will be matched by other filters to ech0 or ethereum. That's not what that's not what we want.

B

Okay, that's for sub chain is where the second layer filters locate one node pod is created the filter. The filter will be created and attached to the sub chain. The filter matches the packets according to the to the source. Part under the action is to redirect the mass response traffic to the interface, where it's requested traffic from if the substring is full go back.

B

The source map and the destination mac address should be also rewritten to r0. Otherwise, the loopback will absorb the package. Good source or definition. Mac address is not r0. Okay, here with the tc command.

B

Okay, next one for load balancer here we assume that the load balancer traffic can be only from oh sorry, assume that the load balancer can be only from from remote can only pass through the default route. That's easier! Zero!

B

If there are multiple default rules selected into which the law is the metric, and that means the high highest priority and we don't consider about the complicated routine configs.

B

Okay, that's almost the same with notepad tc filters. The difference is the ip address is this: is the load balancer external ip and it's this part.

B

Okay, next for load balance traffic from localhost um here, unlike unlinked route, is used for remo for routing the load, balancer request traffic into antiquity and the target the target that ip address is the load balancer external ip.

B

The output interface is unsecured under the next hop installer service. What's wrong, ip address oas table 20 will handle the erp request of the service. Also ip address.

B

Okay, next for response traffic for response traffic, it has two directions: one for actually conjugate wave. This traffic is traffic. This request is from kubernetes node and the one for default. Router interface, this request the traffic is from a remote client and then we should add a higher priority filter to match to match. The traffic would request the traffic from country gateway, and then we can bypass bypass the traffic back to and to get away and the left traffic traffic is for the photo before the road.

B

Interface: okay, next one.

B

Next for cluster cluster ip for cluster ip is only used. It's only. This client can be from part of kubernetes node here used for kubernetes node, world class, ips drive, class ip is created, and if there is no a cluster ip so far, ok, then the unlinked router will be created for the class type. Another prefix is 32, that's the longest one longest mask for root. Okay! Well then, unlike this one add a new add a new class ip, then a new unlinked router will be generated and we can replace the old one.

B

Under the the destination block will will expand.

B

Okay, it's prefixed prefix is 29. Now we'll add a new one, a new cluster ip. It can discolor this. The current root atom can cover this ip address. Do nothing for the uh next one. Next one, the current load, atom kind of current can carry this cluster ip, so it is continued.

B

You continue expand and finally, and the service cell cidr will be.

B

Is- and there is always only one root- entry for rooting class, ip.

B

Okay, the next part is the new os pipeline.

B

The red ones are, you added, um the yellow ones are modified and the blue, blue ones are not changed.

B

I also take an example to explain the new oes pipeline um here is. The here is the case.

B

B

So now to be simple, I'll omit some tables table. 24 is the new table for transforming ice net of service traffic and destroying this drone is also a new drone for s-net for the first for the first packet of request.

B

This this table can do nothing. That's the plus neoplus class track.

B

Okay, the next next.

B

We can show that we can see that this is the new new action and this new accent.

B

This is for the first package of the traffic from remote, remote host client.

B

Let's see what what this table does, okay and we can see the state just stated here.

B

What what it does is, it will generate it. It will match the service traffic, uh this first package and the generator flow to table 70 file. This generated you should learn this learn the flow master response package of the node part.

B

It will. The learn flow will rewrite the destination map with the current package package. Source mac address also set a regular.

B

This means that the package will be output through slow gateway, okay, this and also set right, 0 16 to 1. This means the package should be output.

B

Another action is to this. Action is important. This means that this package requires snap. If the external traffic traffic policy is local, this one this might be set.

B

Okay, that's the other ways. After after the night at table, 42 the destination ip address is 10.0.4 under under the destination mac address user rewriting at table 17.

B

Okay, then, then go to table once 100 file and the first. This flow is a new added one and it's it's used to match the service package from gateway. Otherwise the service package will be matched by this one.

B

That's that's not what we want.

B

Table this is the new table.

B

At here say distant new table button. This id is not a new one. It's flow, it's fall table have been asked that table before now. That helps know the hairpin has not table id is 1 108.

B

According to the city states and rec 416, the accent, the accent is the city and committed to iceland drone here the ice knights address is a gateway at your address and then, after the snipe, the packet is output.

B

Okay for the second target for the second package, the response package.

B

When passing table 24, it will be transformed this destination ip will be recovered and when passing 30, when passing table 30, it will be transformed again this source ip will be retired and when passing table 31 according to its city, state, the packet will be reset will be sent to table 15 and then to table 17 17.

B

Will arrive when the pacquiao route table 70 according to its destination, happy address and its destination mac address will be rewrite rewriting. This output interface will be decided at a table. 18.

B

That's the that's the other way, but for because the their destination ip, the glass is required through the to the to the client. So now these flow entries will be will hit the package this entity flow. This follow entry is for this voyage.

B

This data is reply, the track tracker and the service traffic um it will be. The master package will be sent to 70 75 the 70 file. The 70 file is a new username table, and this table this table is how mentioned this table before a table certified and the generated learn flow is located at the table. 70 file, the first, the first packet of a remote client, will trigger a flow at the table. 70, and this flow is used to set destination. Mac, address, output, interface and output, output.

B

B

Okay, this uh as the second packet is not from gateway, so the first floor will not match it under the light for automatic okay and then then, the second package will return to the client.

B

The next is the is the third package.

B

The other routine, so not that the packing tool can be transformed either at table 24, but package 3 cannot because there is no connection contract connection track on s9 drone about the current connection.

B

And it can be only transformed at table 30.

B

In general, so there will be some difference for packet target 2 type 3 from packet, 2.

B

Okay, next, nothing different as the packages three is from gateway. It will hit the first floor of table 100 file.

C

B

At the table at this table, the fourth floor will hit the package as the as the current connection has been committed.

B

The first the first package has been committed for partner connection, so uh it's one it is going to be. The current package will be transformed without a committee again, and each source ip will be educated with ipo advisor after after the cc operation, and it will be output.

B

Okay, that's the laptop for another case is host. This endpoint is a host network. That's that's!

B

The the first difference is at the table. 70 and uh now we can see that at its destination address is the is on another kubernetes host kubernetes north and it won't hit any flow here. So it can only hit this flow and this flow is prepared for it.

B

You saw sat the detonation michael jackson's entry gateway. The mic address also sat beside the side. Iceland register.

B

But it's it's harping traffic, because it's from uh this from entry getaway and it will it will- it will be out of oas pass-through conjugate way. So it might. It must be.

B

B

Okay, the next uh for her for helping traffic, the the astronaut does not append to as the happy address of astronaut can't be attributed to it, because the anti-gateway will drop drop will drop the package if its destination is um so. This source address should be a different one. Here we we use our watch service versus ip address to to do to do it.

B

Okay, that's the here is the supported cases of the new and then this case, the the north pole. All of the balancer, his external traffic policy, is local and it's the point is our host network.

B

That means it will need the traffic. The package that needs to resolve its source like address and its destination is on horse net. Then the response response traffic, the response packet- won't be want to be rotted into oas. I'm doing this under the connection will be will be broken.

B

The no problem load balancer, almost the same and cluster cluster ip, is all supported.

B

Okay, that's all and questions.

A

Thanks uh thanks a lot for this presentation that was extremely informative. um So if your community has any question curiosity observation, please go ahead.

B

About part, one of part.

D

Two, uh so is there any plan to cover the use cases that we don't support today, I'm referring to the table. You showed at the very end those cases where the external traffic policy is local and the destination and point is in those network. Are we do we have any plan to address it? Somehow.

B

Yes, but uh uh how investigated some possible? Actually, I think I think it may it's not so so worthy to do it I can I can. I can use a link to tc filter to do some tool to add a mark to this kind of mark, but that will introduce another drone to tc filters. The current design of the currently designed for tc filters is stateless and that if the distributors introduce stateful, I think that will that will if and that that will make a effect effect to the performance.

E

The homeowner I tried to ask you in the in the years: could we could we could we use connecting tracker only for the case um that the backend server is a? Is a host network port.

B

That will introduce the city for tc filters.

B

E

What's so useful is that if it's only for the case of the host network.

B

um If we introduce the ct.

B

City operation for tcp orders that will that will take effect to all of our traffic. uh We only for the for this case if.

E

Why why is that? um Could we match ip first, let me see: could.

B

You imagine ip first.

B

If you connect, you feel access to a service app. Your address, you don't know whether it's a point is on a cdr or a host net. You don't know that, so you can't you can do that at roster.

E

Why we don't know when there was a housewife.

B

Oh okay, I know what you mean, but.

E

I'm not sure if other people here has any opinion now.

C

E

I I feel it's hard to say this. uh These are not um important case right.

E

I mean I actually I'm not quite sure, but I assume uh maybe there are some use cases for, for example, traffic policy equals local and then the back end is the host network port.

B

In my opinion, I assume there is a minority run.

E

What the hell, how do you know.

E

I'm not certain either um I'm just wondering if it's supported and if I really have a some service in house network.

B

We can also support it supported, but we will change some behavior and we can do as night for it and and then we can.

E

Yeah I saw that too. I think that at least probably that at least it's better than uh not supported but yeah, as you said, then you change the behavior. It's traffic policy, local, okay, god, how about us to do some evaluation on these two options? um It's that or uh planning trip.

E

If s that is much simpler yeah, maybe we can also consider.

E

But I'm not sure it's very it's not simple or not. Why not you you symbolize to potential approaches and then we can decide which way to go.

B

E

No, I just mean why not, let's look at two options and see if we, if we can, can choose one way to go.

B

No, I'm sorry like that's the adjustment. That's the different character has what you said all clear. Please point a slider or pick number to show the question.

E

Okay yeah, I can summarize my source of language.

C

Maybe there's a solution for this case. I think, maybe, if we use the the mark packing mark and the kind of connection mark to mark the traffic from the gateway interface and we do some special handling in the output chain. Maybe we could avoid.

C

Maybe we could just redirect the reply, traffic with specific connection map to back to the getaway interface.

B

I have tried this idea.

B

You said you transcend yourself, you make up. You add a mark to the request package about the response packet.

C

B

We need the connection.

C

Mark the connection mark is for the session, not for package. So if you, if you set the mark to the connection, then the reply traffic will also have has that mark. So we can match on that mark. But.

B

C

Be it will be a little complicated, it's a little similar to the avoid reaction case.

C

We can discuss this.

B

One and this one can be said either drawing 0 to not the measure that the drawing data cell with the holster, with the host drawing hosting that network. Drawing.

B

Sorry, oh, that is that this mark can be set at os pipeline.

C

uh Yes, os pipeline can set the packet mark, then we need to match the packet mark and set the connection mark in iv tables. Okay,.

B

Let's say yes, that's I think this supports. I see I think this can be possible.

B

Maybe I need I'll try this idea later.

B

A

Do you have a any other question regarding the proposed design here.

F

um I do have a quick question if uh cornell doesn't mind, um I'm looking at the the road number um 17 and 18 18 here um I was not noticing that if the client part is from the remote node and depending on the external traffic policy, um we may or may not need s-nets in two different cases. I'm I might miss when you present it, but could you explain a little bit on why that is the case.

B

um If I list all cases, the time will be out and all the cases- and I just.

F

B

Some typical cases yeah.

F

Yeah right I'm just talking about this specific case because, um for example, if we have a pod from a remote node um trying to access a node port to service now. Why is that? Because of the external traffic policy? We don't need a snap for for a local case.

A

I think young's question is about why, in the case, external traffic policy is local. You do not need snot. Is that correct young? Oh.

F

A

Oh, I I think it's part of the kubernetes specification but I'll. Let hong liang reply.

B

A

It's the specification, I believe, hong kong, that requires when traffic is local. It requires the traffic to come to the server with the client ip right to not be snotted.

B

Yes, yes, that sucks! Oh okay, I see.

A

Any other question we still have quite some.

A

Time, okay, I have uh two very simple questions, mostly as usual because of my ignorance, so considering these few use cases that at the moment we are not supporting, is this affecting in any way conformance tests? I mean: is there any conformance test that explicitly verifies if no deporter node for traffic works with external traffic policy, local in our cases or is conformant conformance tests are still passing fine.

B

I may not have a cat cat, I mean not look at what you said, but this is that you mean I have. If I have. I didn't. I mean test to carry current cases about known.

A

Our load balancer, um no sorry, I meant something different I mean. uh If now we replace q proxy, we completely proceed with entria proxy will the conformance test, like you know, the cncf conformance conformance tests that we already executed in our ci will they'll still pass, or will there be any conformance tests? That might start not passing because of these use cases that we are not supporting at the moment or.

B

um I have added some.

B

I have added some e3 tests for these cases, but to cover confidence test. I didn't. I didn't do anything about it.

A

Yeah I mean we don't take necess, maybe we don't have to do anything about it.

G

I suggest, let's just run over the conversations and see which case they are failing and let's investigate and uh if the conference set won't fit our design now and maybe we can add some new eq tests to cover and skip those.

A

Sounds good thanks, jochen and the other question that I had was whether this design as it is works for ipv6 out of the box or whether we will need some tweaks for making sure that uh um yeah that everything works. Fine with ipv6 as well.

B

Okay, how considerable about ipv6, but um the time is limited- and I have this- I didn't talk about it.

B

But how sporty.

B

At the beginning of the dc, at the beginning of this detail, I have considered about ipv6 and I'm and how tested and how tested about it. This case is that ipv6, but.

B

There are some there are some just. There are some issues, but this, but it depends on kubernetes upstream. So I didn't, for example, for how to list the kubernetes node network cdrs.

B

There are some issues about the dual stack for kubernetes service, so the current design is limited by but is limited by these issues.

A

I think okay, thanks thanks for uh yeah, thanks for bringing this up, that's good. You know, and uh at least from what you've seen are those uh issues that are already being attracted by the kubernetes community. Or is it something that maybe, if we want to have them addressed, we should fix them by.

A

A

I mean my question was more about this, so these issues that you have identified are issues that are already known to the companies community and maybe there's someone being addressed addressing them or.

A

B

A

uh Yes, yes, I mean you, you mentioned that there were some uh for ipv6. You found some upstream issues that were preventing you for implementing design for defining a solution for ipv6. I was just curious if those issues were already tracked in the kubernetes community or whether these there was no one. Looking at those issues at the moment.

B

This is all how notice that there are. There are issues tracking this. There is usually tracking this one, but it seems that it hadn't been resolved.

B

How I noticed this about the weeks about a few weeks ago, so.

B

I I think I need to I need to find it find a little later. I didn't remember the specific problem now, but uh I just remembered that so sorry.

A

No no problem, I mean I I mean you answered my question that it was an issue that was already known to the community and not just something that uh you find it. You find it for the first time. So, thanks for your answer- and uh that is all from me in terms of questions- is there any other?

A

Is there any other comment about the design uh observed today, a design that we discussed today? Sorry, I don't know why I used observed.

A

I'll wait 20 more seconds in case there is a any other.

D

A

Perfect apparently, there aren't any more questions, so I would like to thank hollyang for this very informative presentation, and the only follow-up question is: what are the next steps for the implementation of the of this design.

B

The next one is how to remove how to remove cookie proxy totally. The current design is that we use the endpoint of kuber api server to fast, fast, faster service or other other information from kobe api server, and this is not. This is not perfect because um reflects the information from only the api server.

B

If the api server, this mod are more than one it needs, we don't know what we don't know is classic ipa cloud ip address, so the next step. I think I I need to resolve and to resolve the issue of how to.

B

How to do how to handle the load balancer over kobi api server.

A

All right thanks uh thanks for uh thanks for this answer. Okay, and uh perhaps that is really all for today, at least for this topic- um anything else that the community would like to discuss for today's meeting.

B

B

I think you saw now.

A

Well I'll wait uh a few more seconds to check if there is any other topic that you would like to bring.

A

A

And it appears that uh it's really all for today, so I would like to thank again hong leon for this presentation. Thank again, everyone uh for attending this meeting and uh and well. The only thing left to do is to is to say, wish everyone a good afternoon a good evening or a good day and uh we'll talk again, we'll meet again in two weeks time, thanks everyone for joining and have a good one thanks.

C

A

Thank you, bye.