Antrea Community, 23 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Antrea Community Meeting 05/23/2022

Description

Antrea Community Meeting, May 23rd 2022

A

So good morning, good afternoon and good evening, thanks for joining this uh instance of the andrea community meeting, uh this is uh the last meeting for the month of may 2022 and in the agenda. For today we have a presentation of a design for a proposal for virtual machine support from um running mnd.

A

So I will probably just let them uh present and um yeah and then maybe after that, we'll have some conversation and open discussion. So um I don't know if uh who's going to present whether waiting on mandy but please go ahead.

B

Yeah hi, can you see my screen.

A

B

Yes, thanks a lot and today and I will share the detailed design for the external node and here's the agenda for today, and I will first go through the background information for this feature and focus done mainly mainly on api changes and controller side of changes and wayne will first will further show the details for the agent side of changes and here's some background and motivation in some workloads, like virtual machines or biometal servers are actually outside of the kubernetes cluster and they are not kubernetes nodes, but we still want to enforce our entry native network policies to these workloads and that's why we propose this design.

B

We also expect both windows and linux. Operating systems will be supported for this feature, and we have an example scenario for this feature.

B

Here is the example use case, and we have defined and entered network policy with stack like this, and we want to apply this enter network policy to the external entity, which has labels rho, equals db, and in this graph it is vm1. It has. The label matches the external entity selector, and I need to mention that this part is sorry. This part of angel network policy change is not supported right now and it it is the new feature we want to add it to the entire network policy and in the e-bras there are two rules.

B

You can see that we will allow the traffic from the external entity, which has labels row, equals front end and we will drop the traffic from the api block which, with this within this slider and so in this graph, the traffic from vm 2 to vm1 will be allowed, but traffic from vm3 to vm1 will be dropped with this entire network policy defined, and then this is our target and for the entry network policy applied to the external entity, we want to support it.

B

The the uh we want to follow the same same behavior for as the pre as the current entry in our policy, so we will support actions of allow drop and reject and the two and from field of the anp rule it can be epi block external entity and part of qdm for the egress, and this is for the.

B

It is the background information for this feature, and if this there is no question for this part, I will go through the uh the two api changes we proposed for this feature and the first one is.

B

We want to introduce a new crd called external node to represent the virtual machine or the benelmetal server, which is not a kubernetes node, but has an agent running on it, and it is the definition for the proposed crd and in the external node stack, we will define the network interface and for the inter network interface, you can specify name and ips or either of them and no no way expect. Ipsum is the must-have field.

B

If you, uh I will see later that, if you don't specify ip for the network interface, we will not generate the corresponding external entity for this external node, and I also need to point out that when we define an array for the interfaces here, but for the first release, we will only support one interface per external node and multiple interfaces may be supported in the future. Release.

B

Now another api change is the entire network policy. Api changes, as I have already pointed out and point out on the um example uh case, we want to add an external entity selector in the np applied to field.

B

So for the for, with this change for ap and p applied to field, it can be either port, selector or external entity selector.

B

Now I want to mention that users can use external node label in the external entity selector, and it is because that we see external node is a kind of external entity so that you can use the external node label directly in the external entity selector to choose the applied two vms.

C

Among the acquiring here, I think, even install anything, we do copy labels from install node, so it's downloaded here right.

B

Right right right, I will, I will talk about it in this page. Okay,.

C

Okay, probably I can wait for you uh one more question: uh when we support lamb space group, we will also support each other in the name. Space group and the group can be used for apply too right.

B

D

Yes, uh the answer to our question is yes, uh we will provide.

C

B

And okay, that is the api part of changes and for controller side of changes. The controller will do two things. One is to convert the external node to external entity. Another is to manage the enter agent info level life cycle and for the first one we will convert the external node to external entity based on the following rules and first it will generate the for the external entity name. If the network interface name is empty, we will use the external node name directly.

B

If the network interface name is not empty, we will instead use external node name heaven interface name as the external entity name and as tng has already pointed out, the all the external node labels will be added to the external entity, and external node name will be set. The external name field in the external entity.

B

The last one is that we will populate the external entity and pawn field with the information set in the external networking interface, and if there are multiple ips for an interface, multiple endpoints will be generated accordingly to the external entity, and here I have an example for this conversion.

B

When we define an external node, which is called vm1 under the namespace vm demo, it has the labels row equals db and it has the interface interface name can be specified or ignored in the definition, and it has one ip here for the generated external entity. It has the same name as external node, because interface name here is not specified.

B

So we will use the external node directly and for the namespace it they are the same and labels they are the same. And if there are multiple labels here- and it will be multiple labels here and we will, we also add a field called only reference to let this external entity to point to the to the external node. So we will know that no, which external node is, is used to generate this external entity and for the endpoints it is populated based on the interfaces field and another important field is the external node field.

B

It will be set the same name as the name for this external node, and basically the general idea is that we will generate an external node, an external entity, for each interface defined in the external node. As currently, we will only support one interface per node, so currently, external node and external antenna is one two one mapping.

B

Yeah, uh if there is no question for the conversion here- and I will go ahead.

B

Another part of change from controller site is the entry agent info lifecycle management.

B

Basically, controller will watches the external node, create and delete event, and based on that, it will create the entry agent info and delete the entry agent info, whose name is the same as the external node name, and so with this change.

B

It is the enter controller who create and delete the entry agent info, and it has the entry agent who will update the content of this entire agent info and and the advantage for this change is that we can only grant the update privilege for the entry agent info and don't grant the create and update privilege to um to this vm so so that it is a more secure way from the vm perspective.

B

Yeah I have talked about the api changes and the entry controller side of changes. I'm not sure if anyone has any other question. For this part.

B

Yeah, I will, if not, I will let you to further go through the details for the entry agent side of changes and share the detailed design.

D

Maybe you can continue yes from agent, actually, the testicle of two realization wise for realizing external node, and the second is for realizing internal policy, which was applied to the target, where all we could say the external node.

D

So one agent says starting up. We show the first secure environment variable with the external node name as a variable keynote name as what we have done on the container case, then uh after agent is that it you know it knows who it is. Then um agents will get the external node from the kbps server uh with a filter by its own name.

D

So uh for each vm it will not get other uh vms, external node information and then uh agents who can get the network interfaces configured in the external node and to try to guide the first responding network interface from the health and then agents will try to realize interface on os for the details about how agent realized narrowing based on last, I will uh use the next page to introduce, introduce and after the agent realize the interface, the notification unless it maintains a new external entity.

D

Interface can fake in this stock, which can be consumed by internal policy.

D

Now the internet policy realization it is similar to the existing workload until agent just build watches, internet policy, internal objects from an intro controller and it released the external entity from interface law, which was created about external neutralization and then unsure agent can realize the internal policy rules on oas.

D

So from the page we can see the the normal workflow. um Actually, we are focusing on these two passes and monday. Maybe you can go to the next page.

D

Okay, this page, I was focusing on the uh realizations on everything phase. uh First of all one case, uh the major difference from the container case is that so we should maintain uh um manage. The whole network is what we are focusing on, so uh to not break the existing processes uh consuming the networking we will try to use a new network interface can configure with the existing name and ip and max and routing of the never interface configuring in the external node.

D

So when, after an agent finds interface by the ipl drives on the host, it first attached the exit first server name, the network interface. To, um for example, I use the pinik zero and in the unit diagram, and then they uh create the pinik zero to os. Actually, this is the physical interface uh on the os at the vm. Then agent will try to create a new os internal part which is configured with the it is there virtual it is.

D

There is actually the interface name before the data agent and then until agent will move the ip mic routing even denies configurations to the either uh gth0. I mean the old internal pulse from the open entries until agents will try to.

D

I should will try to configure some open flow entries to ensure the packet can be forwarded in from the eth0 to pin x0 out from pinx 0 to e200 directly.

D

We don't have some ip pipeline consumptions, but that's the nice details and for the interface config part after the uh parallel parts they are added onto the os. They create an interface config. I mean one interface: config is used to map the pair port. So uh in the new type point we can represent external entity. We have to maintain the purpose. We have two overflow ports, one is the internal passport and another is the mapping uplink port and the interface config name is actually the.

D

Actually. It is general. I mean the the name defined in the external networking space. Then uh android agent also set up some open appliance for the external node we have ipv pipeline, but for the external node ip pipeline we are only focusing on the egress and the ingress, so we don't have their two hours forwarding. As I mentioned, because of the pair pause. We will try to forward package from one to the pair directly and as we should maintain the host network, we should also maintain the non-ip package in the os pipeline.

D

So we we need to introduce a new table focusing on the non-ip packets, but for this new pipeline we directly forward packets from one to the pair with the changes on the external node for osf path. The new pipeline definition has only four tables, and I listed the table id and mapping in in this page.

D

That's the main change for the agent size, um I'm not sure. If you any questions.

E

Yeah, I have a question so um so. For example, on the bare metal use case, there could be multiple interfaces and it could be possible that we have different vlan settings for different interfaces.

E

Do you need to reflect that in the open switch configuration project configuration.

D

um By now uh we in our first release, we only want to support single interface and we're talking target thing at the ram case. So maybe we'll have a few thoughts about environmental case, but when you, when we maybe uh have some thinking about it later,.

E

Okay, so the workloads actually are the applications inside the vm, so you only need to enforce the circuit process on the vm interface on the bridge.

D

E

Okay got it thanks.

D

Oh one more I want to mention is that, since we are uh focusing on the egress ironing rights, uh we still use os contracts, but only one os contract zone is used to maintain the commission space. um That one thing I want to mention for the security part.

F

I have a question you might have talked about it, but I may miss it. I see in the first in the about picture, the physical interface is called ether 0, but in the picture below the the physical interface is called the penix0.

F

So where you do a interface rename when.

D

Attaching um actually.

F

D

First, do the rename on the interface and attach it to os. But after if the user delete external node, I mean, if we want to remove the interfaces from os, we will rename the penis 0 back to it. H0. I think that's your question right.

F

Yes, so this is maybe to keep the origin name to be the one and used by the sys. The operating system.

D

Yeah yeah crisis: um this is because uh well synchronized. Maybe some other processes are listening on the ets zero, but if they just use the region name to os and then create a new report with a different name and since the ip and ipn routing migrated from the existing interfaces to the os internal parts, the uh the other processes listed on the um the pth0 might be broken.

D

So that's the motivation why we need to rename the interfaces and try our advice to keep the ip and mic are configured on the interface with the same name.

F

E

So I also have a question, um so we are using just one bridge. We are not using multiple bridges for multiple interfaces right and then yeah.

D

E

D

Running springs.

E

And then you have, we will have some flows to cross the th0 to p nick 0, and the package will not, for example, coming into each zero and coming out from team nick one. So you always have photos to uh to exchange the package between the matching virtual interface and physical interface.

D

Correct, as you may know, that for container case uh when the packet engine enters enter the os popular in the table zero, we only set the source smart register mark just to know that where the packet is entering live from, but in external case external case in the first table of ipad line, uh maybe not just the the first table, you know as pipeline.

D

We are already citing the target of ports in a register one and uh in the last table we only output the package to register one which was passed in the table in the first table. So we don't need the letter to order three table uh to forward package according to destiny or destination mic, but we don't need to analyze in external notice.

E

C

So winning um I remember in nst design uh we choose to attach the fader interface to the bridge with a wizard pile, and I remember one reason to be for the. I think it's for the who says restart case, um I think there's some difference between internal port and the wii supplier. In that case, I'm not really sure. I'm not sure you know about that or have you solved it.

D

um We used to have some short discussion about this design, but I think it might introduce some some issues because we need to maintain different name spaces for this case, but I didn't think uh deep for for this uh solution, because we think this workflow, maybe we could have some evaluation if you think.

C

It is needed yeah yeah, maybe maybe check, what's the reason that we used to use we supply. I remember some reasons for that.

D

Sure I will check with the other device about that is done. Why is relevant to namespace? Sorry, um according to my original discussion with them, they said that they want to keep the name of the uh the pair uh the same as the existing interfaces, so they want to use a name base for each uh interfaces.

C

Yeah yeah, maybe they they do that uh for their purpose. But in your case even you just put a visa power in the default name space. It should be the.

D

C

As the internal problem.

D

Yeah, it is simple, but actually I think uh asians should maintain the uh the the I mean that if we want to use a namespace to configure the network interface with the the same name, we should require other interface other processes to go into their namespace sure. But.

C

I'm not saying uh we must put the visa power in a different name. I'm just saying: could we replace internal poll with we supper even when we put all the device in the same name speed the default namespace.

D

But so I I didn't just see the advantage for this path. Yeah.

C

That's what I forgot so, let's, let's I hopefully you can you, can you can double check? um What's the reason they use with isabel um in nfcd?

C

I think there are some reasons about uh open with stockist sure I will check.

D

A

And one more small question for me: do we support also dynamic addresses on these interfaces? So, for instance, if ethernet 0 is dhcp, when we move it to an internal interface, will we still support it?.

D

uh We did some uh tests on the hdcp since we moved the interface last internal, using the same same name as the existing business, and uh actually we can capture the dhcp package after we move the ipo guys to the uh to the right internal parts, and we see that the digipeat client is still working.

D

But since we moved the ipr aesthetically on the os internal parts, so the the cp client only renewed the appeal, drought and not to request the new ones. That's what we found in the test.

A

Yeah, I think that that is good enough right. That's what we needed.

A

Thanks for your answer,.

D

D

um Actually, um I only prefer this this product design, so I don't know if other questions.

A

Well, um I believe that uh we probably want to open uh for conversation with the entire community. That was uh that is, like a uh a let's say, that from a design perspective, probably a it's from this presentation, it's less than what it looks like, because there are many changes that are going into the obvious pipeline for supporting uh vitro machines.

A

So uh one question that I had is: what is your plan for implementing the changes in the obs pipeline? Are we going to treat external nodes as a special case, or are we going to develop a completely different pipeline for that or what is your? The solution that you have in mind.

D

um First of all, the code changes we are using a visual gate to control the uh branches and for the github we are creating a beta branch to maintain the conductor.

D

All the changes are ready, and so, uh since we are supporting the flexible pipeline, I mean for the osw path changes since we are already supporting the pipeline, so one agency is running on them, also a parameter server.

D

The pipeline only is introduced for external cases, so we don't have the case that the os purplis uh full container and external uh existing on the same same agent case uh and uh actually for the uh code plan. We plan to have this code in the next release, for I'm sure, um maybe 1.8.

D

um So we hope we can complete all the code and merge the and then we convert the pitch branch to main branch.

A

Oh okay, great and uh well. I would like salt to solicit more questions for the community. Any sort of questions please go ahead.

A

uh Janjun has a question on the chart uh regarding specifically table 14 no ip. I was also curious why no ip john john suggests that maybe should be non-ip.

D

So there's already amazing in the code.

A

Okay, so if there's another question, I would like to seek a clarification on the relationship between external node and external entity. uh Can you go back to the slide that mandy presented so um yep perfect this one? There will be a controller. I assume that reads this information and creates the external entity uh resource now. uh Is this just uh taking data from one entity and moving it into other data, or does it also involve a searching for a vm using some provider.

B

No, we will just monitoring the external node crud and create update or delete the corresponding external entity.

A

Okay, so we are not searching for the actual vm or bare metal node that has data interface right.

B

And if, if user can fix some invalid information, assuming in the interfaces and when the agent site monitoring the external node, it will search the interface by ip it. If, if it cannot find the corresponding interface- and it will not do any further procession and we can see some errors from agencies.

A

I see- and uh I have another question- sorry in terms of information- the resource on the right external entity. What kind of information is adding on top of the external load, because I see that uh I it seems to me- I'm sorry. Maybe I'm mistaken here, that there are exactly the same attributes just in a different order.

B

uh You mean the owner reference here.

A

No, no, no, the only reference is fine. We understand it in terms I'm talking in terms of specification. It seems to me that we have exactly the same data right there. There are no additional data in external entity, uh so there are no additional yeah right.

B

Yeah yeah for external nokia note case the we will only set the endpoints field for external entity here and for for the definition here. They are slight slightly different because the interface um network interface we define it can has one name but multiple ips for a interface, but for the endpoint existing definition is one name and one ip. So if there are multiple ips here, you will define an array with multiple endpoints in this field.

B

Yeah we will not set other fields. uh I know you, you mean that external entity has other fields other except the endpoints, and for for external no case we will not set those fields. Only endpoints field will be set.

A

Okay, okay got it thanks.

C

A

I can give.

C

You a little contest here. uh uh Actually we have two two types of uh install entities.

C

I think what mundy and the winning shield here is just one type, uh these types created from install node, basically, so when we create a cell anything from install node for that when finally can be another type, it's more like in this public car support with this cover the wamps and the latest service in public health, but we're doing the wrong agency in inside the room, um but we just so it's not easy to note, especially so known, still means uh know that we really wrong when we switch an agent until agent in the node, uh but in the 18th code.

C

In this case I mentioned that we we don't really run wrong agent in the agent is when, but we just discovered when the cradle is not entitled for the win, um so you say well, actually we are introduced, it's only a before historic node, so other times only for the agent. In this case.

C

That's why you don't have this interface or you know, multiple interface or it's installed on node crd earlier, but we just installed. We just have install entity, but since we start to add the agent model for when and now we we decided, we need a new crd to express again with agent and we call this toner node, but still uh for most of our code to handle policy and uh other you know, data paths.

C

Logically, we just want to use external entity, I'm not sure in my my my.

A

uh That is correct. That is correct, and that is also I understand now why there are two we are using two crs, but uh I have a final question so do for the external entities which are owned by an external node?

A

I assume that we also have controls in place to make sure that endpoints and other spec information are not accidentally modified.

A

I don't know if I was clear.

B

Yeah, sorry, sorry, I didn't catch the question. Yeah.

A

So, let's assume that I am a clumsy user, so I have an external entity which is attached to a vm uh without reference by an external node. Then I decide to go to two end points and I modified the ip addresses here in endpoints now. Will there be a mechanism that resynchronizes again the external entity to the external node, or do we have to uh do? We have just to tell users that you should not go and modify this.

B

uh You mean uh you, if user accidentally identify uh modify the field. Yes for the current in implementation. I will. I will correct this after in the reconciliation, if the I mean, if the extern and if the controller restart, we will reconcile all the external node and he can change it back.

B

But during the controller, when the controller is running, we didn't check. The external entity appear periodically to make sure user doesn't make any uh modification.

B

But actually, I think, as as we have already pointed out, two kind of external entities- the previous external entity, which is for the agentless vm, and it doesn't have the only references here because they are not created by external node and so for for we we hope the user should not change the external entity which are.

C

B

By external nodes, but if they modify this, we can only correct it when, when the controller is starting in the reconciliation, yeah.

A

Sounds good but let's assume that a user does this, because they are probably not that smart or there will be no impact on network policy enforcement right. It's not like they will change the way in which network policies are enforced if they modify like adding endpoints here.

B

And for the over as part of.

B

Part change because we monitor the external node, it will not have any impact, but for, if you change the external entity part because the agent side network policy controller is is watching the is actually watching the external entity selector. So if you modify the external entity by accident, I think it may have an impact.

B

A

Okay, so uh now sorry, I don't want to steal the meeting, but I have to ask another question. So do we have to worry about the case where the user is not clumsy but maliciously tries to modify this data to impact the way in which network policies are enforced?.

B

I'm not sure, maybe we can investigate it to see if there is a way to forbid forbid the users to change the the external entity generated by the anchor controller.

E

B

E

Will be the users because, for example, the name space is vm demo, so the users are in uh are referring to the kimnes that mean or I referring to them- the name space manager of william demo, because maybe we can assume that him nation, the namespace, managed manager at the guy who are managing the security policy and they will not maliciously override the information in the namespace and the workload user inside the vm can now switch to kinase and has no authorization on managing these information.

A

Yeah, I think that's a good point. I believe that uh you know these are kubernetes crs, so even securing those should be managed with the kubernetes airbag so yeah it should not be. It should not be a big deal. It's just. It's just a matter of making sure that uh these uh that you know that access to external entity as the proper airbag controllers.

E

Got it so maybe later we can have, for example, validating new metadata in webhook. If there is only reference refer back to the external node, maybe the validating webhook can reject any modification. That's not from the from the entry controller.

A

All right, sorry for sorry for asking so many questions I know and well. I do hope that somebody uh somebody else from the community has any question here.

A

So please, if you have any additional question regarding this feature, please go ahead.

A

Waiting a few more seconds.

A

All right it appears, then there are no questions and uh we're looking forward to see this feature in uh included in andrea 1.1, and I guess that now we can move to open discussion. So if there is any topic that you would like to bring up any bug that you would like to discuss anything that you like to complain about, please go ahead.

A

Okay, so just a quick update from me uh last week there was a cubicon uh in valencia, spain. As you know, we had uh an entry office hour, uh chan, chan and myself hosted it.

A

We didn't have many participants, but at the highest we we had like 15 participants in in the meeting. We had a small introductory short introductory presentations about a presentation about andrea features, the scope of the project, the status as a cncf project, and I have to say that was pretty much it.

A

There was still very little user interaction as it typically is for these uh office hour sessions, but I believe that there was already an improvement compared to the previous edition in terms of community participation.

A

So that's all- and uh I will just wait a minute for uh check if other team members have anything to discuss, and otherwise we can call it a day.

C

uh Software, you didn't go to could be calling in person.

A

No I'm just curious. Okay, I did not go to cubicle in person. Eve was there from our team. Susan was there mother.

C

A

I personally felt that I couldn't I mean I was still not okay, with going into a room full with thousands of people and and also to be honest with you. I did not have time it was pretty much like taking a week off and I did not.

C

A

To take a week off- and you know, I was planning to travel to palo alto and in the end of june to see the team, but uh they closed the campus. So I don't know if I can, if I can travel anymore.

C

Oh really, I didn't know that.

A

uh There was an email uh on friday that the vmware campus is uh closed to non-essential uh employees but yeah, but that's pretty much it so.

C

Yeah I look for the some. Sometimes you will be able to come.

A

Yep anyway, so I believe we can stop the recording and uh that's all uh for today.

A

So uh thanks again for joining and uh and uh well- and I wish everyone a good day a good afternoon and if you are in the west coast. I wish you a good night.

A