Antrea Community, 10 Aug 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Antrea Community Meeting 08/10/2020

Description

Antrea Community Meeting, August 10th 2020

A

B

All right anton and the agenda was not yet full. When I went to sleep last night, I'm checking on the slack channel if there been updates.

C

Oh yeah, we have a few topics. I think there.

D

C

Yeah, I think people agreed that talking about the progress on the cloud support side was a good idea and uh raul who's been working on this a lot has agreed to join the call.

C

Perfect and I think what else did we have what people join now.

C

Yes, cody wanted to give an update on on the an update series about the upstream netball network policy. V2 meetings perfect. So we can do that as well and I think someone else also suggested talking about plans for openshift support. So I don't have a lot of background on this, but.

B

That's right, that's right! That was a looking at this luck channel that was puja, suggesting it right. So, let's uh we have quite a few people on the call we have 20 people on the call right now. um Recording us started. So I guess we can start with the first topic on this agenda and I guess that would be raul, providing an update on andrea cloud support if I'm not mistaken,.

B

Raul, the florist.

E

This is rahul. This is my first. This is the first time I have joined this community meeting and I'm kind of unprepared, but I'll try to go through uh genuine. If you can help me was aks eks and gk ever discussed in the meeting and.

B

I think we can assume this is the first time we talk about it from a community perspective, yeah.

A

Let's take everybody.

E

To the basics, rahul, okay, so it was never discussed. I just wanted to know the expectation, because I didn't get a chance to prepare.

F

E

F

Talk about any cloud support specifically, so you can.

E

F

E

C

Yes, so here I I I let uh raul know about this like 30 minutes ago right. So it's just like an informal discussion about what we have so far and recent changes.

E

Okay, all right, uh so with respect to cloud, we started the effort with aks engine aks engine is an open source project from microsoft and which helps in deploying a k-8 cluster in azure.

G

E

So with that, as part of that effort, we integrated entria as the supported cni, which can do both uh cni chaining mode, in which uh azure cni can be changed. uh We can be chained with uh entria and andrea can enforce. Network policy only and azure will take care of the networking part.

E

We do support the regular and cap mode in aks engine 2., so we started with the regular encap mode in aks engine, and then uh we did a cni chain in aks engine with azure cni, so that is just used to enforce uh network policy and the advantage of doing the chaining was we wanted the networking to be taken care by the uh underlying cloud cni like azure cni, so that the pod actually gets ip from the v-net rather than an overlay uh ip.

E

So it has advantages with respect to flow visibility and other stuff. I won't go into the details of it, but that's where we started with aks engine. I hope this. This background is good enough and I can proceed.

H

E

Okay, okay, so then we wanted to do something similar for all. Other clouds like aks engine is an open source project. It's not a managed service.

E

Azure provides uh something called aks managed service, which is aks, uh aks, azure, kubernetes service, sorry, and and that you could use the azure portal or azure cli to deploy a cluster and the cluster would be managed by uh microsoft.

E

So it's a managed service so uh with uh starting uh with 0.9 relays, which is happening. I think this week we do support uh cni chaining with azure uh cni, which means basically, you deploy a cluster using azure managed service, which is aks, you'll choose azure cni as the networking plugin and today they support calico and azure, as the policy plug-in in their managed service like aks engine is a different project. An aks managed service is a different project.

E

Aks engine is an open source, so we can contribute and add our cni's in that project, but aks manage service. uh It's it's managed by uh microsoft, azure folks, so they base. They just support two network policy cni, which is azure cni itself and then calico, and I guess uh cody can uh probably share more details. We plan to add entry as one of the supported cni uh in aks engine aks manage service also, which requires microsoft support.

E

But uh what we did is, uh if you just deploy a cluster uh using azure cni and don't select any network policy, you could very well apply entria uh yaml out of band uh once the cluster is up and running and entria will do the uh will enforce network policies uh only it will operate in a network policy mode.

E

So we did it like out of band, you deploy a cluster and then you apply entry or yaml and we would open it in a cni chain mode wherein azure will azure cni will take care of the networking part and then entria will take care of the network policy part, and once we get this endorsement from microsoft, maybe our cni would be listed as one of the supported cni on the portal itself.

E

So that's where we are with 0.9 release. We would support aks managed service, which would be an uh which, which would require an out of band.

E

You basically deploy a cluster and then apply entry or yaml out of that any questions. So far on the azure cloud before I move to vks.

A

Somebody community wanted to do this today. Is it possible? Can you give them where they would go to um use antrio with aks engine, for example, uh sure I.

E

Can share the screen. I have the documentation too.

E

You guys can see my scene right, yep, okay,.

E

So master documentation, so we do support dks, gke and aks. So let me go through the aks documentation uh so.

F

Aks engine oil- I think you asked for aks engine right.

E

I started with aks, which is azure cloud, so I'll just complete this flow, and then we can move on eks, okay,.

D

I think cody was asking about the aks management service.

A

Yeah, so I wanted, I wanted to point out both actually so so go ahead. Rahul this will. This will work.

E

So this support uh I mean like this- would be enabling 0.9 release right now. It is an amazing master branch. So, basically, what you do is uh there are some prerequisite that you uh install azure cli and then uh create a resource group. uh Sorry create an aks cluster and then specify just the network plugin as azure.

E

This will deploy aks or kubernetes cluster on azure cloud, fetch the details and then see the cluster is up and running and then to enable entria in a cni chain mode. We expect customer or user to apply this entry or node in it, yaml, which basically preps up the node in such a way that the entry cni can operate uh can operate in a cni chain mode. So we modify some settings on the worker node like change, azure cni to operate in transparent mode.

E

Azure cni supports like two more bridge mode or transparent mode, so for entry or cni to work with azure cni, it has to operate in transport, uh transparent mode. So that's what this uh node in yaml does it's, basically a script which basically toggles the mode and then with 0.9.0 release. Once this is done, you can apply the entry yaml and once it is applied, uh it will operate in the cni chain mode because uh entria aks in entry, aks yaml. We have set the uh traffic mode to be network policy.

E

Only more so user doesn't have to update any uh parameters or config in this yaml file, and this just shows that there's a node in it and then the entry I agent and the controller is running on this aks cluster and one last thing which we need to do is uh once you have so there are so once you bring up the aks cluster right. uh There are few pods like core dns dashboard q proxy.

E

They are basically uh they are up and running with azure cni itself, and once you insert uh entry or cni entry is not aware of this pods so that uh and it and there would be a there- could be some deployment also which entry may not be aware of.

E

So what we recommend is once you have uh installed entria, then you should restart pods in all namespaces, be it the cube system or be any deployment on any namespaces so that the parts can be managed by entry or crew and entry can enforce the network policy right now, it's a manual step. We expect user to uh delete those ports which we're already running without entry uh going forward.

E

We plan to remove this step also in a way that, as part of node in it itself, we would restart any existing containers which are not managed by.

E

Entry: okay, I'll, take some questions here.

H

So cody, uh how far have we gotten with like getting an official support with microsoft? Because if we end up with a you know, we we're in like at the initial stage itself, you can uh supply, saying that you know entry will be the official or andrea will be the cni for this cluster. Then we don't need to uh restart these pause, because nks would be there from the beginning.

A

Correct so what we would like to see happen and we've requested, is that microsoft provide us a way to.

A

Have a hook during the bootstrap of their cluster so that we can ensure that our cni is installed before these other pods get started. So we don't have to go through this whole restart process.

A

We have a similar, maybe not exactly related, but a similar situation in as well, and is something that any type of a managed, kubernetes service provider you know, needs to ensure that all the components of the cni have been installed before we actually start the um you know start immediate pods, so that we can make sure that they get. You know plumbed with the right cni and um we're aware of those pods, so we can apply network policy.

A

So that's that's an ongoing conversation, we're having with microsoft, hopefully we'll get that resolved in the next week or two.

F

E

So, just to show you what I was referring to so this is the the azure cloud and they offer this aks kubernetes service.

E

Here you can create a cluster unit is cluster using just specify the resource group cluster name, what version of kubernetes you want and so on, and how many nodes do you want in your cluster uh this?

E

They support both linux and windows, worker nodes and with respect to the policy network policy like I was mentioning they just support, calico and azure, and when the eventual goal is to have entry as one of the supported network policy, cnn.

E

Okay, this is with respect to aks, managed service and there's something similar, which is called aks engine. uh Here we can contribute and it supports um a lot of uh cni's.

E

Let me just let me show you that too network c okay, so here with respect to network policy, uh they support calico, celium and entria, and we added entry as a supported network policy cni in this aks engine open source project, and if you deploy a cluster like if you are deploying a cluster, the way you deploy a cluster is you specify a json file in which you select your network policy plugin and network plugin like entria or azure or whatever?

E

And you just apply this json file you on aks deploy and it will deploy a cluster for you in azure cloud.

F

Now for question here, uh so you modify this page to add andrea or it's. It's.

C

F

Important you should put, you should try to be the first one right. Just follow the half are ordering. We are the first.

E

uh These two were there, so I just added one here.

E

I'm like, I would have to move it.

E

But yeah, so this is how you deploy it and then, as part of another update, we are uh planning to us enable windows, support in aks engine 2., uh louis and meaning they're working with me, and hopefully we can get windows support in aks engine.

F

So, for a case engine reminding me, we support latter points only mode already. Well, we just support uh we we have to uh to both the policy and.

E

Support the modes we do support. If you select both the network- plugin sorry network, plugin and policy plugin as entry, then it will operate in the uncap mode. But if you just select network policy as entria, then we operate in a similar gene.

F

I assume there will be some validation, for example, you cannot do uh that policy within trial for network plugin to be a category.

E

Yeah yeah, so that's validation takes clear. I think so we do cni chaining with just azure cni and if you specify any other cna, it's going to prove an error got it.

A

Can you speak to if somebody wants to do a a diy or or build your own kubernetes cluster in microsoft azure? Can they use antria for that.

E

uh Sure they can use it yeah.

A

E

A

Would they wait, would you recommend policy only or in cap mode.

E

uh I would okay, so if, for policy only mode, if you are operating, if you are bringing, if you are trying to do it in your bring your own cluster, then first the azure cni needs to be installed right uh and then I'm not sure if azure cni uh can be. Let me.

F

So cody for your question. uh At least we don't support anything uh uh for diy cluster, for lighter policy, only mode.

E

Right now, yes, you do.

A

But but we would support but to clarify we would support the ncat mode.

F

A

F

Time mode and maybe rahul you can confirm, I'm not sure it's low income mode and probably not in low income. Do you think hybrid mode can work in a case or not. Maybe not. I.

E

Don't know, hybrid mode cannot work in that case, because uh azure is l3 cloud within the same subnet. The packets are always sent to the default game. Okay,.

F

uh High curricular, maybe low income can work by the way, never validate right.

E

Yes, so I was evaluating that option of supporting no end cap using the ccm module, which is cloud controller manager.

E

But then I I didn't complete that investigation, but it is possible to do that.

E

H

E

Caveats by the way, uh with respect to uh it might, I might be have I have to go into the details of it. But uh if there are, there were a few caveats. If we have to support it.

E

Okay, uh was there a question.

A

E

Sorry I didn't get it. I.

A

Was asking if there were any other questions on azure? I was going to have you give them an update on.

H

A

C

Next, uh we can't hear you anymore cody, but I think he's saying uh if there are no more questions on azure, we can probably move to.

A

Move to amazon, absolutely yeah, sorry about that, can.

C

You hear me now: yep.

E

Okay, uh so this completes uh um like the aks uh azure support, azure cloud support, which is via aks engine and aks manage service. uh Next is eks. Eks is aws cloud. uh We supported this for us. We supported cni training in eks.

E

Let me just confirm the version I think 0.5.

E

Yeah 0.5, so the uh for aws cloud uh eks is the is the managed service, uh kubernetes managed service, uh and we supported the network policy only mode wherein we we do cni chaining with aws cni and entry, and this support was added in 0.5.0 release itself.

E

uh The process of doing the cni chaining is kind of similar with what I have discussed for aks um cody. You want to talk about our engagement with aws folks, so I just go into uh what was supported earlier and what kind of enhancement we are doing for 0.9.0 release.

A

Absolutely so um in aws or or amazon, we, there are two different approaches that we've been pursuing.

A

One is in eks and being able to have mantria be supported in policy only mode where we are chained with the native eni cni, that amazon provides.

A

We also have been investigating in eks capability of running as a primary uh cni as well and I'll I'll discuss. Why um we're pursuing that in terms of the advantages it would have for a user in just a second.

A

Additionally, we wanted to make sure that if you built your own cluster in amazon on ec2 that we could support, you know in-cap mode and potentially some other modes as well, and so, uh let's start with the managed, kubernetes or eks in amazon,.

E

Yeah um we are breaking up in between.

A

Something must be up with my internet. Let me let me uh let me move to a different room. Maybe it's my wireless or something. Is that any better yep?

A

Okay, let's try this this right here, so um we're going to start with the managed eks, so policy only mode.

A

We wanted to be able to support chaining with the native eni, cni, and so um initial tests showed we had a a race condition. That's not necessarily a a problem within antria. It's actually something that's common to, probably all the cni's that would need to chain in any situation. Basically, the the issue is um we, like many other cni's, install the actual cni binaries that kubla calls using um kubernetes as a deployment mechanism. So basically we have a daemon set uh that that also deploys our agent, but can also, as.

H

A

Of the the deployment as an init container can deploy additional binaries um that are that are used for the interface for cni.

A

um If those binaries aren't present and registered correctly, um you can actually if a cluster comes up, pods can get scheduled and started um and they won't be registered with andrea right. Andrea won't be responsible for some of the provisioning that takes place, and so once andrea comes up. Things like network policy, for example, wouldn't be enforced until those pods are restarted in and plumbed with the appropriate network interfaces that andrea, then can enforce things like network policy and and and we've called this out um to amazon.

A

As you know, this is a potentially something that we actually, as um both amazon and and the entry community need to take up stream uh to discuss as a potential issue with the way cni is used in kubernetes right. We need a way for the kubernetes framework to if multiple cni's are being specified or maybe a way to specify that multiple cni's are being specified in terms of a chaining mode, ensure that all of the initialization has been done for all the cni's before pods actually begin scheduling.

A

So it's almost a an update to you know whether or not you know nodes are ready for for scheduling um in the interim. um We've worked out a plan with amazon, where we can um restart um any pods um that have been scheduled um once the the cni initialization has completed right, so that we can avoid this. This race condition, if you will, where, where a pod could have been scheduled um before andrea, had a chance um to initialize all of its components and be able to enforce policy.

A

What we want to ensure is that, if, if we're making a guarantee, that policy is being enforced um by antria, that we don't want pods having an active, you know network connection uh to the cluster to the world right without for any given period of time without that protection, and so um this is a mitigation right now that we've put in place that we can go ahead and begin using the cni with amazon, but with the plans to you, know, fix this upstream um later on.

A

uh Please the the engineers that are on the call, please correct me. If I, you know misstated any of this or if you wanted to add some additional details to what we're doing in terms of policy only mode with eks.

B

C

You were correct, yeah, I don't know. If raul wants to talk about the solution we have in 0.9 the demand set. We have to uh to to restart the.

A

That would be great.

C

E

Yeah, I can talk about it, uh so let me go over the initial support and what enhancement we are doing in the guess, ek, sorry, so, yeah with 0.5.0 release, you could just apply the entria pks yaml file and we expected a user to modify a few uh parameters in entry or config file with respect to default mq uh and the service sider. So this page is not updated.

E

Okay, so yeah, so with the 0.5 release, we expected user to update the default mtu and service sider.

E

Before applying this entry, eks yaml in 0.9 release, we have removed this extra configuration in a way that the default mtu now would be auto discovered. So once you apply entria, it's going to figure out the m2 empty of the primary nic and will set the default and I'll adjust the default mtu. Accordingly, user doesn't have to modify this argument uh and then the service sider, uh our parameter, is not needed with entry or proxy. So in 0.9 release we are enabling.

E

uh Proxy uh with the network policy mode, so these two uh uh arguments are not needed.

C

I mean like this service cycle.

E

Was kind of error-prone in a way that there were a few cases where people miss updating the service cider and then there were random packet drops and so on. So we thought it's better to remove this manual configuration.

F

E

F

This just to just to add the some more information since someone said there is many needs to support upstream cover proxy.

F

uh Just as john who said in 0.9, we will switch low income mode, hybrid mode and the policy only mode to use until native proxy, and it's no more window monitors service data parameters for volts modes and also means for with no more requirements, parameter and because gk is in low income medicine roughly top later, and the foreign sdks is probably only about both reuse and generative proxy from their.

F

C

I think isn't it just eks and aks right now. I think we haven't done gt. Yet we have an empty.

E

F

Okay, sorry, we we do support gk with low income mode right.

A

Oh we're getting we're getting there yeah we're we're focused right now, though, just on amazon.

C

Just look at the yaml and we don't enable entry a proxy right now.

E

Right, so that's what I was saying: the entry of proxy or gk ngk yama is not uh toggled really.

F

Then we should fix that. uh I think shoes, pilots actually remove the curry project, support for non-encampment.

E

Let's take it offline, okay, uh so yeah, so we remove these two limitation and then we can operate in a cni chain mode with the aws cni and inferior cni will operate in just the network policy only mode. There is a documentation which talks about the cni chaining, which is the policy only mode. If you guys are interested in looking at how we did the cni chaining and what we call the l3 mode. This is a good start. You can take a look at this documentation.

E

uh Let's go back to eks, so there were two things which we remove and the third thing which uh cody was mentioning about a rate race condition in which, if you have two cni's, like aws cni and an entry or cni uh chained together, then it may so happen that we have an existing issue in which like. uh If there are some parts which are running before entria is uh deployed, then you need to manually restart it right. Otherwise, the engineer will not manage those sports and network policies won't be applied.

E

So that is one issue and we documented saying that once you install entria, you uh delete the existing parts in all namespace, but the problem is a little tricky, maybe that, let's say, if you're doing a scale up of your worker nodes, then uh it may so happen that the worker node comes up with aws cni and as soon as it comes up, a pod gets scheduled and then uh then, when entry is applied after the pod was scheduled, then it may not be managed by entria. So this is little.

E

uh This is the risk condition which cody was talking about, and here uh we can't expect user to monitor for scale-out event and then kill pods which are running on that worker, node and so on. Right, so what we are doing as a temporary fix, I mean like the proper fix, can come from upstream obvious upstream uh kubernetes, but uh as a temporary fix for uh eks. uh What we are doing is um we are adding a node in its script.

E

E

Or I'll just put.

E

So you guys can see your screen right and the font is clear enough right.

E

Yes, okay, perfect, uh so what we're doing is we are adding a node in its script, uh which will basically uh wait for uh entria cni to be uh present in our cdni config file, and then what it does is once the entries dna is present, then we fetch the existing parts from aws k8 agent, and so using this call, we fetch the existing parts which are running on the worker node and then kill it kill the container uh running on that worker mode.

E

So what would happen is after we kill the container cubelet will restart kubrick will respawn the pod or the container, and then, as the config file had both aws cni and then entry cni, then the pods will be managed by both the cmis.

A

One question for you: rahul the ci test that we've created to test to test this.

A

Are they generally applicable that we could? We could take those upstream as part of the conformance test, so that we could determine if other c is have the same problem as part of our effort to drive this upstream for a fix is? Is there any way that we can use what we know now to determine if there's problems and say other cni's that run against those performance tests.

E

uh Sure so uh at least we know from ceiling that, uh if you don't restart the existing parts, it will not be managed by an uh ceiling that that is what we know for sure that they also recommend restarting the existing parts, but for calico I have not seen any documentation wherein they talk about this problem. They don't mention that you would have to restart any parts.

A

Sure what I'm suggesting is that would be the information that we have here, um coupled with some of the work that we've already collaborated on on. The conformance test would be a great addition right to to help support um fixing this upstream.

E

Yeah sure we can use those yes, we can. We can, uh at least for the race condition, would be little difficult for us to reproduce using performance test, but we can mention about these problems when that, when we do cni chaining, we would have to restart the existing parts.

E

E

A

Yeah we pause for questions here on on eks in policy only mode and if.

C

A

Questions there, the other thing that we're working on so I'll pause here for just one second see if there's any questions.

E

Let me continue this uh like another for another two minutes. I just want to update.

H

E

We do run by weekly, ci or aws, aks and eks, and the way you can fetch that information.

E

C

Sorry, uh it's it's twice a day, yeah not twice right.

E

So this uh we had, we have a jenkins job in which we run uh conformance test on eks, gke and aks, and if you see they are all passing and as this test we do, we operate in network policy. Only more and uh and maybe uh antonio or yang can talk about. uh There are a few tests which we skip. We run mostly the network policy test and then for basic uh community test. There were some things which we skipped.

E

I won't go into the details of it, but this is the page which you can refer to uh to check on the cloud ci and from the supported, uh and we are running a eks ci on amazon, linux, 2 os and for gk. We run it on ubuntu and for aks we run it on ubuntu 16.04.

E

E

Yeah, let's take questions now, so I.

E

Think, and also one another thing if we have this conformance test. So if you want to deploy a cluster in aks, eks or gk, you can also use this standalone script, wherein, if you pass in your credentials like azure, app id tenant and password, it can help you deploy the cluster and set up entry or two in network.

E

Policy, so argentine jobs indirectly calls this script which basically deploys the cluster applies entry and runs a sonobot test on it. But if you are just interested in deploying a cluster and trying out entry, if you pass in the required arguments, you should be able to deploy a cluster.

G

I think the only pitfall right now is that for for gke and naked uh ndks, we still um sort of you know, put uh modified an mtu and uh service cider for for because I didn't know um you know we can, we can omit those in 0.9, so this probably needs some little bit. Modification.

E

uh Yang as part of yes eks, we we expected the default mpu and service sider, but I just like I mentioned with 0.9 release. We will not need those uh configuration.

G

Yeah yeah exactly so.

C

He's fixing the script for eks, but we need to do gke as well: okay, cool the raul's pr is taking care of the eks one. Okay, okay,.

A

So salvatore, I'm gonna do a quick time check. I think we've got 18 minutes or so left in the hour. So I know we have another couple topics.

A

um Should we continue on with the uh talk to talk about the primary c I option in aws and also gke, and what we've done there or should we switch to one of the other topics.

B

um The other topics, as I remember we had the operator, which is probably a discussion- that we might also defer to the next community meeting in my opinion, and uh can you remind me what the other topic was.

A

I was going to give a quick update and I can defer mine to the very end on kind of what we're doing in the network policy v2 uh working group. So so, let's, let's continue and try to finish out. I think uh what all the good work that rahul is talking about here on on cloud and then I'll I'll save the last a couple minutes for a quick update and we can take any questions from there perfect, so so rahul.

A

um The other thing I was going to mention is you know this concept of you know. We talked about a policy only mode. We could also be a primary c I in in both eks and in your build your own cluster on ec2 and the reason you might want to do that you may want to use ip spaces that are orthogonal to you know what was provided in a vpc or in the case of aks.

A

What was provided in azure net talk to us a little bit about um how we would go about you know, setting setting that up and how it would differ with with policy only mode, and maybe what some of the the caveats we're running into there as uh genji.

E

You to take this with respect to the node, ipam and other stuff, which would have to do to support primary cnn.

F

Yeah hello, could you hear me we can hear you, okay, uh yeah, just as suafu said for untrained to be department thing. I uh we require load ipam to be enabled for combat control manager, but it's not the case in eks. That is, I'm not sure about the aks gker.

F

Well, it's not the case for eks and we have no way to enable load and pan by our own script and according to our discussion with uh amazon folks, they don't have plans to enable load plan for eks either that is starting this year, according to them so uh sam's the only choice for us to do ipam by ourselves.

F

If we want to be the primary singer in eks, probably something we can consider for 0.10 or 0.11, we love these guys, but maybe we can fix us that for only planets for the other 10 generator 11.

A

F

There's a parameter.

A

On on the control plane that we have to set right for to enable node ipam capability in kubernetes, and so that's not exposed currently in eks, so therefore we have to implement our own ipam. That's where you're going on that. Okay, perfect.

E

So yeah, so this limitation is only for eks, which is uh amazon managed.

E

So, but if you are doing like ai or bring your own cluster in or set up your own cluster in aws, then entria can be just a standalone cnl and it can operate in both end cap and.

E

A

We're having some audio problems.

E

Yeah, so we zoom.

A

uh So what rahul was I had some audio problems. Hearing your whole, I think what you were trying to say is: um if you build your own cluster right, you have control over enabling this capability in kubernetes right to turn on node ipam, and so in that case we can be a primary cni right now. I've also.

F

Mentioned for building on cluster, we can also enable hybrid mode. If you can fix some emi settings to allow multiple ips, for example, and in syria we can be. We can also run in low income, but I'm not sure of what I did or not, but in theory we should be able to do low income mode too.

A

And so a common setup for for.

A

I I was just going to mention a common setup for those who may not understand what hybrid mode is for us.

A

um If, if I'm running uh my own cluster in ec2- and I decided to put, for example, nodes in different azs, which would effectively put them in different subnets, I could run in no end cap or or run you know, without encapsulation, between the nodes in the same subnet, but as traffic crosses from one node and one subnet to another node in another subnet.

A

We would automatically encapsulate the traffic to enable that right right.

A

So therefore, we get, I think, more efficient. We don't have all we don't have the overhead of encapsulation for for every um for every traffic path, only those that are crossing subnets, yep now jinjin. If, if we turn on something like hybrid mode, it does disable some of the diagnostic features correct, because some of the diagnostic features rely on underneath.

F

Yes, encapsulation can.

A

You talk to that a little bit.

F

Sure, I think you mean truthfully um beautiful, trace flow um in the contemplation up to zero downline. We require to leave tunnels, and so, if you do low in cap or you do police only mode or if you do hybrid mode.

F

Then you will not have three sort of features for hybrid noise metal complex if the source that nails are in the same subnet, you lose stress flow feature. If they are in different stuff that you can still see, you can still do tricks.

A

Thanks for clarifying that yeah, so that's that's! You know. If, if uh one other question I had on that, jinjin do does our um command line utilities and different things that that might be using those tools? Do they inform the user that they can't work? Because of that, or is that something the user just needs to know.

F

uh For crd, I think we.

F

Let me see what uh I I think we at least we we don't feedback anything into the idea. I think uh there.

H

F

It will be ironing, actually I forgot. Probably we can double check for front. um I forgot we.

C

F

Fitted as of to say or not,.

A

Great well, that's good, that's good information to know if you're trying to run trace flow and you're wondering why it's not working, you might check to see what mode you're running in so I think we've covered all the uh the different uh cni modes that we support and ways of running in amazon. The only one we haven't gotten to is a gke. So does someone want to talk about what we're doing with no end cap and in gke rock classic here.

A

E

Can you hear me.

F

uh You're breaking, I guess hello,.

A

We can you're breaking up just a little bit yeah.

E

A

E

H

So before we move to gk, I have a question only case um you guys, I guess some of you are already aware that uh amazon is working on the uh next generation. uh Vpc cni right- and I think uh you know, apart from the the other drawbacks, that they have over the pod density and etc. The thing that is relevant to this discussion is that I think they're also trying to solve or introduce a network policy controller, along with the ability to apply security groups directly on the on the paths with some crds.

H

So what do you guys uh think is the strategy there, for you know in terms of like the collaboration and uh like how does how does it affect our road plans for road map.

A

So we're definitely looking at that um in terms of their their their cni implementation. So so right now, if we go as a primary c, I we do have an advantage in terms of you know, pod density right. We don't have to necessarily scale hardware to match the number of enis that are needed um for additional pods. If a if a user, for example, wants to run a lot of small pods, there's, there's definitely a cost advantage um of running something like antria as as a primary cni.

A

We have definitely discussed the capability of including.

A

Security group affiliation um as something that we can enforce with with network policy. It's definitely something that's on our roadmap, um and so we will be addressing that. I do think that there's still going to be some additional advantages um with antrio proxy, for example, and the way that we handle vips um et cetera, that are going to give some performance um advantages.

A

um But you know it's certainly you know something that's healthy in the in the ecosystem, right. We. We fully expect that uh um everybody's going to be pushing you know either either directly from the the cloud supply the cloud providers or from other cni's right to keep pushing this envelope forward.

A

So that's not something that we didn't.

A

A

Rahul, do you want to go ahead and did that? First of all answer your question or whether some specific insight you wanted to talk about on the roadmap.

H

um No, I mean in general, I feel like the the this vpc and I that they are trying to develop is trying to cover you know all of these cases.

H

I see a lot of people like you know, waiting for that, at least I'm following that issue, and I see a lot of people like uh following that issue and waiting for these developments to happen directly within the amazon vpcc. Instead of you know them relying on calico or andrea for policy only mode. So so I just wanted to share that with the team and see if.

A

H

A

Now that that's good insight- and the other question is also going to be- is like how much uh extension from upstream kubernetes network policy they're going to support right, there's also a lot of work that uh this team is doing, uh and this community is doing right now on um extending you know the capabilities with like cluster scope, network policies and other things to that nature. That still may be, you know very different than than what's offered you know, natively by by amazon, for example, in in their policy provider engine that they're working on.

H

Yeah that was going to be my next question. So how much effort do you guys think that is? It will be to uh enable the cluster network policy crds and like the native ntr network policy, crd that we are trying to uh implement, or we have already done that as an alpha feature in 0.9.

H

So how much effort do you guys think that it will take for us to uh enable them across all the clouds, not just amazon,.

F

Sorry, I didn't.

H

F

It usually, we enable our policy crd right, at least for aksgas gk. Here we can easily change the yammer file to enable the feature gate for unshared policy, and you can have a generality policy if your question is for eks engine, uh because in your case engine it's a supposed thing, I I think for that one.

F

If we want to enable uh trucks policy, we need to add option to to aks engineer.

H

Yeah, probably for the aks engine, I think the changes need to be done in their quality right. So.

F

We we can enable we can also enable and share policy by default right and uh yeah. We could.

H

I don't know what should be the graduation criteria there, but it is. I.

C

Think certainly.

H

Something that we should think about.

F

I I I guess something we we can fully control on our side. If we want to enable that, by defaulting against engine, we canceled it.

A

So time check, we've got less than five minutes. So do we want to go ahead and see if we can cover gke as well?

A

Maybe a quick overview of what we're doing there.

E

Okay, uh just just to let you know that this is where you set the uh the traffic and cap mode, and these are the supported modes like end cap, no amp.

E

And this is present in the yaml, so this is how you can in which mode uh the fjse- and I would get going back from gke.

E

So gk is for the google cloud and the support is pretty limited here in a way that we support it only on ubuntu nodes. uh They have container optimized images, also like cos where we don't support it. We support it on ubuntu and they have concept of vpc native enabled and disabled.

E

What I mean by vpc method is: if you enable this mode, then you get ip addresses for the pods from the vpc uh slider itself.

E

Okay, and if you disable this more so we can operate in both the mode. So this is one specific uh concept of vpc native, enable and disable. uh You can read about it on google, but we can open.

C

E

You know that we can operate in both templates. uh This is the prerequisite that this is how you set up the gke node, and then you basically create a gk cluster using this command.

E

I think uh when we enable entry of proxy this documentation needs to be uh updated to not have this uh parameter mandatory engine I'll talk to you offline. So, basically, how you do cluster on gke then similar to the way we have node in it for aks eks. We also have a node in its script.

E

The way uh the clusters are deployed in gke is the cubelet uh parameter for the cni actually points to cubenet, okay, and even if you have some config file in the cni directory, it won't be honored because the cni is pointing to cubenet.

E

So what we do with this node in it is we modify the cubelet parameter from cubenet to cni so that, if we add our cni entries cni in the config file, uh the cubelet can honor it. So that's what we do that this node in it does it basically updates the qr parameter from cubenet to cni, so that entryway can be inserted.

E

uh Support for this was added in 0.5.0 and we have a ci uh running, I think from liver.6 yank and confirm, but the ci is already running on gk.

E

Then the next step is you just download the yaml and apply it.

E

And then insider thingy we can once we enable and proxy for gk. We can move this too and then. Lastly, the common problem, which we discussed for all engineering mode, is once you have the entry up and running. You need to kill the existing parts or restart the existing parts so that they can be managed by entry.

E

uh One uh note is, we are actually not doing any cni chaining for gk, okay, uh like for aks, we change it with azure for switching it with aws for gk is there is no chaining? What we do is we just intrigue, becomes the primary cni and then, uh and we operate in no end cap mode, which means there is no encapsulation required.

A

C

A

uh In this mode, the the ip addresses like who, how is ipam working with with uh gke? Can you talk a little bit about that? I know we're at the very top of the hour, so we may have to make this pretty quick.

E

Right so we use the so the way they.

E

Let's say if you are operating in a b c, disable mode, so they allocate extra cider for that vpc. What I mean by that is, uh it is all gke magic. Maybe that uh let's say you, you deploy a vpc in 10.0.0.

E

right and once you have a dpc native, more disabled, they allocate a secondary sider, called twenty zero zero zero and carve out and carve out ciders uh subnets for each fault each node. The way we do it in entry, so as part of the node ipam controller, the gke, whatever submits they have allocated for a particular worker node. That information is passed down to the worker not through through that node controller itself.

E

Does it make sense, got it.

H

A

Yeah, I just wanted to highlight that to the audience yeah.

E

Yeah, I think the google tweaked it in such a way that, if they, if they are allocating some xyz subnet for a particular worker, know that information is propagated down to the kubernetes construct itself. You don't have to call any cloud apis to figure out what is the subnet a worker node is working on. They can fetch this information from kubernetes like from the north spec right.

A

Excellent well, since we're at the top of the hour uh salvatore, I will hold and and and bring a update from the working group uh for our next session um and let you know what we're working on. But if you're interested in participating, please check out sig network github, there's actually a link to what we're doing in the network policy v2 working group. If you'd like to participate between now and then.

H

A

H

So curry, actually I wanted, you know more than an update, uh because uh I also wanted to probably we can share the the user stories and the excel sheet with the group uh before the next.

C

C

H

From the community from our community as well, so that we have uh a fair idea of what to emphasize in the next meetings.

B

Yeah make sense, um I mean I guess you can share them on the on the vlog channel, so we can have a look, and uh I guess there will be also more conformance tests coming as a part of this activity. Is that correct.

A

At some point right now we're just evaluating user stories to.

B

A

To determine uh what makes sense for a v2 iteration.

B

I see perfect, and uh it seems then that uh we are done for today's meeting and we also have an agenda for the next meeting, which should be a conversation on the uh network policy v2 and uh and a discussion about a potential entry operator which, in light of what we discussed today for public cloud, uh becomes a little bit interesting because you know there is going to be some overlap and uh anyway, that's something that we can discuss in two weeks time anyway. uh I think uh that is all for today.

B

Is there any last topic that uh the community would like to bring up going five, four, three: two one: zero: all right, uh I'm going to stop the recording right now, and I would like to thank everyone for attending this instance of the andrea community meeting have a nice one, everyone and goodbye.