Antrea Community, 14 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Antrea Community Meeting 09/14/2020

Description

Antrea Community Meeting, September 14th 2020

A

Let's start recording okay, so welcome everyone to the september 14th or 15th instance of the andrea community meeting. For today we have an agenda which probably is not going to take the full hour, but uh since.

B

A

Have to discuss a little bit of naming yeah it likely will. So, let's see the topics that we have on the agenda today, uh you can find the agenda on the community slack channel.

A

So first we have a discussion of a solution for audit logging, foreign network policy, and that is a us from machi and uh I'm sorry if I killed your guy sorry and I'm sorry if I killed your name and then for today we have a discussion of a short name for andrea network policy.

A

This should be anything is an easy naming task, so should not be too much and uh abhishek will lead that one and finally, uh with the chen which which one today we are going to discuss the uh a proposal for changing api groups all right. So um these are all the topics that I know of. If there is any other topic that you would like to bring up, we can easily discuss it after uh in the open discussion section of the meeting.

A

So is there any objection to this agenda.

A

Sounds good, so uh if you want you can get.

C

Started I'll show my screen.

C

Thanks for today, I'll do a presentation of audit logging for custom network policy and a demo um it's my intern project, um so the overall goal is to implement firewall logging used for auditing and analyzing traffic when it is enabled individually by each cluster network policy.

C

um From from the flow on the right, we could see that it goes from the yam spec to entry, our controller into agent obs and then uh send packeting back to agent, uh and then we store the log.

C

So moving on to the details, um first, uh we changed the crd and spec. We added a new field, enable logging in the ingress and egress for a cluster network policy, and um another notice is that we also start packing if entry policy feature gate is enabled.

C

um At that time, we subscribe to the channels which will be used later then moving on to andrea controller, convert this back to computed rule and uh which enable includes enable logging. So from this spec it would be true or false for personal network policy, and it will be default to false for network for kubernetes network policy because it does not support this field.

C

uh Then, moving on to entry agent, um we register pack in handler with openflow under reason. 0 and 1. also check roll, enable locking to create different flows.

C

I would talk about this later uh and we initialize a custom logger uh that will be used to lock uh the information then moving on to obs. um We load this position in register seven.

C

So this is uh that we created a new register to particularly load the allow or drop information to extract it later for logging, and then we send packeting to controller with a corresponding reason. So currently uh we use zero for network policy and one was previously used for trace4.

C

I will also talk about that in detail later um also in the entry agent here. The network policy controller based on the received packet reason we forward the packet to subscribe channel, um which was uh subscribed uh during the start. Packaging uh then use the registered handlers to handle packets.

C

um The these were registered in agents from here then. Finally, in entry agent, we handle the uh the packets uh reports, the information to of the packet load and also from the registers retrieve the information that we previously loaded then use initialize the logger to log.

C

An example of this is like um it would be date, time, file, table name, uh cmp name, disposition uh of priority source, ip destination, p and protocol.

C

I will show this later during the demo, so the lock tool used in this project is the golden lock and lumberjack. So lumberjack is maintained uh properly and it has a recent release version 2 in 2017..

C

um I have set the file rotation to be 500 mega size. So after that the original file name would be renamed with the current timestamp and create a new file with the previous thing to log, and there would be of maximum three backups and the previous logs will be compressed and after 28 days they would be deleted.

C

um There are other choices like that, but um once when searching online, I found that that doesn't have its own rotate rotation, so it also uses lumberjack. uh Regarding this, um I I just choose to use the traditional lock, because that there's no much time difference in terms of log.

C

um Then it's the packaging reason. It's currently used to separate feature queues which still correspond to obs reasons. So after some testing, when we set the reason to be zero, it would correspond to the obs reason has no match. um If we set it to one, it would correspond to the overs reason action and for the other um correspondence they do not return a particular packing packet.

C

So a better design is to extend the user data field in packaging for separating different subscribed features or channels. This is a feature task related with openflow as well.

C

Currently, the solution is to use zero, uh so we register we register zero for network policy logging and one was previously used by trace wall and finally, race limiting.

C

um Currently, it is set on the agent size, with rates limiting q on the pro is that it does not change the obs, but it still occupy the traffic from obvious to agent, whereas cropped packets, because um for allowed packets, they would of a flow, would be established and it would there will not be continuous um uh packeting sent back from the obs, but for if we have an attack with dropped packets, then uh the obs will continuously send packets back to agent.

C

So we want to do this rate. Limiting um a better design is to introduce rate, limiting on obs side um to rate limit the packaging side sent to controller action for network policy specifically, and we want to prioritize trace law over network policy.

C

um I was instructed with the solution to use meter and we are investigating on this to introduce it to uh with as a future design. um Then I'll do a.

C

C

um First I'll introduce the topologies on my my uh my other.

C

There are the current parts. Now um we could see. There is a client and server that I previously created um later in the demo. I would log into client and then a ping server to test, and these all run on the 23 pods and I would dump the flows on this pod to see the change.

C

So at first um I'll do and.

C

C

Somehow, blocked by the zoom.

C

um Now I've applied the allow custom network policy and we can see in um when we dump flow for table 89 uh here um instead of directly send resubmitting to table 105 it's there, it sends the controller first, um the reason here it says no match, but it's actually using the reason zero.

C

Now we could see. The uh here is a log. They originally look like.

C

C

Now uh I'm on client and painting the server.

C

Here it there is an allow locked here.

C

We can see that uh here is the table name for 89, and this is the customer policy that I previously defined. um The action is allow a priority. The uh priority is the same here as the flow and source is the client destination server protocol, because I used ping. It's.

C

C

And I'll set the log to false and see if there's any other.

C

C

Notice here the flow has been changed to resubmit directly after actions.

C

C

uh Nothing is locked here, so it's correct.

C

Next I'll do a.

C

C

um For drop the flow so send the controller directly instead of a drop action.

C

Now, if I do curl, I will not receive the content.

D

C

Are several locks dropped drops logged which correspond to what we previously discussed because it tried to log several times um so for the rate limiting we are still um investigating on the obs side, but uh currently the rate limiting queue is working on is working.

C

C

Command from the website.

C

Basically, this file- it's after I curl this address it will redirect to get the uh the second address that I put in so I'll. Try this.

C

C

Basically, it's sending 50 requests every.

C

C

um Now it's a huge file, but um I could do a graph to see the counts.

C

C

Sorry I appreciate tested with 101.

C

C

Because the default controller is limiter that I use um allows 10 uh 10 requests per second and 100 buckets size, so.

C

C

Now it should be.

C

There should be less than expected number of.

C

C

Basically, now it's um like 88 and 89, but from the commands it's sending 101 calls per second and um yeah, but for the rate limiting, uh of course, the better way is to implement it on the obs side, uh which I will be looking at it um yeah. I think that's all for my presentation. Thank you.

A

Thank you very much. That was a really great and I I really enjoyed it and really nice and slick. um I don't have a lot of questions because it was extremely clear in terms of uh or of functional, behavior of the rate limiter. I just wondered that that one is, uh is it causing the recast uh for logging from you know the controller request from open with which to get cured, or will it cause the request for being dropped?

A

That's just a qualification that I wanted.

C

It's on the agent side and it's for the package received from obvious typing.

A

That is right and when a packet is rate limited um sorry, the behavior is like an http rate limiter. Well it it will drop the request from obs the you know, the controller request from obs, or will the request be cued and just so just be put in a queue.

C

From what I read, it will be dropped.

A

Okay: okay, thanks.

E

I had a question from a user perspective, so.

F

E

This was fantastic again. I agree with uh salvatore that the logs were really clear from a user perspective, two things that I think would help.

G

E

Is it possible to convert known ip addresses within the cluster to pod names or some type of a qualified pod name in the logs, um because you know at the time the logs were taken? If there's not a period of inventory of the time, then it would be really relate those logs and number two.

E

Once we have tiering, you know fully enabled also recording the tier name might be useful for downstream uh filtering for parties that are only interested in looking at logs from our particular.

C

Tier currently, uh currently I haven't thought about uh the tearing side of this um that I could look into it like.

E

That and that's very alpha yeah, that's a very alpha feature right now, but just keep that in mind as you're going forward. What about the ip addresses? How feasible is that.

C

um I'm not sure if there's a place for extracting that, uh I probably need to look into that from currently what I have seen. I I'm not sure there is a place to extract that.

C

Impossible information.

H

It's possible to convert one at least one ip to the name, but I'm not sure. What's the cost cost of doing that, uh if the overhead is high, then maybe it's not lasting.

E

Sure there's yeah.

H

E

Expected there may be some overhead um yeah if y'all, if y'all could, just uh as you're developing this feature out, spend some time. Looking into that, I think that would be a a great usability enhancement for uh users to have both ip address and a pod name and if that's not feasible, let's look and see what that impact is.

C

Sure yeah, thanks for the suggestion.

H

I also have two questions so first, I'm thinking uh it doesn't make sense to keep the pack hider. Sorry, the the local message highly like a packed in don't go there. I think, probably it's not very interesting to to the consumer of the logs.

H

If we remove that, we can receive some bets.

C

H

Another one I'm not I'm not clear about that, um so that does typically include the some pilot feels like the the the size of the packet interlocks.

H

um I think that information is very easy to get right.

C

C

If that is you.

H

Know, of course, yeah, but I'm not really sure it's very. It's really makes sense, but I'm just thinking at least if that helps, then it seems very easy to also add that information yeah.

C

Perhaps more useful than packaging.go.

H

B

E

It also is it also possible to capture uh the port information. I didn't see that.

C

Or information- um I I looked into that, but it's not directly in from the packet, although it's ipv4, but the structure of ipv4 does not have a port in there.

C

The uh the ipv4 structure that we currently use.

I

uh It's possible that you have to do some additional parsing compared to what we do for trace flow today, to get to get that information.

E

Yeah, it might be useful for attack analysis to have that information as well.

I

And uh a minor point, but I guess related to what uh janjun said earlier. Maybe the timestamp uh should be changed for uh and in input in a different format with milliseconds milliseconds, especially when you have a lot of packets like this.

C

I

C

E

Do we also indicate on the time stamp is.

F

C

E

Or utc, or something like that, do we make that apparent.

I

Yeah, it's probably easy.

A

I

As a locator, so okay.

A

That is correct. These are probably little user experience changes that can be done over time, but they don't they don't affect the you know. The validity of this demo, which is, is a really nice addition to the to the platform.

A

H

A

And the final question that I had is: if we wanted to measure the logging overhead on, let's say, while testing its scale in your opinion, how we should proceed to measure what is the upgrade of logging or by any chance did you already make some measurements of that.

C

Sorry I was not familiar with the metrics. What kind of.

A

Actually, I'm just thinking in terms of let's say packet processing overhead, which can be measured. Let's say in term of true, but let's say: is there? Is this logging affecting throughput in any way or is it pretty much transparent to me? It seems that the at least from the obvious data path perspective.

A

The impact on throughput should really be negligible, but I don't know if you're ready. Yet uh if you actually measured it already.

C

Oh yeah um there's a resource on measuring the cpu and memory cost of this um that I'm looking into um I'll include that report in the in the pr. For this.

A

Okay, all right and since I'm I'm a very patent person, sorry from a reliability perspective. Clearly, if the agent is down, nothing will happen right here. I'm looking at the flows that you have installed. If the agent is down, the packet processing will proceed as normal. Is that correct.

C

I'm not that familiar with this, the fluid.

B

For example, in this case, the cnp.

H

But actually I was thinking uh in another way. I know people are talking about more contacts, interlocks and some more user-friendly format. Actually I was thinking if we believe performance is more important, uh should we use some way to compress the the the data a little. uh For example, we just use fixed format and we don't use this keyword like protocol or test source priority and then even for stable. We can use a short name or just integer, then we can probably reduce the size or not.

B

All right, so maybe you.

F

B

Well-Documented format and then people can refer to okay.

H

Right uh for, of course, I I don't know how people typically do this traffic logs, so I'm just yeah, especially.

E

If you're shipping them yeah, if you're shipping them across the network, this is a compressible but uh and easy to read. But we could probably compress it a little bit in terms of the format.

B

Sure I think another question that uh had come up in the past was uh whether we should include these logs into the support bundle by default, or it would be like a explicit ask for it, because I think at the moment we are planning not to include this in the support bundle.

B

If that is a fair thing,.

E

Anyone are, they are they are they? Are they needed to diagnose anything in the support, bundle.

H

E

It the primary goal.

H

For support for different purpose from support bundle in my understanding.

G

Is there any plant for this? You know logging to syslog server.

H

E

I don't know that that needs to be a concern of andrea right on on how the logs are collected and shipped. What do we think? Should it be a platform concern.

H

But you, if you want to support this logo, you at least need us need to purpose some solution for that right. If we just.

G

Go to the file.

H

G

Go ahead, okay, yeah! So that's why you know like uh do you plan to, like you know, having like a function to forward inside out there to for the login for to the stock server? Or you know this is my ask you.

B

Know I think this is something that we can maybe add into our you know items to work on like for future items. It's like you can get back to this.

E

Okay, I think that would be lower lower priority. I mean, I think I think if we log the file, we can leave that as a user concern right now, for you know where they want to ship that, um because the entry could ship it. You know we can build a lot of different options there, but I don't think that's the primary thing we should be focusing on.

E

Unless somebody has a different opinion, I mean I just um it should be a fairly straightforward task for a platform at metatrader to pull off. I would think.

G

uh This is like our you know, from networking operator perspective. You know uh using like insects yeah, you know most of customers brought packets to this look and to look at you know what connections are flowing in the environment and what connections are dropped. So that's. Why, like you know, I wonder if we can get this function or not.

F

E

We'll definitely note that thank.

E

A

Any more question on this.

A

A

Are we good to move to the next one? I guess so so, thanks again uh guf for your uh for your uh nice presentation here and abhishek. So what is this story about the short name for network policies.

B

A

Okay sounds good, so uh sean. uh Would you like to discuss uh renaming for api groups, changes to api groups.

F

Sure, okay, sure let me share my screen.

F

So let me introduce the background first, so can you submit screen.

H

Yes, that's it.

F

Thank you. So if.

H

G

F

Preos community meetings, we discussed the internal api name and finally, we choose control plane as a api group name. So in in entry 0.9.3, we made the change to rename the networking to control plane and, as a result, the net policy info enforcement will not be will be disrupted until the upgrade is done and recently junction raises the issue that it may be serious, because if the cluster is very large and the agents uh upgrade in a loading update fashion, which could take many minutes to to uh to upgrade the whole cluster.

F

So in some nodes the poles will not be enforced. The policy narrow policy of some ports will not be enforced correctly. So we came up with a proposal that, as the deleted uh networking api group back and so both old and new api. At the same time and in implementation we we are not. uh We will not add all the deleted code back and we are only uh add the api group back, but uh we will show the request with the same storage as the control plate storage as a control plate api.

F

So it will not cause much code redundancy and um with this code I have verified that we can.

F

I also added uh another test uh for upgrades so that the newer entry controller can work with older entry agents and pass the conformance and network policy test. The components and the narrow portion test is well-fed manually because we don't have a good mechanism for that.

F

But I added a simple upgrade test that only upgraded entry controller only and the basic function still works, and so this is the proposed change and if you have a objection or any better idea, you could share your ideas and if we agree with it uh by the way we um currently I'm thinking, we keep two minor two older minor release uh for compare uh for compatibility, because we already have the upgrade test from two to older minorities, and it also.

F

It is also consistent with kubernetes compatibility.

F

And if, if we agree with this, we could you could review this pr and we could document the api compatibility and the uh the break uh duplicate duplication policy in the docks of the ripple. So that's it.

H

uh Child crushing here so uh you are seeing the photological process rolling up. It can take like seven minutes uh that.

F

H

H

Update what the image and restart or you're talking about the case like in some discord, they need to recreate the rams.

F

Even it's just a normal cluster that update your operator. Image is because we set the updated strategy as loading update, so it will execute in the loading update fashion. So uh you know first update one and then wait for it. uh It's finished then update another one and then.

H

F

That case, do we have a way to guarantee controller's upgrade first, because we, if user, is using the yam to upgrade so the controller template and the agent template the update at the same time and the controller.

G

F

One instance, so it must be upgraded before other agents. It might be the same time as the first agent.

H

Okay, this is faster than me, yeah, okay and then for the upgrade highest reason. We are able to uh easily enable tasks for older version of the entry, because I'm seeing that because we see requirements from upgrade from 0.7.2.

H

To 0.09, um maybe it doesn't have an issue whether we we leveraged it. So I'm just wondering: could we just reuse your task to support that.

F

Which, which version you want to test.

H

From 0.7.2 to 0.009.

F

Oh okay, so in this test we I this is upgrade from -2, so so it should be upgraded.

H

F

But we only tested the first patch release of the of each minor release.

I

Now the latest one yeah, the latest one, the last one, so it should be 0 7.2, yeah.

F

That sounds great, but this test is very simple: it's very basic: it just checks some networking connectivity and one-nil policy scenario. If we want to test the whole performance and narrow policy, we should have check-ins jobs for this.

H

Okay, I understood.

J

Yeah, I agree so if we so so, this code is oh under github workflow, I got it so we can discuss how we can create the jenkins job um for this before we release a new entry.

J

Please uh sorry, I have an other question so, for example, if we deprecate, um for example, deprecate some resources or api group and then in in some new release, we finally remove those resources, but uh during the upgrade there must be someone to delete the old resources for us right. So if the there are some leftover api service resources, uh it should not interfere with us unless, uh because there is actually no agencies using are using that api. Oh the api group.

F

Yes, yes, for the for who deleted the resource, it is actually the entry control itself that, if we, if we decide to remove it, we could add the service api service here, so that the you mean the you mean, the the uh you mean the api service or other things.

J

I mean api services, custom resource so and the config uh yeah, the e-pen service custom resource.

F

Yeah, if it's just api source, it will be removed by enter controller in future, and we.

J

Will go controller can issue a delete request to convince api to remove the old ones.

F

Yes, yes, we guaranteed to delete the permission of this api services.

J

F

F

Okay, yeah any other questions.

A

And I think that uh that's all for this topic and uh well uh thanks for sharing this, and uh I guess that uh if you have any other question comment or if you just want to look more in detail at what john is doing just go to gita pr and we can move continue the discussion. There say that so our last topic for today agenda is uh the renaming of uh the short name. Sorry for network policies. Abishek, do you already have an idea in mind.

B

Yeah, so uh I think this came up uh because you know someone on the slack channel kind of was trying out the network policies and used the short name for andrea network policies, which is the name space network policies and he ended up using kubect network, and you know by default it returns the upstream kubernetes network policies.

B

um So we kind of like you know. Let them know that we both both share the same short name. uh You need to append the group name uh after the the short name so that uh you will get the correct uh set of network policy, but you know I'm talking to and offline and I think uh kind of feel like you know, if it's a short name, then it should be, like short, and I don't want to append anymore uh details behind that.

B

uh So that was one of the reasons why maybe we thought that maybe we reconsider the name there and the other thing was that you know it's kind of like the cluster network policies have a short name of cnp, uh perhaps uh to be a little more. You know on the uniform side, maybe we name the name: space network policies as either an ent or.

B

A

So, do you understand correctly that one of the proposals was amp yeah? I mean at least from my perspective, I'll be perfect. I don't know if there is a any other idea.

H

B

H

The problem- that's not just for untrained, positive, just control proceed. We can use a and p um okay for class. I was using cmp okay.

B

So yeah, I just ran all the commands on my.

B

B

This is something that we can also take offline, but I thought this was a light agenda on the meeting, so maybe you can kill something: okay,.

H

I personally feel saying pm here sound good, because I don't know how to keep. I think.

K

I think we were originally going with cnp and amp, and then you know when I opened up the pr um people started to see that the amp is not sort of like correspond to um the entry namespace network policy, full name which is still named network policy. So people feel like uh there's a discrepancy between the shot name and the long name, and uh there was a discussion to you know: rename it to net poor, red and amp.

K

But you know, as you can see here, if we do not put the security, it is, you know too much redundant, sure name and probably even it's even you know better to just type entry and network policy instead, which does not make a lot of sense.

F

Okay, I think it was me who raised the question.

B

I think janet talked about uh you know the cube. Sorry, the ant cattle also has the same network right.

F

Right, yes, so are we going to change the the encounter, the short name of anakato that we get the internal network policy? I think it is uh and how to get net poor right right now.

B

Yeah, I don't think we need to change that. I mean that's my opinion.

F

Perhaps it's different because it's uh it is for all club. It is for all neuro forces, including kubernetes and cluster and antenna neural policies. So and I don't have any better idea than amps. So it sounds good to me too.

I

Yeah, I think a np is fine too. I don't know. Maybe people want to support acnp as a as a new short short name as well. I think you can have more than one for cluster network policies, so it's symmetric.

F

Yeah, oh sorry, please go ahead.

B

Yeah, I think you can have multiple short name. I believe it's a list that it takes so.

F

By the way, when supporting the nail policy metrics, uh I have to define matrix names for clustering policy and antenna policy too, and because they are in the same group, the the same matrix group and including kubernetes narrow policy metrics. So currently, I named them uh narrow policy metrics for kubernetes neuropolicy and cluster policy, matrix for clustering policy and entry network policy, metrics for anti-name space, neural policies.

E

F

E

We should keep in mind is, as we develop uh cluster network policy in upstream v2 api. You know we may have some name clashes, so we may want to you know specifically call out antria in some of these names at some point.

I

Yeah I mean the ancient cluster network policy starts to be a bit long, but yeah that does address that concern. I guess, and it is symmetric with anp.

E

The other way we could also look at this is maybe we only need the entries portion on the short names, because the longer names are they're already fully qualified right by the api group.

B

Yeah I mean as long as if there is no clash, then you don't need the fully qualified name, but if there is a clash, then you can uh use the.

I

Right I mean yeah like if there is ever like an upstream cluster network policies. uh The problem is, if we do cube, call get cluster network policies and it's going to be returning like the kubernetes ones, unless we rename it to entry or cluster network policies, and I mean, and then there is no.

I

E

Yeah make sense that that's that's a ways off, but that's something that we should probably be. You know thinking of a possible.

B

Transition make sense, yeah.

I

I don't mind having a long name like entry or cluster policies. If we have a short name that makes sense like acnp.

B

Yeah, I think, as long as the kind remains the same, I think these are just short names.

H

Are easy to remember so there was real difference to users.

I

Do you think you can uh summarize a proposal in on github, or maybe a bishop.

B

Yeah I'll open up an issue, and then I have these different proposals from today.

A

Thanks all right, so uh I think this also uh summarized the conversation on uh short names for andrea natural policies. Is there any other topic that you would like to bring up for today? Anything else that you would like to discuss any.

A

A

And this is the sound of silence, which means that perhaps we don't have any other topics for today, so we can close our meeting here. I would like to thank everyone uh for attending this call and wish you a good morning afternoon, a good evening or especially for the folks in the west coast, a good night.

H

A

Thank you thank.

H

You bye-bye, thank you bye-bye. Thank you.