Cloud Native Computing Foundation FluentCon: Cloud Native Logging Day with Fluent Bit and Fluentd EU 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lightning Talk: Scaling the Fluent Bit Kubernetes Filter in... Drew Zhang & Nithish Kumar Murcherla

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Lightning Talk: Scaling the Fluent Bit Kubernetes Filter in very large clusters - Drew Zhang & Nithish Kumar Murcherla, Amazon

Fluent Bit and Fluentd users have experienced scaling issues with the Kubernetes filter in very large clusters. The Kube API Server could not handle the quantity of requests and became unresponsive. To address this issue, a new proposal for relying on the kubelet to obtain pod metadata was implemented. By this approach, Kubernetes filter could get metadata from the node and could mitigate the work load for kube API server. This session discusses the scaling issue in detail, the solution that was implemented, it’s trade-offs, and how it was tested.

A

B

A

This is joe from amazon, aws ecst, I'm now focusing on container metrics and the logs related work both inside the device and on the open source, transit.

B

Hi, I'm nitish kumar murcharla a senior cloud support engineer with aws as well. My major area of focus is containers and container related technologies and services on aws.

A

So today we're gonna talk about uh the new feature we just finished uh inside of the beach, which is a skill in the front base, kubernetes filter in very large classroom. So for this topic.

A

This feature is trying to address the suitability issue which of course, have seen in the kubernetes filter.

A

Some customers use fluency to some customers using funding and they are seeing the api server for struggling and unresponsive when they try to scale up their kubernetes filter acquires clusters- and this is due to that friendly- is spamming. The api server with list of post cable cost.

A

These are expensive cores and can bring down the kubernetes api server and control plane, and this makes api server became the bottleneck of the scalabilities.

A

So, let's see the architecture.

A

The request, which run b and from b try to send out, is to get my data from the api server, and actually this information could not only get from the api server, but also from the kubernetes.

A

And so the kubernetes is the primary uh node agent that runs on each node and the run bits could instead call in the api server but just called the kubrick to get the same information. So we proposed an idea that we can provide customer an option to get the meth data information from corporate instead of to the case server, so that we can unblock from the account.

B

A

From the bottleneck and in this solution we will reduce the required to api server to zero and just increase the one required per node for each correlate.

A

And, let's let me just talk about all the results and our features.

B

Thanks drew uh so for this, uh I have baseline uh flow and d. First uh with fluency, uh we saw over 67 000, uh so so I created a cluster of 2k nodes and and over uh 30 000 pods uh and every node has fluent fluency agent running on it. So for 30 000 parts we saw a hr and I was turning at a rate of uh 1000 pods per uh per hour.

B

uh At that rate, what we saw is uh uh the fluency agent was making uh the locks that that you're, seeing on the screen uh shows that the fluency agent is making uh 60 60 to 70 000 api calls uh with that churn rate uh that is collected over three hours of time. uh Apart from that, we also see a lot of watch and get api calls that are made to api server by fluency.

B

You can also see that the p99, when a list board less spots are less spot, is called by by the fluency agent or other agents. You can see that uh the p99 is high high at that point of time. These are some inter internal metal metrics that have collected from eks ek. Servers uh just wanted to showcase uh for for your understanding how how we are unders, how we know that the list api call is taking a lot of time. uh A lot of time here.

B

uh Yeah, these are the continuation of the metrics as well. uh You can see the list. Latency is spiky there in the third third third row.

B

uh Yeah uh for customers, uh as we are internal uh to aws, we were able to see those metrics uh uh from the internal dashboards that we have. However, customers might not have abilities to view the that granular metrics, so I also set up prometheus on my eks cluster to showcase the the uh amount of aba calls being made and the ehcd request. Latency and api server request. Latency metrics there.

B

I made over uh 800 lists and get it made whenever I turned my cluster with a lot of parts uh in my deployments. uh I I see that there are a huge number of api calls being made to api server due to which my cube ctl commands were slowing down.

B

That is what I'm trying to show in this particular screen with this particular screenshot.

B

Apart from that, what I also noticed with fluency was, uh I was running a thousand clients of uh log test uh uh agent that is available on uh that is available uh and uh each each uh sends about 600 lines of logs uh per minute, uh and I had over a thousand clients running on my on my uh cluster for scale testing uh and, as you can see, uh wherever the pods have landed, uh there were multiple boards landed on one single instance, and I I see that uh the fluency is taking approximately over uh is taking over 1500 megabytes of memory, uh which is a lot uh and also on the bottom of the screen.

B

You can see that the cube ctl call that I made to api server took over 15 seconds, uh and sometimes uh this was not the maximum. This was stop. This data was recorded when, when I just made an api call or when I just initiated a uh update to my cluster uh it sometime when I, whenever I listed all the parts in the classroom, it sometimes took over 60 seconds as well, and timed.

B

Out yeah, uh so later I switched to fluent bit with the cubelet uh feature that we have developed. uh As you can see on the screen. uh I am querying for a data of over week. I've been running that feature for over a week in my uh in my performance cluster and ran uh quite a bit of tests on that, uh and I haven't seen a single api call being made by fluent bit with q, use, cubelet or true flag to api server.

B

uh This significantly reduce the amount of workload or or the reduce the amount of memory pressure on on api server and as as fluent bit, is no more making. That list calls to fetch data or like.

A

Fluency, that's all for our sharing thanks. Everyone for attending.

B