Calyptia Fluentd, 30 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Scaling Fluentd From 0 to Millions of Events- TK Kubota, ITOCHU Techno-Solutions America, Inc

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Scaling Fluentd From 0 to Millions of Events- TK Kubota, ITOCHU Techno-Solutions America, Inc

We have seen the impact of data formats, messaging sizes, event rates, and even data sources causing challenges. For the successful operation in a large scale environment, understanding the performance characteristics of Fluentd and designing a proper solution architecture are important. TK has had the opportunities to work with many large institutions with thousands of servers deployed with Fluentd / Fluent Bit and will show how internals of Fluentd / Fluent Bit impact performance as well as benchmark results. We will share tips and tricks on how to optimize and tune parameters for your own deployments based on characteristics of input, filter, buffer and output pipelines. Lastly, we will discuss how to go beyond tuning and how to increase performance with additional compute / multi workers / and deployment architectures.

A

Can we get started?

A

Okay, nice meeting you I'm tk solution, engineer working at institute, techno solutions, america, I've been working for enterprise, support for foreign d and frambeat and help our clients to design a currency and friend beat solutions.

A

Today I will talk about scaling fluency from zero to millions of event, from not only a performance perspective but also operation perspective.

A

Okay, let me start from a considerations when you bring friendly and friendly in production.

A

After you finish a solution, validation in london poc, you will bring a friendly and different bit in production and you will see a different behavior in poc and in production in terms of data source.

A

Even if you start small, for instance, 10 to 100 nodes, the number of end points could increase easily and traffic late will increase accordingly. One of the typical issue we see in our enterprise support is data format. Issue endpoints sometimes send a unknown data format which is not validated through a poc, and also some of the endpoint might send a sensitive information like pii and internal system information like ip address and hostname, and then the point at destination might be added in the future based on the use case.

A

So the beauty of randy and friend bit is bender neutral too, and you can see messages to a variety of endpoint like sprung, kafka, hardwood and elasticsearch.

A

But from operation perspective you need to update the configuration continuously to adopt changes and enhancement. That could happen in the future.

A

From operation perspective, there are also challenges for it: operation team. In terms of data sources, data sources are typically managed by application team and it operation team does not have full access to the environment and also destinations are managed by different team.

A

For instance, ai and ml team manage hardware environment for application running on spark and security operation team might manage a kafka sprung elasticsearch for sim solution so uh that it is challenging for it operation team to manage end ruling a pipeline from data source to destination, since it is difficult to log in to environment and change, setting or configuration directory.

A

In my experience in enterprise support, we see three challenges for it: operation. Team first point is managing endpoint in the large-scale environment. You need to manage hundreds or thousands of data sources, and, if you adding new uh featuring rules, it is required to apply changes across thousands of endpoints.

A

Second point is performance. When new data sources or a destination will be added, you need to scale a performance of randy and rambit to prevent loss of data.

A

Third point uh is seeking ensure shipping proper information to the destinations. As I mentioned earlier, a rubbish event from data source might contain unnecessary information.

A

Endpoint might send a chatty debug information or they might send messages which include sensitive information.

A

Okay, let me talk about the implementation model. Before we move on to performance, okay, we have to implement transition model a further only model and folder, with aggregator model, with water only model you need to deploy current d or from bit instance on data sources and further send data directly to destinations like spring candlestick or kafka.

A

On the other hand, of water with aggregator model, we bring aggregator between data sources and destinations, and we deploy light weight, for instance, or endpoint, and endpoint will send messages to aggregator and aggregator will do a data processing task like parse original messages and filter unless future unnecessary messages out and then aggregator will ship messages to final destination.

A

So uh in the large-scale environment we typically recommend our. We typically recommend users to use photo with aggregator model, since there are three benefits to bring labor data between data source and destination.

A

First point is: it is easy to change configuration rules so with foreign remodel, and if you have thousands of endpoints- and you will add another endpoint- you need to change configuration of those thousands of nodes, but with aggregator model you need to change only a configuration from aggregator and you do not need touch configuration on uh in the point side.

A

Second point is: it is easy to scale performance independently on aggregator, as I mentioned earlier, with photo and aggregator model water will do minimal, processing and aggregator will perform data processing tasks like parts of complex filter and loading to final destination.

A

So you can scale a performance by adding a resources on aggregator and third point is lessons less resource usage in foreign. As I mentioned, the fabricator perform all tasks, so you fo and the point uh for a running endpoint, do not consume many cpu resources.

A

Okay in terms of folder and aggregator, as so as for further endpoint, are typically managed by application team and application team like to minimize resource consumption on the endpoint, and then they also like to minimize dependencies in installation so that we recommend users to deploy frame bit as a folder, because frame bit is a lightweight instance and 3d has no dependencies in installation and all required. Plugins are already included in a rampant agent.

A

As for aggregator, you know it is required to perform data, filtering pulsing and loading tasks, so aggregator need flexibility in data processing and loading, and we recommend users to use run d because, with friend d you can use thousand province available in the community, and you can combine community plugins to achieve uh required. Filtering rules and also friendly has a merge worker function in which enable user to achieve high availability and high performance at scale.

A

Okay, now, let's move on to our scaling performance before we dive in. Let me explain when the data pipeline at first, as you know, randy did pipeline consists of mainly on four component input: filter, buffer and output in front d.

A

First I'll receive messages from data source, and you can define a custom filter right in which data or a filter unnecessary data out and then a fluency will appear for local event as of as a chunk, and we can define a custom flash interval and if, if we reach a first interval of when the output programming will flash buffer event to the destination, there are two areas you we need to consider in terms of performance.

A

First area is balance of input and output. So, if you ingest too many input, data, of course friendly can not a process or all uh ingesting. The data in your first interval.

A

Second area is uh understanding how friendly consumes complete resources? Okay, let me start from input and output.

A

So uh in the production we recommend the users to use process, send file rather than memory how to protect data, so so that it is good to know about our behavior when you think about performance.

A

If the incoming traffic from data source is stronger than our output capability.

A

The amount of buffer will increase linearly and eventually the amount of buffer will reach the system remit. When that happens, a friend d will discard, oldest or latest data, which cause loss of data.

A

On the other hand, if you have enough output capability, ruined d can loaded buffer space so that monitoring the buffer status. If rent d can flash event with infrastructure interval is important to understand, output, behavior.

A

So to understand a characteristic.

A

Of cpu usage, I learned simple, simple benchmark and I use run bit as a data source and friend bit will send data to a friendly aggregator and the friendly aggregators, and I ship messages to kafka in this benchmark. I use a 10k event per sec and I use flashing tower of 30 seconds yep.

A

uh As you can see from this graph, a vertical axis is cpu utilization and horizontal axis is time and during the first fraction interval for nd, accepting trophy from data source and perform filter and buffer process after 30 seconds a fluency flash buffer events to the destination, as you can see from this graph, a friendly consume, cpu resources when french, pressing buffer and you need to design current d to use cpu efficiently to flush buffer or using a defined interval.

A

One of the approaches to use cpu efficiency is to use short flash interval. Like three seconds I learned a similar benchmark, always flashing turbo of 30 seconds and three seconds.

A

As you can see from this graph with frustrating tower or 30 seconds, you can see a peaky behavior. I mean a cpu utilization goes up in every 30 seconds.

A

To avoid this behavior, we recommend the users to use shorter, frustrating turbo right three seconds and with a shorter flush interval, or you can rather cpu utilization.

A

Okay, then, let's move on to performance measurement in this benchmark. uh Ie. You know a friend d, when d has multiple function and I measure performance when adding the number of workers- and I also prepare two instances for this benchmark for virtual cpu.

A

Eight gigabyte memory instance and eight virtual cpu in 16 gigabyte memory instance, and I use friend bit as a folder and friend bit- will tell messages from local file and send messages to aggregator with folder protocol and aggregator will uh receive input and perform filter buffer and then uh ship messages to final destination. Kafka.

A

So uh island benchmark with multiple workers scenario of what, in the first scenario, islam benchmark with a single worker and with a single worker randy, can accept messages up to 12.5 k event per sec, and if we ingest a 15k event for a sec, full nd can not a flash buffer are within defined, flashing double, but still cpu utilization is 25 to 30 percent, which means uh almost 70 percent of cpu is idle status with two workers.

A

Fluency can accept incoming traffic up to 25k and with two workers for nd can use cpu around 60 percent. But when we ingest 30 k event per sec, our full nd cannot flush but for reason interval. Then. I also tried to run same benchmark with four workers and with four workers uh currently can accept a 30k event per sec, and then rcp utilization was 90 to 50, 95 percent.

A

So also with that being said, with multiple workers, you can utilize a cpu resource efficiently and, uh and it is possible to accept more traffic from endpoints.

A

I also run similar benchmark with 4 virtual cpu and 4 workers scenario and eight virtual cpu, and it workers scenario when, with four virtual cpu and four workers we can a friendly can accept 30k event per sec and cpu utilization was almost 90 percent and when with 8 virtual cpu and 8 workers, when the aggregator can accept 60k event per sec and cpu utilization was 65.

A

But when we ingest a 70k event per sec, randy cannot crash buffer within interval.

A

So with that being said, if we increase the number of workers and also increase the number of gpu, you can scale aggregate performance.

A

There is another thing you need to consider when you measure performance is message size, so I also run similar benchmark with.

A

10 byte message size and 100 byte and 400 byte with 10 byte message. A friend d. I can accept 63k events per sec and with hundred byte red d can accept 50k and when I increase the size up to 400, byte frame d can up still accept 30k.

A

So uh when you bring fluency in production, you need to consider typical message, size and long benchmark and also need to design the number of workers and number of cpus based on use cases.

A

Okay, in summary, I talk about how to scale foreign d from not only performance but also operation perspective, so uh in in my experience in enterprise support for full nd, we found that folder and aggregator model helps user to tackle operation challenges at scale and aggregator also resolve issue in managing free to data sources.

A

You know, if you have thousands of data sources and you have aggregator, you do not need to change configuration of data sources and just change configuration by aggregator and also understanding characteristics of fluent bit and fluency for a better resolution architecture, as I described in my presentation, it might be better to understand a buffer, behavior and design a friendly to a flash buffer within defined flashing interval and in my benchmark I observed that performance of randy or could scale by adding cpu and workers.

A

That's it thanks so much for attending my talk. If you have any question, I'm happy to answer.

A

A

Thanks so much.