Calyptia Fluentd, 14 May 2021

Previous Meeting

⏯

youtube image

►

From YouTube: Petabyte Scale Logging with Fluentd and Fluent Bit: A Use Case From... Hanzel Jesheen & Anurag Gupta

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Petabyte Scale Logging with Fluentd and Fluent Bit: A Use Case From Intuit - Hanzel Jesheen, Intuit & Anurag Gupta, Calyptia

In financial and software companies like Intuit, it’s mandatory to provide a reliable observability layer so internal end-users like developers from different line-of-businesses can access their application data. In the last few years, we faced several problems when scaling up logging. For all the new applications we deployed in our Kubernetes clusters, the data rate increased, creating decreased throughput, higher resource consumption, and unexpected backpressure. In this session, you will learn how we leveraged CNCF projects, Fluentd and Fluent Bit, to configure reliable buffering for network outages, flexible configurations for backpressure management, and tips to avoid common mistakes that will save you hours of troubleshooting. A secondary title for this talk would be: “How we achieved Petabyte scale logging with Fluentd and Fluent Bit”.

A

Hi, everyone welcome to petabyte scale logging with fluency influent bit a use case from intuit. My name is onorock, I'm part of the company calyptia and I help run product there.

A

So who are we so today it's going to be myself and hansel who's joining me from intuit who's on the cloud observability team there as a senior software engineer and for myself I'm part of the oss group. That is a maintainer for the fluent bit project and I help drive product for both flipped and flinty.

A

So for those who are unfamiliar with what is fluenty and what what is fluent bit, they are cloud native computing foundation projects. They are part of the graduated projects alongside kubernetes, prometheus and others. They started 10 years ago and the primary problem they solve is how do you take data from point a and send it to point b?

A

And if we look at this problem over the last 10 years, where you see that data sources continually evolve, we have more things like mobile applications system, logs, kubernetes, microservices containers, app metrics, and we have additional destinations that just keep compounding. You might have things like uh amazon s3, you might have splunk elasticsearch, loki, azure, gcp kafka and those destinations just continue to evolve and expand. You might need to send data to multiple. You might need to send data to all of them.

A

You might need to send data to a single one, but in any case, fluent d and fluent bid are really vendor neutral solutions to saying let's collect data once and send it to as many destinations as you require.

A

Now, when we look at challenges for logging at scale, we have to look at this and perspective of a broad overview. One is high. Scale can really equal, very high costs. You can have things like reliability and buffering. uh How do you make sure that logs get sent where they're supposed to? And you don't want to open up a request every single time that you have a new message?

A

Think about that when you're sending 10 000 to 100 000 messages per second networking, always an issue here with things like ephemeral, workloads and kubernetes or you're in the cloud you might have air gapped environments? How do you make sure networking doesn't hurt? You, when you're doing logging at scale event throughput, you might need to maximize how many messages per second you're sending you might have latency requirements. You might have to get messages so that they, your developers, your operations, folks, can go and start to debug and diagnose all that information security.

A

So again, in a large scale environment we might have sensitive information that might make its way through. How do you look at securing the data transition and then, last but not least, when you're running this in production, how do you look at operationality or ways to minimize the performance impacts uh at a larger scale?

A

So what we're we'll talk about in in this session is how into it solved many of these challenges, uh but also how fluent d influent bit have solved these challenges over the course of the 10 years that that project's been running, one is filtering, parsing and compression, so support for compressing workloads being able to take data and throw away stuff that isn't used parsing data, so you can take log messages and make really good sense out of them.

A

So taking like, for example, an apache http server log and giving you the additional information like ips, which you can then go enrich with geo information, things like source ip, the request, type, reliability and buffering so within fluency influence bit. There are file system and memory buffers. So if you need really fast performance- and you want to send bulk messages, you can use the memory buffer by default, but both of these projects also support file system buffering, which is great in case of things crashing.

A

For example, you might have a server that goes down, but because you're, storing everything on the file system, you're able to recover all of that information. None of that data is lost. They all follow in at least once mechanism which leads into networking. So when networking goes down, we don't lose data. We have a configurable retry mechanism as well as back pressure handling. So as the load increases from say, a file or an application, fluentbit is able to understand that. Maybe the application can't receive too many messages at the second and we'll recover.

A

Try at a at a desirable time in the future, and especially when you're logging at scale, you might have thousands and thousands of endpoints or thousands and thousands of applications logging data. How do you make sure that all these retries don't try exactly at the same time and unfortunately cause the denial of service?

A

These are places where we've built a lot of that functionality within the open source. Fluentine fluid bit event throughput another place where, if you need additional performance and you have the resources to dedicate, you can use aspects like multi-worker configuration in both 4d and fluent bit, and and give you the ability to utilize a lot of the server resources that you have at your disposal with security. There's two places to this: if you're securing sensitive information, you can use anonymization filters, you can use parsing to remove those secure fields and then, of course, with tls.

A

Both fluently influent bit support the ability for encryption and transit operationally. We we can use things like different architectures, so both products, both projects have been designed. With this ease of use and flexibility in mind, you might want to minimize the resources spent say at your source within containers within your kubernetes nodes, within your servers and do more of that processing at a centralized layer, and we call that the forwarder aggregate, aggregator called architecture.

A

So just some common architecture patterns right. We have the forwarder only which allows you to take all of these sources and independent pieces and send data they each individually handle back pressure so you're, not necessarily relying on a single point to grapple the data, enrich it and then send it out.

A

The disadvantages is you have to do a little bit more configuration management um and it is sometimes hard to end more destinations, which falls a little bit with the configuration right if I'm sending data to uh splunk, elasticsearch and kafka. At the same time, I might have to go and configure that individually on a on a forwarder basis.

A

Now, a common architecture pattern we see with large scale is forwarders with aggregators, so that is going to be less resource utilization on the edge devices all they're doing is collecting that data and forwarding it onto an aggregator. It allows you to process and and scale the aggregator to your independently.

A

You can add more back ends with this with in a single tier and then, of course, the disadvantagers you do have to dedicate some of the aggregator instance.

A

Now, that's all for just a general overview of how fluenty and fluent bit uh our projects other employees within the cloud native computing foundation, and now how you can look at some of the challenges that they solve for for at scale. I'm really happy to pass it over to hansel who's, going to be able to talk a bit about the use case. Specifics add intuit and give us an idea how petabyte scale can can be achieved in reality. So with that, let me hand it over to hazel.

B

Thank you interact now, we'll get into the use case I can into it into. It provides financial solution for consumers, small businesses and self-employed individuals. Major products from intuit include turbo tax, quickbooks and mint at into it. Kubernetes is adopted widely across four business units. It's used extensively at over 30 segments with more than 5000 engineers onboarded to it. We also have over 80 engineers, developing our kubernetes platform, we'll see how fluent d and fluent bet are used to handle logging from kubernetes cluster set into it.

B

The use case is to transport logs generated within the container, which is returned to std out or std error stream to our persistent lock store to enable this. We have a daemon set deployed in the cluster. This ensures that a logging part is present in each and every node. This part contains fluency of one bit, which is then responsible for collecting the and forwarding the logs from that entire node, the docker logging plugin fights logs into the files which are tailed and processed the log events must also be enriched with some metadata.

B

This would enable us to identify the cluster namespace power and container where each event originates from when running, search, queries at the log store and all this must be fast and efficient. We want to be able to transport huge volumes of data with as low end-to-end latency as possible.

B

We have a centralized, persistent log store where all customers can see and query log events. We wanted a common path to transport, all the logs and it's a pipeline. This would simplify the connectivity between the vpcs containing the kubernetes clusters and the vpc containing the log store.

B

You only have to expose the log store to the pipeline instead of all the clusters. This helps us avoid sending the log events over the internet.

B

Sending data over the internet is very expensive and we can cut down the cost drastically by sending data through aws network only during any connectivity issues with the lock store once the buffers get filled up, the logs are gone for good. If you have an intermediate store for these logs, this can be prevented.

B

This also provides an opportunity for other applications to read the raw log data. This is especially useful for security compliance and analytics applications.

B

This is our name approach, which is a string pipeline. There is a demonstrator, so each kubernetes node will have a logging port running in it. This part will tell the log file from the host mounted as volume.

B

There is a fluent bit plugin that gets the community's metadata from its api and enriches the log events with that. These enriched log events are then sent to a kinesis data stream. The events are then routed to kinesis firehose.

B

Now the kinesis fire hose has a native capability to write to the log store and that's how the english log data gets to the persistent store.

B

This provides a reliable scalable mechanism to transport the logs from hundreds of kubernetes cluster onto a centralized position. Log store. We only need access to the lobster from fire hose, and this data transfer happened within a releases network. Also, the logs can be read from kinesis data stream, while just using a consumer.

B

This pipeline was working well for us initially, but as we started supporting more and larger cluster, we started facing some challenges.

B

With this pipeline, we had a hard limit on the number of events that can be transported every second in each pod, with the solution that used fluency as a logging container, we were capped at 2500 events per second per node.

B

Once we should serve it to fluent bit base logging container, we were able to double that limit to 5000 events per second per node. However, this was still not enough, especially for larger nodes, which contain a lot of containers packed inside.

B

Also, as you saw, there are two hops between the cluster and the log store. Each of these hops were introducing some latency, which was quickly adding up. We started seeing the median n2 and latency getting as high as 30 seconds.

B

This was not acceptable for our most time critical applications, so we decided to reduce the number of hops and hence the time it takes for the transport to maintain this streaming pipeline. We had to have some persistent resources running the lock load that needed to be transported was extremely elastic, especially based on different times of the day.

B

However, these resources, even with the auto scaling, enable were not able to scale efficiently to match that log load. This was making the pipeline inefficient in terms of the cost and ended up being too expensive.

B

We wanted to cut down some costs as well, so in simpler terms, the target for the new pipeline were to increase the throughput dramatically without increasing the resource consumption by that port, reduce the end-to-end latency of the log transport pipeline as much as possible and to minimize the cost needed to maintain it, and this is where the s3 pipeline comes into play.

B

This is how the architecture of s3 pipeline looks like the space enclosed by the blue. Dashed line represent a node running in the kubernetes cluster. Each node contains a daemon set power containing fluently.

B

The locks written by the container on its std out and std error streams will be written as files that are rotated in the node, and it's all done by the docker daemon.

B

These files are exposed to the container by using a volume mounted directly onto the host the fluency container, then tails these files and listens for any new log events and for any new log events fluently, pushes it onto a buffer, which is specific for each container at a certain fixed interval 10 seconds. In this case, the buffers are flush onto an s3 bucket object after compressing it with g6.

B

So each s3 object would contain logs generated by the container for 10 seconds compressed as a jzip file in the pipeline fluently is not responsible for enriching the log data with metadata, but rather provides the logs, as is onto the object.

B

Now we have a separate deployment in the cluster which ensures that a single power will be running across this cluster. This deployment communicates with kubernetes api and gets information about any new containers for any new containers created in the cluster. It will fetch its placement and other metadata, including its labels and annotations, and pushes it on to a separate extra bucket within the log pipeline vpc, we have a deployment running in aws, far gate. This deployment listens to any new objects in the log s3 packet for any new object in this bucket.

B

The path would indicate the container for which the logs are for the log pipeline deployment, then fetch the corresponding metadata from the metadata bucket.

B

Both these are then sent over together onto the log store present in a separate icon and bpc. This communication happens over aws's transit gateway. The log store is then able to apply the metadata to each and every event in the log s3 object.

B

The key takeaways in the architecture are that the log data is compressed from the cluster and only deflated at the log store and all the data transfer happens within the aws network and, moreover, crossover transfer have been via a transfer gateway created specifically for that purpose.

B

As you can see, fluently is deployed in the cluster as a daemon set. This ensures that there is a single power in every node and each pod is then responsible for transporting logs from all the containers in that node. Regardless of how big that node is from this, we can realize that this part will become the bottleneck and the weakest link in the chain, so the throughput will be set by the moon that can be transported by a single part.

B

The biggest gain in the throughput can be achieved by making demonstrate process transport more and more events. One mechanism to achieve this is to do as little processing as possible at the daemon set part.

B

That is the key to note in the new pipeline design. Next, we wanted to identify all the things that can be offloaded from the fluency and how we facilitate that function. Elsewhere.

B

After multiplication, we were able to pick two cpu intensive processors that can be offloaded without any change in the events at the log store. These were providing us with this massive, genuine support.

B

So the first thing that we were able to offload was timestamp parsing timestamp has to be passed from each event for a couple of things, to attribute a timestamp for each event and to identify the starting line of new event in case of multi-line events.

B

This was very cpu intensive, but we were able to offload this onto our log store without any change in the end user experience. The second thing we offloaded is the metadata enrichment in the previous pipeline. Each event was enriched with the required metadata by fluency itself.

B

This has now been offloaded and done by the metadata deployment external log pipeline deployment and the log store. The metadata deployment communicates with the kubernetes api to get the required container metadata in an s3 object. The external log pipeline deployment then collates this container specific metadata with each s3 log object, which is also container specific.

B

Both these are then sent together to the log store, which is up, which can then apply the metadata to all the live events in the file. The additional advantage with this is that the metadata enrichment happens in batch across all the events in ns3 object at once. This reduces the processing needed and the amount of data transfer.

B

When looking at the cost breakdown for the pipeline, we found that the data transfer cost was the highest component higher than even compute and storage. So if we can reduce the amount of data sent over and use a more cost effective way to transport it, they can make huge savings in the cost. One added benefit is that, as we reduce the amount of data transfer, the latency also goes down, as mentioned before, fluently compresses the data via gzip and the buffer.

B

When it's flushed to an s3 object in our normal load load, we were getting around 10 times compression. This gc file then be returned to log store asses.

B

What that means is that the log data stays compressed as soon as it leaves the clusters and until it reaches the long story or the log data is always compressed in transit. Just this one change cut down the data transferred to one tenth, the volume to that of before, also as we are applying the metadata in batch for an entire three object, each object even need not have the metadata along with it. This was also giving us large strings in terms of the amount of data transferred by cutting down the size of each row.

B

Log event: this is when we compare it to the previous string pipeline. We also have set up an aws transit gateway between our log pipeline vpc and the log story. Pc and all the data transfer happens through this. This was giving us around 60 in cost saving, as opposed to sending the data over the internet using an app gateway to demonstrate how this new pipeline performs. I have set up a test united cluster with three worker nodes.

B

Each node contains the daemon set bar and a few other parts to generate some sample log events. This cluster also contains the metadata deployment, as you can see, from our metrics dashboard, that with three nodes, it's able to transport 90, 000 events per second without any errors.

B

So that means that each node can support up to 30 000 events per second. We also see that the cpu of the parts stays stable and that it can constantly support the log over a period of time.

B

The lock generator in this each node collectively was generating 30 000 events per second and we have three such nodes, so the total log load was around 90 000 events per second, and I ran this log test for an hour, so that should give us 324 million events at our lock store. That's exactly what we see in our log dashboard.

B

We can now see that the end to end latency, which is a time difference between the container logging and event and the logs, are indexing it. We see that the median end-to-end latency is just around 9 8 seconds and the 99th personnel is less than 14 seconds. That's a massive drop from our streaming pipeline.

B

Let me give a summary on the improvements of this s3 based pipeline over a streaming pipeline that we saw from the demo.

B

In the case of throughput, we saw a massive six times jump in the number of events that can be transported from 5000 events per second per node to 30 000 events per second per node. Moreover, we have tested transporting over one gigabyte per second from a single kubernetes cluster.

B

Next up, we see that the median engine latency drops to less than one-third from around 30 seconds in the streaming pipeline to just over eight seconds in the s3 based pipeline. The difference grows considerably when considering the 99 first l, which has a latency cut down by more than 75.

B

Finally, looking at the cost, we see the cost saving of more than 92 when compared to the streaming pipeline. This is due to not having to keep persistent resources running compression of the log data and using transit gateway. This was coming around to more than 50 000 us dollars saved over every petabyte transfer.

B

This is a story of how we were able to create a petabyte scale, logging pipeline at intuit, using fluency and fluent bit. Thank you all so much for participating in this session, and I would like to open up the forum for questions at this point. Thank you.