Cloud Native Computing Foundation FluentCon North America 2021, 30 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Integrating Fluent Bit Into a PaaS - Bojan Djurkovic & Dhruv Kela, DigitalOcean

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Integrating Fluent Bit Into a PaaS- Bojan Djurkovic & Dhruv Kela, DigitalOcean

At DigitalOcean App Platform we use Fluent Bit for log processing of internal components and for customer-facing functionality. This session will review how we collect internal logs from dozens of Kubernetes clusters into a centralized logging system utilizing a custom plugin. We will also describe how we dynamically configure Fluent Bit for customer applications' log processing. This presented several operational, scalability, and security considerations. Orchestrating Fluent Bit in a security-restricted environment proved challenging. We also need detailed observability into applications' logging pipeline health and metrics. Different implementation approaches and potential future improvements will be discussed.

A

Hello: everyone, I'm dhruv, I'm a I'm a software engineer with the marketplace team at digitalocean.

B

Hello and my name is bayan and I'm a senior server engineer on the app platform team.

A

Today, we're going to be talking about integrating fluent bit into a pass when the context is a platform as a service. The challenges are different and hopefully we'll be able to go through some of them today. So as for the agenda, we are going to first provide a context as to what our platform is and what are we working with.

A

uh Then we would move towards internal logging, external logging, the issues which we faced and scope for future improvement.

A

Digitalocean app platform is a fully managed platform as a service that lets customers build, deploy, manage and scale all kinds of different applications.

A

These applications could be web apps, static sites, apis background workers and in a different of a different number of languages such as php, python, ruby, javascript, etc.

A

Use this in create applications via our cloud dashboard command line, tool or api, regardless of the mechanism used, the application ends up represented as a manifest specifying different components in the application each application may have one or more component. For example, an app may have an http web api and a background worker.

A

We take these components, stitch them together, build them and deploy them on our platform.

A

App platform is built on top of existing digital ocean products. Primarily, we are built on managed kubernetes container registry and spaces offering, but underneath we leverage almost all core infrastructure products.

B

App platform consists of numerous kubernetes clusters in various digital ocean regions and data centers. We treat our clusters as cattle, not pets, and can reconfigure, destroy and deploy new clusters as needed at a high level. There is a core cluster reconciliator component or management control, plane, that's responsible for managing clusters in the data plane.

B

Each cluster has several node poses some dedicated to running customer application, workloads some are dedicated to posting application, builds and some host our internal in-class or control services such as prometheus, istio and celium.

B

All the reconciliation is done, programmatically using go client, libraries and the combination of template and manifest and go code.

B

Similarly, there is a control component responsible for managing the builds and deployments of actual customer applications. These are orchestrated completely dynamically, as well as needed via go client within each cluster. We need logging visibility for our in cluster control components.

B

These logs need to be ingested securely via websocket into digital ocean central logging system. In our main management management control plane, we use flow and bit deployed as a daemon set within cluster to collect processes analogs into the central login system.

B

We use a custom, websocket output plugin to send the logs, and this is an example of our custom website and plugin configuration looks like we have options to specify the end point and any additional headers. For example, the authorization bearer token.

A

While we just covered internal login, a different use case is where we want to send actual customer application logs to other logging systems. We want to support sending logs using syslog or http protocols.

A

Applications run in a secure, isolated, sandbox environment using the g-wiser runtime fluent pods that perform the actual lock collection and processing also need to run in the sandbox environment.

A

We need dynamic and fine-grained control over logging configuration as applications, change are deployed or re-allocated around the cluster.

A

For security and performance purposes, we need to be able to throttle and dynamically disable logging for applications. We also need to have detailed, observability and insights into the health and metrics of different applications. Log logging, behavior, for example. We need to track how much logging data each app is outputting.

A

We need to know if logging is working and is healthy for a specific application. We need to know the overall health of the logging system itself.

A

All of this needs to happen dynamically and programmatically as part of a normal application reconcile flow covered.

A

Earlier for application logging, we choose to utilize fluent bit operator to orchestrate all the necessary components and configuration we deploy one instance of the fluid bit operator in each of the clusters.

A

The operator manages the fluent bit daemon set, which controls the fluid bit pods on all the relevant customer workload uh nodes as applications are deployed by the apps reconciler discussed earlier input, filter and output. Custom resources are created for each application and login destination. Endpoints these custom resources get reconciled by the fluent bit operator into a secret containing the fluent bit configuration file.

A

This gets mounted to the fluent bit pods, which then performs the actual log processing. One initial problem which we had was that the fluid bit operator did not expose the go client. We had to add the go client packages to up to the operator in order to use the custom resources programmatically.

A

This is an overview of a single applications. Logging pipeline. We collect application logs using the tail input. Since we are running on container d, we use the cri parser to parse the log records. Next, we throttle the log processing.

A

We also use modify filter to set some additional application, specific metadata to the log records and finally, we route the records to the appropriate output based on the application's login configuration.

A

So here is an example of an input. Configuration applications live in their own namespace, which is determined by a unique identifier. We have to be very specific about the input path we specify, so we only collect the current running containers in the apps name space we ensure we do not collect any kubernetes system logs and we tag the app very specifically so that we can control the routing by default. The tail input uses input output.

A

Notify system calls to collect data from log files as it is written to, because both the application pods and the fluent bit pods run in separate gvisor environments. The file system call restrictions enforced by g-wiser, make this type of lock collection, prohibitive and non-functional.

A

Fortunately, fluent bit eight point: fluid 1.8 came with the tail input plug-in option to use stat watcher to collect log data which works around sandbox issues.

A

The modify filter.

A

Is used to insert additional metadata into the log records? These include the component name and the app name for outputs. We match exactly the component within the app that needs to be that needs to get sent to the log destination for syslog output. We enhance the transport record with the enriched metadata values inserted in the modify filter.

A

These then, can be parsed at the destination to associate logs with the actual app and components for nicer, aggregation.

B

For monitoring all of this, we have some specific needs for detailed metric collection in the log processing pipeline.

B

We need to be able to tell on a per component level how many log records and how much bandwidth has been processed. We need to be able to collect all the data all of this data in the prometheus exposition format, and we need to be able to connect the metric values to specific application and customer identifiers.

B

We would like to have a time series with metadata such as an app name id and customer id like seen here. Flimbit provides an api for exposing plug-in metrics. However, the api is limited in the data it exposes. He only provides the name of the plugin.

B

Furthermore, even when using an alias to uniquely identify plugins, the name exposed in the matrix api is truncated to 32 characters. Even new uids cannot fit nicely in the alias name.

B

Ideally, we will like a solution similar to this, where we can specify the extra monitoring labels for the plugin, which then would be exposed in the monitoring api by fluent date. We have created a github issue that hopefully can be addressed at some point in the near future.

B

As a worker, we dynamically generate prometheus recording rules for enhanced metric collection. This enables us to add automated metadata to the ingested metrics as they're scraped from the fluency api.

B

The name is a unique identifier for the plugin that we generate that can fit in the alias field and connects metrics to app and customer ids.

B

This is a summary of some of the issues we've encountered when working with fluent bits. The tail input plug-in did not work under gvisor and we had to use the disabled. I notified watcher option that only became available in 1.8, dynamic configuration of fluid within kubernetes is still cumbersome requires some requirements.

B

A fluid bit operator uses a custom process called fluid bay watcher that watches for the changes within the configuration configuration file and restarts the actual flowing bit process. When changes happen, the fluent bit watchers also uses input output, notify system calls which do not work under gvisor. In this context, so we had to add support for stat polling to fluently. Watch your component within the operator to enable dynamic configuration within g visor, as mentioned earlier flimbit operator, did not have go client, you had to add it as well.

B

Finally, uh with some investigation, we found an issue where fluent bit operator is using a lot of cpu, rendering the configuration file into the secret. We had a fix for this and submitted a pr upstream.

B

For some future work, we wanted to explore the native fluent bid website plugin introduced in 1.7 and see if that can fit our needs like our custom. One gvisor encourages performance mentally when working with file system io. We want to investigate this further with respect to tail input, plugin performance.

B

uh Additionally, we want to optimize the fluent bit operator operator, fluid bit operator stores, a single config file for all the fluency damage and in a single kubernetes secret data in kubernetes secrets has one mappy byte size image, which can be problematic if we store large amounts of configuration data, one relatively easy improvement might be to use compressions such as gzip to shrink the configuration file. Additionally, we can try to fix this with some more work and display the config files as they grow and rejoin them with their red by film bit pods.

B

Finally, we would like to solve the matrix metadata really re-labeling, the issue that we mentioned earlier for our monitoring functionality.

B

Thank you and we can answer any questions.