Calyptia / Fluentcon Europe 2022

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Fluent Bit: Logs, OpenMetrics, and OpenTelemetry all-in-one - Eduardo Silva & Anurag Gupta, Calyptia

Observability continues to be an interesting challenge, data collection for logs, metrics, and traces is an expensive operation and different approaches exist. Fluentd and Fluent Bit have been the long-term defacto standard for logging, but recently the projects expanded their scope to support and assist with Metrics and Traces. Fluent Bit now supports data collection, aggregation, and delivery for the world of OpenMetrics (Prometheus) and OpenTelemetry. In this presentation, you will learn how to collect and aggregate Logs, Metrics, and Traces all together without losing control of your data by connecting different protocols at scale. Fluentd and Fluent Bit embraces OpenMetrics and OpenTelemetry vision, come to learn how to optimize your observability pipelines, and implement the best practices for your production environments.

Challenges of Logging on the Edge - Mickey Pashov, Medtronic Digital Surgery

In this talk we will describe our journey from arbitrary log collection from edge devices to building an observability pipeline with FluentBit. We will discuss the reasons why we chose it over other agents - highly performant with a small memory footprint, open-source, vendor-agnostic, great community support, extensibility via Lua filters, and a plethora of core input and output plugins. Some of the challenges of deploying on-prem such as firewall and web filter rules; authN and authZ to the cloud; offline-mode and packaging and deployment using an unfamiliar toolchain. We will also discuss the the challenges of deploying an observability pipeline on Kubernetes with FluentD. From setting up the Helm charts and livenessProbes to TLS termination and autoscaling.

Demystify the Fluent Core Event Loop and Join in on its Optimization Efforts - Matthew Fala, Amazon

At the most fundamental level, Fluent Bit is an event driven system that relies on a core event loop running on each thread to initiate, resume, and clean routines required for delivering data. Have the core Fluent Bit event loop demystified, and deep dive into the data uncovering existing problems that occur due to the event loop in high throughput use-cases. The talk brings you into the community discussion on how to best optimize this event loop system and uncovers efforts spent on implementing the identified improvements. For the sake of understandably, a highly correlated analogy will be conveyed to modal the central concepts (the event loop, coroutines, cpu, yield, ready list, blocking wait) providing means to think through event loop problems and ideate solutions conceptually without full knowledge of unrelated systems. Attendees should after the talk be able to: 1. Understand the Event Loop system conceptually 2. Gain insight into the conceptual problems of the Event Loop system 3. Gain insight into the currently proposed conceptual solutions 4. Have the knowledge and tools to join the discussion, and propose new solutions.

Fluent Community Meeting

Hands-on Workshop: Fluent Operator | Intro - Benjamin Huo & Han Zhu, QingCloud Technologies

With the support of Fluentd, the original Fluent Bit Operator has been renamed to Fluent Operator. In this session, Fluent Operator maintainers will elaborate on the key features of Fluent Operator as well as its design principles and architecture including: 1. From Fluent Bit Operator to Fluent Operator. 2. Fluent Operator design principles. 3. Fluent Operator Architecture. 4. Use Fluent Bit as a light-weighed logging agent on Kubernetes. 5. Use Fluentd as a global log forwarding & aggregation layer on Kubernetes. 6. Build a flexible and multi-tenant log processing pipeline for Kubernetes with Fluent Bit and Fluentd. 7. Demo. 8. Community & Roadmap.

Hands-on Workshop: Getting started with Fluent Bit and Metrics with Prometheus - Anurag Gupta, Calyptia & Gibbs Cullen, Chronosphere

The Fluent Ecosystem has recently added support for Prometheus and Open Metrics. This hands-on session will give attendees a chance to both learn and try out the new capabilities while being guided by a Fluent maintainer and Prometheus community expert. In this session we will cover the following:

An introduction to Fluent ecosystem and Prometheus metrics
How to spin up Prometheus as well as visualization platform (Grafana)
Setting up Fluent Bit to collect and send metrics to the backend; Viewing and visualizing data that is sent by Fluent Bit
Question and answer section

This session is suitable for learners of all skill levels, and attendees can expect to leave having a better grasp of how these Cloud Native projects can be used together to strengthen observability.

Lightning Talk: Data Flow Control in Cluster Logging Pipeline - Pranjal Gupta & Eran Raichstein, IBM

Logging pipelines are crucial in ensuring container logs are reliably collected and routed to persistent storage. Logs generated by workloads (container processes) are written to files by Container Monitor processes (e.g. Conmon). In production environments, as Fluentd deals with a massive volume of logs, the log generation rate often exceeds the rate of log collection, which causes log loss. There is a need to prioritise application logs so that administrators can collect logs from high priority workloads in a controlled manner. In this talk, we introduce a new feature in the in_tail input plugin, which uses group rules to rate limit log collection. We share exciting insights from our systematic study about log loss on Fluentd plugins using our open-source benchmarking framework. We also present a Log Flow Control framework that allows users to define and enforce log rate limit policies to control log loss predictably.

Modernizing to the Fluent Stack: An Asana Story - James Elías Sigurðarson, Asana

In 2021, Asana migrated its event emission infrastructure to use Fluent Bit, from an old logging system based on Facebook's Scribe. At Asana, event emission requires high flexibility, performance, and durability, as it powers important Asana features, such as mobile notifications. In this session, we'll take a look at how we modernised our infrastructure, and used the Fluent Node library to meet these challenges, eventually scaling up to 3.5 billion events per day.

Monitoring Fluent Bit in Production - Pandu Aji, Microsoft

Fluent Bit comes with built-in features to allow you to query internal information and monitor metrics of each running plugin. But how do you decide which metrics to use for measuring the reliability of your pipeline? How do you leverage the storage metrics, and what are the challenges? How do you detect if your logging pipeline is unhealthy and if there is any congestion? This talk will describe our Fluent Bit monitoring story. It will briefly touch on some of the design choices but will primarily focus on how to monitor Fluent Bit in the production clusters.

Opening + Welcome: Fluent Bit in 2022: Logs, Prometheus and OpenTelemetry - Eduardo Silva, CEO & Founder, Calyptia

Observability continues to be an interesting challenge, data collection for logs, metrics, and traces is an expensive operation and different approaches exist. Fluentd and Fluent Bit have been the long-term defacto standard for logging, but recently the projects expanded their scope to support and assist with Metrics and Traces. In 2022, Fluent Bit now supports data collection, aggregation, and delivery for the world of OpenMetrics (Prometheus) and OpenTelemetry. In this presentation, you will learn how to collect and aggregate Logs, Metrics, and Traces all together without losing control of your data by connecting different protocols at scale. Fluentd and Fluent Bit embrace OpenMetrics and OpenTelemetry vision, come to learn how to optimize your observability pipelines, and implement the best practices for your production environments.