30 Oct 2021
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
From FluentBit Operator to Fluent Operator: The Long Way to Come- Benjamin Huo, QingCloud Technologies
FluentBit Operator was created by the KubeSphere community to solve several problems:
1. Collect K8s logs through a light-weighted agent like Fluent Bit.
2. Control Fluent Bit via Kubernetes API.
3. Collect logs and then send them to the final destination without having to go through Fluentd.
4. Enable dynamic config reloading for Fluent Bit to reload its config whenever the config changes without restarting the Fluent Bit Pod.
FluentBit Operator has reached its maturity level gradually after two and a half years of iterations, and KubeSphere and Fluent Community collaborated to transfer it to the Fluent organization.
Sometimes Fluentd is still needed as an optional sink to aggregate logs received from Fluent Bit, then filter and forward to more destinations. So FluentBit Operator maintainers are considering adding Fluentd CRDs and rename the project to Fluent Operator.
In this talk, FluentBit Operator maintainers will talk about the architecture and use cases of FluentBit Operator and the design of Fluent Operator.
From FluentBit Operator to Fluent Operator: The Long Way to Come- Benjamin Huo, QingCloud Technologies
FluentBit Operator was created by the KubeSphere community to solve several problems:
1. Collect K8s logs through a light-weighted agent like Fluent Bit.
2. Control Fluent Bit via Kubernetes API.
3. Collect logs and then send them to the final destination without having to go through Fluentd.
4. Enable dynamic config reloading for Fluent Bit to reload its config whenever the config changes without restarting the Fluent Bit Pod.
FluentBit Operator has reached its maturity level gradually after two and a half years of iterations, and KubeSphere and Fluent Community collaborated to transfer it to the Fluent organization.
Sometimes Fluentd is still needed as an optional sink to aggregate logs received from Fluent Bit, then filter and forward to more destinations. So FluentBit Operator maintainers are considering adding Fluentd CRDs and rename the project to Fluent Operator.
In this talk, FluentBit Operator maintainers will talk about the architecture and use cases of FluentBit Operator and the design of Fluent Operator.
- 1 participant
- 26 minutes
30 Oct 2021
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Getting More Out of Lua in Fluent Bit- Pranav Marla, Fidelity Investments
Fluent Bit has many useful plugins and features but by far its most powerful feature is its support for custom scripts written in a programming language called Lua. Unfortunately, it can also often be one of Fluent Bit's most underutilized features, with most users limiting themselves to minor variations of the sample scripts provided in Fluent Bit's documentation. This is because Lua is not as well-known as other programming languages, and its unfamiliarity can make it seem intimidating to people and prevent them from taking full advantage of its capabilities. This talk aims to help users become more comfortable using Lua outside the bounds of those sample scripts by walking through some novel scenarios, as well as sharing some tips and tricks, to help users take advantage of Lua to unlock Fluent Bit's true potential.
Getting More Out of Lua in Fluent Bit- Pranav Marla, Fidelity Investments
Fluent Bit has many useful plugins and features but by far its most powerful feature is its support for custom scripts written in a programming language called Lua. Unfortunately, it can also often be one of Fluent Bit's most underutilized features, with most users limiting themselves to minor variations of the sample scripts provided in Fluent Bit's documentation. This is because Lua is not as well-known as other programming languages, and its unfamiliarity can make it seem intimidating to people and prevent them from taking full advantage of its capabilities. This talk aims to help users become more comfortable using Lua outside the bounds of those sample scripts by walking through some novel scenarios, as well as sharing some tips and tricks, to help users take advantage of Lua to unlock Fluent Bit's true potential.
- 1 participant
- 25 minutes
30 Oct 2021
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Hands On Workshop: Getting Started With Fluent Bit and Metrics With M3DB - Anurag Gupta, Calyptia & Gibbs Cullen, Chronosphere
The Fluent Ecosystem has recently added support for Prometheus and Open Metrics. This hands-on session will give attendees a chance to both learn and try out the new capabilities while being guided by a Fluent maintainer and Prometheus community expert. in this session we will cover the following: * An introduction to Fluent ecosystem and metrics * How to spin up an open source metrics backend (M3DB) as well as visualization platform (Grafana) * Setting up Fluent Bit to collect and send metrics to the backend; Viewing and visualizing data that is sent by Fluent Bit * Question and answer section. This session is suitable for learners of all skill levels, and attendees can expect to leave having a better grasp of how these Cloud Native projects can be used together to strengthen observability.
Hands On Workshop: Getting Started With Fluent Bit and Metrics With M3DB - Anurag Gupta, Calyptia & Gibbs Cullen, Chronosphere
The Fluent Ecosystem has recently added support for Prometheus and Open Metrics. This hands-on session will give attendees a chance to both learn and try out the new capabilities while being guided by a Fluent maintainer and Prometheus community expert. in this session we will cover the following: * An introduction to Fluent ecosystem and metrics * How to spin up an open source metrics backend (M3DB) as well as visualization platform (Grafana) * Setting up Fluent Bit to collect and send metrics to the backend; Viewing and visualizing data that is sent by Fluent Bit * Question and answer section. This session is suitable for learners of all skill levels, and attendees can expect to leave having a better grasp of how these Cloud Native projects can be used together to strengthen observability.
- 2 participants
- 23 minutes
30 Oct 2021
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Integrating Fluent Bit Into a PaaS- Bojan Djurkovic & Dhruv Kela, DigitalOcean
At DigitalOcean App Platform we use Fluent Bit for log processing of internal components and for customer-facing functionality. This session will review how we collect internal logs from dozens of Kubernetes clusters into a centralized logging system utilizing a custom plugin. We will also describe how we dynamically configure Fluent Bit for customer applications' log processing. This presented several operational, scalability, and security considerations. Orchestrating Fluent Bit in a security-restricted environment proved challenging. We also need detailed observability into applications' logging pipeline health and metrics. Different implementation approaches and potential future improvements will be discussed.
Integrating Fluent Bit Into a PaaS- Bojan Djurkovic & Dhruv Kela, DigitalOcean
At DigitalOcean App Platform we use Fluent Bit for log processing of internal components and for customer-facing functionality. This session will review how we collect internal logs from dozens of Kubernetes clusters into a centralized logging system utilizing a custom plugin. We will also describe how we dynamically configure Fluent Bit for customer applications' log processing. This presented several operational, scalability, and security considerations. Orchestrating Fluent Bit in a security-restricted environment proved challenging. We also need detailed observability into applications' logging pipeline health and metrics. Different implementation approaches and potential future improvements will be discussed.
- 2 participants
- 14 minutes
30 Oct 2021
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Keynote: Accelerating Adoption of Fluentd/Fluent Bit With Confidence- Hisatomo Takana, ITOCHU Techno-Solutions (CTC) America, Inc.
Fluentd/Fluent Bit CNCF projects have been accessible to everyone as we embrace the community effort to bring vendor-neutral, unified, and robust approaches. Through our Fluentd community survey, we have discovered some challenges that community members have when using Fluentd/Fluent Bit.
In this session, we will discuss our approaches on how to reduce operational challenges. We have been contributing to "Diagtool", which is already available in td-agent, allowing you to automate troubleshooting. In terms of security, we developed the "Sanitizer" plugin which masks sensitive information on the fly and helps you to solve security incidents such as data breaches by making data meaningless for attackers.
We, ITOCHU Techno-Solutions America, Inc, are a part of the community to accelerate the adoption of Fluentd/Fluent Bit through commercial services and contributing plugins/tools for our community.
Keynote: Accelerating Adoption of Fluentd/Fluent Bit With Confidence- Hisatomo Takana, ITOCHU Techno-Solutions (CTC) America, Inc.
Fluentd/Fluent Bit CNCF projects have been accessible to everyone as we embrace the community effort to bring vendor-neutral, unified, and robust approaches. Through our Fluentd community survey, we have discovered some challenges that community members have when using Fluentd/Fluent Bit.
In this session, we will discuss our approaches on how to reduce operational challenges. We have been contributing to "Diagtool", which is already available in td-agent, allowing you to automate troubleshooting. In terms of security, we developed the "Sanitizer" plugin which masks sensitive information on the fly and helps you to solve security incidents such as data breaches by making data meaningless for attackers.
We, ITOCHU Techno-Solutions America, Inc, are a part of the community to accelerate the adoption of Fluentd/Fluent Bit through commercial services and contributing plugins/tools for our community.
- 1 participant
- 10 minutes
30 Oct 2021
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Keynote: Scaling Observability Pipelines in the Enterprise- Anurag Gupta & Eduardo Silva, Calyptia
Our infrastructure is continuously growing, generating more data. Extracting value from that growing data and scaling up analysis is a constant pain. Modern tools need to solve both critical performance tasks like data collection as well as pre-processing enrichment and delivery.
Fluentd and Fluent Bit CNCF projects are the de facto Logging and compatible Metrics solution for any cloud environment. In this session you will learn about our journey on scaling up data processing for financial institutions, performance improvements and how Calyptia, the company founded by Fluent creators and maintainers, is taking the Fluent ecosystem to the next enterprise level through cloud services and free tools for our community.
Keynote: Scaling Observability Pipelines in the Enterprise- Anurag Gupta & Eduardo Silva, Calyptia
Our infrastructure is continuously growing, generating more data. Extracting value from that growing data and scaling up analysis is a constant pain. Modern tools need to solve both critical performance tasks like data collection as well as pre-processing enrichment and delivery.
Fluentd and Fluent Bit CNCF projects are the de facto Logging and compatible Metrics solution for any cloud environment. In this session you will learn about our journey on scaling up data processing for financial institutions, performance improvements and how Calyptia, the company founded by Fluent creators and maintainers, is taking the Fluent ecosystem to the next enterprise level through cloud services and free tools for our community.
- 2 participants
- 9 minutes
30 Oct 2021
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Scaling Fluentd From 0 to Millions of Events- TK Kubota, ITOCHU Techno-Solutions America, Inc
We have seen the impact of data formats, messaging sizes, event rates, and even data sources causing challenges. For the successful operation in a large scale environment, understanding the performance characteristics of Fluentd and designing a proper solution architecture are important. TK has had the opportunities to work with many large institutions with thousands of servers deployed with Fluentd / Fluent Bit and will show how internals of Fluentd / Fluent Bit impact performance as well as benchmark results. We will share tips and tricks on how to optimize and tune parameters for your own deployments based on characteristics of input, filter, buffer and output pipelines. Lastly, we will discuss how to go beyond tuning and how to increase performance with additional compute / multi workers / and deployment architectures.
Scaling Fluentd From 0 to Millions of Events- TK Kubota, ITOCHU Techno-Solutions America, Inc
We have seen the impact of data formats, messaging sizes, event rates, and even data sources causing challenges. For the successful operation in a large scale environment, understanding the performance characteristics of Fluentd and designing a proper solution architecture are important. TK has had the opportunities to work with many large institutions with thousands of servers deployed with Fluentd / Fluent Bit and will show how internals of Fluentd / Fluent Bit impact performance as well as benchmark results. We will share tips and tricks on how to optimize and tune parameters for your own deployments based on characteristics of input, filter, buffer and output pipelines. Lastly, we will discuss how to go beyond tuning and how to increase performance with additional compute / multi workers / and deployment architectures.
- 1 participant
- 22 minutes
30 Oct 2021
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Security Logging Use Cases: Building an Open-Source SIEM- Jonah Kowall, Logz.io
Understanding threats is the goal of a SIEM, which collects, enriches, and correlates events and threats. Learn how to build an open-source SIEM with Fluentd and Fluent Bit. SIEMs primary challenge is data collection and scale. Security infrastructure generates diverse data. We will cover data sources along with how to parse and security signals. We will provide real-world examples of how these data collection systems are used to bring together security data into an open-source SIEM. Learn how extracting metrics from logs with Fluentd can provide additional data to understand your organization’s security posture. The EFK Stack is very popular for log analytics. This includes the life cycle of collection, indexing, and storing them. Log data is valuable, but use cases for logging are operational for observability and debugging. The security world typically uses other tools, but building it on top of the same logging stack is efficient. SIEM takes a centralized approach to collection, enrichment, and analysis. As we know in today’s environments, we must ultimately federate this work to the edge to reduce data volumes and take action faster. While this is not something that SIEMs do today, it’s something that the future SIEM and technologies like Fluentd will provide.
Security Logging Use Cases: Building an Open-Source SIEM- Jonah Kowall, Logz.io
Understanding threats is the goal of a SIEM, which collects, enriches, and correlates events and threats. Learn how to build an open-source SIEM with Fluentd and Fluent Bit. SIEMs primary challenge is data collection and scale. Security infrastructure generates diverse data. We will cover data sources along with how to parse and security signals. We will provide real-world examples of how these data collection systems are used to bring together security data into an open-source SIEM. Learn how extracting metrics from logs with Fluentd can provide additional data to understand your organization’s security posture. The EFK Stack is very popular for log analytics. This includes the life cycle of collection, indexing, and storing them. Log data is valuable, but use cases for logging are operational for observability and debugging. The security world typically uses other tools, but building it on top of the same logging stack is efficient. SIEM takes a centralized approach to collection, enrichment, and analysis. As we know in today’s environments, we must ultimately federate this work to the edge to reduce data volumes and take action faster. While this is not something that SIEMs do today, it’s something that the future SIEM and technologies like Fluentd will provide.
- 1 participant
- 33 minutes
30 Oct 2021
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Why Do My Logs Suck?! A Practical Guide to Improving Your Logs- Patrick Stephens, Couchbase
For the cloud-native transformation of Couchbase, we had to do something about our logs. We had various logs scattered around from different language binaries (Java, Erlang, C, etc.) and in different formats with none fit for “human consumption”.
If you’ve ever started tearing your hair out over differing log formats then this is the talk for you! If you have ever tried to understand what a developer was thinking when adding a log over multiple lines then this is also the talk for you! We include some specific examples of difficult logs and how we solved them with no change to the log itself.
Pragmatism was essential:
We did not have the resources to refactor every log statement across all these teams/codebases.
We needed to support an approach that allowed us to provide an out-of-the box solution but also easy integration with anything already deployed.
We had a desire to focus on CNCF or at least open source solutions that are loosely coupled to lower the limits for adoption.
A blog post covering more in-depth examples to support this presentation can be found here.
Why Do My Logs Suck?! A Practical Guide to Improving Your Logs- Patrick Stephens, Couchbase
For the cloud-native transformation of Couchbase, we had to do something about our logs. We had various logs scattered around from different language binaries (Java, Erlang, C, etc.) and in different formats with none fit for “human consumption”.
If you’ve ever started tearing your hair out over differing log formats then this is the talk for you! If you have ever tried to understand what a developer was thinking when adding a log over multiple lines then this is also the talk for you! We include some specific examples of difficult logs and how we solved them with no change to the log itself.
Pragmatism was essential:
We did not have the resources to refactor every log statement across all these teams/codebases.
We needed to support an approach that allowed us to provide an out-of-the box solution but also easy integration with anything already deployed.
We had a desire to focus on CNCF or at least open source solutions that are loosely coupled to lower the limits for adoption.
A blog post covering more in-depth examples to support this presentation can be found here.
- 1 participant
- 21 minutes
22 Oct 2021
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Fluent Bit: Past, Present, and Future - Eduardo Silva, Calyptia
Fluent Bit is a project that gained high popularity in the ecosystem, with high growing adoption of our users between individuals and companies. The project vision, status, and future roadmap are such important as our efforts as a project must be aligned to solve market needs. In this presentation, Eduardo Silva (project creator) will talk about the past, present, and future of the project.
Fluent Bit: Past, Present, and Future - Eduardo Silva, Calyptia
Fluent Bit is a project that gained high popularity in the ecosystem, with high growing adoption of our users between individuals and companies. The project vision, status, and future roadmap are such important as our efforts as a project must be aligned to solve market needs. In this presentation, Eduardo Silva (project creator) will talk about the past, present, and future of the project.
- 1 participant
- 14 minutes
