Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022, 1 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Build a Cloud Native Logging Pipeline on the Edge with Fluentbit Operator - Feynman Zhou, QingCloud

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Build a Cloud Native Logging Pipeline on the Edge with Fluentbit Operator - Feynman Zhou, QingCloud

FluentBit Operator was created by the KubeSphere community to solve several problems: 1. Collect K8s logs through a light-weighted agent like Fluent Bit 2. Control Fluent Bit via Kubernetes API 3. Collect logs and then send them to the final destination without having to go through Fluentd 4. Enable dynamic config reloading for Fluent Bit to reload its config whenever the config changes without restarting the Fluent Bit Pod. FluentBit Operator has reached its maturity level gradually after two and a half years of iterations, now it has became the subproject of Fluent community. In this talk, FluentBit Operator maintainers will talk about the architecture and design of Fluent Operator, and demonstrate how to use FluentBit Operator on K3s to process logs for the edge and IoT scenarios.

A

Hello, everyone: this is feyman from beijing, china, it's so glad to be here. It's my second time to attend kubecon eu and I was looking forward to attending this meeting in prison. Since we have a couple of friends and partners in spain, but unfortunately, due to the pandemic, we can only give this speech at the front of screen, so I hope everything could be recovered soon and we can meet everyone and have a cup of coffee in person.

A

So before we get started with this presentation, let me briefly introduce myself so my name is feyman joe I'm from cube sphere team, I'm a senior community manager and qing cloud cube sphere, and I'm also cncf, ambassador cdf, ambassador and fluent member.

A

My skills include, but not limited to kubernetes, linux, flambeat, frontie, devops and servlets, and I'm really enjoying technical writing advocacy and outreach and host events. Alright. In this talk, I will demonstrate how to build a cloud native login pipeline on the edge with fluent operator.

A

So in this presentation I will walk you through the challenges of logging in kubernetes, especially in the enterprise environment, and next I will introduce two popular login solutions you might have ever used or heard about it, the fluent bait and friendly, and next I will introduce and demonstrate how fluent operator empowers fluent bit and fluently, and then I will give a little bit deep dive into fluent operator and talk about this architecture and workflow.

A

And finally, I will give a live demo and talk about this use case in cubesphere all right. When it comes to the challenges of logging kubernetes, we always receive some demands and complaints from different teams for security and compliance reasons.

A

For example, our developers said that hey, we have a huge amount of logs produced every day. They come from different data sources and data formats. Our administrators said that hey, you have to make sure to troubleshoot your logs in a lightweight and secure solution.

A

You also need to keep everything traceable and our security teams requested that hey. Could you ship the logs and data to multiple destinations and outputs to audit and visualize them?

A

Oh man, all of these things are in accounts right, so in the real enterprise environment we actually have some logs come from different places, such as parameter servers or virtual machines. They can also come from embedded devices edge, container port, tcp or udp, and all of those data are in different data formats, such as json, logs apache, logs ngx, logs or container locks, and in some typical cases you might want to shift them to different destinations, such as elasticsearch or open search, locky, splunk, mongodb or s3.

A

Well, considering data security and reliability when processing logs in a multi-tenant environment, you have to isolate the log data and make it only visible to a specific user or in a specific namespace. Okay. How do you guys debug your kubernetes workloads in your daily job?

A

I think the native method, such as cut commands, is the most popular way for you to retrieve the logs from a specific container.

A

Apart from that, if you are running your applications and infrastructure on the public cloud, for example azure on aws, you might also adopt the logging solutions that are powered by the cloud providers such as stackdriver cloudwatch, and something like that.

A

So you, you might also find that there are some sv solutions, such as splunk, sim logic, data log, etc. So far, we also see a lot of popular open source solutions such as elk loki from bit and front d.

A

But in this talk we will only focus on the open source, login solutions like from base and front d.

A

You know handling data collection as scale is complex and collecting and aggregating diverse data requires a specialized tool that can deal with the scenarios like different sources of information, different data formats, data, reliability, security, flexible routing and multiple destinations.

A

That is why from base comes in when we take a look back at the history of floom bait, you will find from bit this project was started in 2015..

A

Now it has been a sincere sub-project under the umbrella front d. Ecosystem from b was written in c, which is a lightweight and zero dependencies project from bit is also plugable. It has around 70 plugins available. Last but not least, floom bit is quite lightweight, since it it has low cpu and memory usage.

A

Okay from bit also has monitoring and string processing capabilities that are not listed in this slide so far from bit had reached 1 billion downloads and has been adopted by thousands of organizations such as aws, digitalocean, microsoft, cube sphere and so on.

A

As you can see from this graph, this is a data pipeline that represents a flow of data that goes through the inputs, filters and outputs. The input plugin, which is used to get the information from different data sources password, which is used to convert from unstructured data to structured data filter, is used to match, exclude or enrich logs with some specific metadata and output is used to define the destinations for the data, for example, remote services, local file systems, loki kafka or something like that.

A

Okay fluently is a data collector which allows you to unify the data, collection and consumption for better use and understanding of data.

A

Okay, you will find that fluently is much mature than from bait, which was started in 2011, and it has also been a sincere graduated project. Fluently was written in c and ruby. It is also pluggable and has around 1000 plugins available. So to summarize, flundi allows you to build your own, unified, login layer, and this layer allows developers and data analysis to utilize many types of logs as they are generated.

A

The most important thing is: it mitigates the risk of bad data slowing down and misinforming your organization. Okay, at this point, let's take a look at the comparison of bloomberg and fluently.

A

This table describes a comparison in different areas of the project, so you could find that both fluently and from bit can work as aggregators or forwarders. They both can complement each other or use them as a standalone solution.

A

That is why fluent operator comes in and supports. Managing both front bit and front d so from base was born to facilitate the management of front bait and fluentd. It allows you to manage the lifecycle from bait and fluently before we deep dive into it. Let's look back at its history of fluent operator. This project was open sourced as front bait operator by cubes 13 in january 2019.

A

After eight versions iteration, it has been donated to the upstream fluent community in august, 2021.

A

So after it has integrated the fluentd into its operator, it has been renamed to front operator. In march, 2022 in april, front operator has reached the 1.0 which marks the maturity of this project all right. This is the initial reason that we finded this project from bait operator, as we have seen that the front base cannot reload configured gracefully and it does not support dynamic configuration.

A

It requires users to restart its film bit, part and reload it manually, so it is not intelligent, especially in the production environment. That is why front beta operator comes in okay. Let me give a general introduction to fluent operator, as we mentioned earlier, the front operator which is used to deploy and destroy flambet demo set or run the status set automatically.

A

Second, it has custom configuration which allows you to select the plugins like input, filter output by labels. So, as we mentioned earlier, the dynamic reloading is also the most important feature that has supported in the front operator. It has supported, update configuration without rebooting from base and front deposit multi-tenant log. Isolation has also been considered in front operator.

A

You know fluently supports multi-tenant log isolation through label router plug-in last but not least, front-operator provides packable deployed components either from bit or front d can be deployed separately, as you can see that, although both floam bit and fluenty are able to collect process and then word logs to find out that destinations, you know, although both from bit and friendly, are able to collect process and then forward log to different destinations.

A

They have their own strengths in some different aspects.

A

So from bit plays the draw as a logging agent on each note, since it is super lightweight and efficient wavelength is more powerful to perform advanced processing logs capability because of its rich plug-in system, as we mentioned before, float operator was used to manage the front bed as its inception.

A

So if you only enable floom bait, then the workflow will be quite simple.

A

As you can see from this diagram, the front bit component defines the from bit diamond set, and this configuration meanwhile fluent operator provides a customized front bit. Docker image from base config, selects the input, filter and output plugins and generates the final configuration into a secret.

A

So how does front operator manage from bit and this crd to make it work better with kubernetes?

A

As you can see from this diagram, each city, such as class input, class filter and class output, represents a from base configuration section which are selected by cluster from base config, while label selectors front operator watches those objects, constructs the final configuration and finally creates a secret to store the configuration which will be mounted into the front bit demo set.

A

So the entire workflow is showing, as this graph. So at this point to enable flowbase to pick up and use the latest configuration whenever the floam based configuration changes. A wrapper called front-based watcher is added to restart the flame bait process as soon as frame bit configuration chains are detected in this way, the frame bit part is not required to restart it to reload the new configuration the frame based configuration is reloaded in this way because there is no reloading interface in front bit itself.

A

So you can learn more details from these links as below. Alright, as we introduced before fluently, is much powerful to perform advanced data processing because of its rich plugins.

A

So we added front d, support in front operator and renamed it now. You can receive logs through networks like http or skslog, and then process them and send those logs to the final destinations, such as elasticsearch kafka and s3.

A

Rent operator provides three kinds of mode that you can use as you want. There are floom bits only from bit plus front d and front d. Only so front operator includes crds and controllers for both frame based and front d, which allows you to configure your login pipeline in these three mods, as you want from bit only mode. That means, if you just need to collect logs and send those logs to the final destinations.

A

All you need is just from bait. So let's take a look at the front base only as you can see from this graph different bit crd cluster fluid bit config selects class network plugins and generates the final configuration into a secret.

A

Then the other plugins, like class input, class filter class output and class parser are selected by class from bit config, while label selectors all right. Let's take a look at an example, use case of collecting kubernetes application logs and send the logs output to kafka.

A

So you could define the plugins you want, while the label and selectors, for example, it has defined the filter plugins like kubernetes nest and modify plugin here next, let's take a look at the fluent dip only mode.

A

If you only need to receive logs through network like http or syslog and then process and send those logs to the final destinations, you just need to enable fluentd.

A

Let's take a look at how those fluently crds work fluently is used to define the fluency state facade and its configuration.

A

A customized fluent image is required to work with fluently operator for dynamic configuration, reloading fluently config, which is used to select class level or namespace level plugins such as input, filter and output, and generates the final configuration into a secret.

A

Similarly, the class fluently config, which is used to select class level plugins and generate the final configuration to secret the other components such as filter and output, are similar.

A

They are used to define the namespace level and class level configuration respectively again, let's take a look at an example, use case of using fluentd to receive logs from http and send the output to standard out.

A

You could define your plugins via labels and selectors, which is similar to the previous sample.

A

Apart from the flame bait or friendly, only mode, there's also a strong combination of fluently and flambeat in front operator. It has more flexibility since you could leverage their different strengths respectively.

A

If you also need to perform some advanced data processing to the logs or route them to more destinations, then you just need to enable flindy and choose the front bait. Plus one d mode with its rich plugins fluently, plays the role of a log aggregation layer and it is able to perform more advanced log processing.

A

You can fold logs from front bait qfnd with ease using fluent operator. Next, let's take a look at a real case. Study from our team.

A

You know: cubesphere is an open source container platform built on kubernetes cubesphere has a built-in login console. It allows users to search the logs and configure the log collectors such as kafka fluently or elasticsearch cubesphere adobe's elasticsearch serves as the backend logging service with firmbait. As a log collector, it runs from bit demo set on each node to collect the container logs and application logs.

A

In this way, different tenants could search the logs in a unified, login console and you are able to configure the logs only visible to the specific tenants or in a space in order to help you guys get started with fluent operator. We have prepared a workshop with step by step labs.

A

This workshop has involved a lot of typical and interesting use cases.

A

For example, it starts from installing front operator and helps you to deploy from base and front d next. It involves three kinds of modes that you can leverage from the operator to send the logs to different destinations.

A

For instance, we will use front bit only mode in this demo to collect kubernetes application logs and send them to kafka or elixir search. Typically, you can also enable from bit plus 1d mode. It has also some typical use cases in this section, especially in the multi-tenant scenario.

A

Finally, you can also leverage fluently only mode to use front d to receive locks from http and output to standard out feel free to try it in your local machine or your edge devices.

A

In this demo, we will collect kubernetes logs and forward it to elasticsearch by using fluent operator with from bit only mode. As we mentioned earlier. We have prepared a step-by-step workshop and walk you through how to deploy front operator and play around it. You can check out this lab from github and try it yourself.

A

All of the demos sample code and documentations are available at github.

A

Alright, let's get started with this lab. I have already prepared a k3s cluster and a minicube cluster before this lab, and in order to make this lab much efficient and convenient for you to set up in your local machine, I finally choose to use mini cluster for this demo.

A

Any other kubernetes distributions or native kubernetes are also supported in this lab. You can clone the demo repository to your local machine and get all of those share scripts. For the first step, we will deploy a kafka cluster and an elasticsearch cluster, so you can choose to forward those logs to kafka or elasticsearch, as you want.

A

After a couple of seconds, we could verify if all of those kafka cluster resources are running.

A

After all of those kafka, cluster resources are ready. We could go ahead and deploy elasticsearch cluster with its hem chart.

A

It might take a couple of minutes to set up the elasticsearch cluster, so we can take a look at the manifest of fluent operator and to see what has been defined in each yaml file since we choose to collect kubernetes logs and forward to elasticsearch using front operator with front bit only mode.

A

So so we dig into this folder as we can see from the architecture diagram of front operator. All of those front-based crds, such as from bit class front bit, config class input, class filter and class output have been defined in those yaml files.

A

The destinations such as elasticsearch has been defined in the output crd. If you want to define the other destinations, such as kafka or loki or s3, you can define it here.

A

Let's go back to the command line and to see if elasticsearch is running or not all right. Everything looks okay,.

A

The livestick search cluster has been deployed successfully and now it should be ready to store the logs okay. At this point, we could find that the two designations, elasticsearch and kafka are ready to use.

A

So for the next step we are going to deploy the lock, forwarder and processor. You know from bait to cloud kubernetes locks and ship them to these two designations, so we could use front operator deployment script to setup it and wait for seconds after we see the front operator part is running. That means we are ready to use front operator to deploy front bit or front d to process logs all right.

A

Let's enter the manifest folder and choose from bit mode only as we mentioned earlier, all of those crd yammer files have been defined in this folder.

A

So next you can use kubcar to apply to create all of those resources.

A

Here we can here we can point the targeted folder to this place.

A

Apply it and then wait for seconds you will find all of crd files. You will find all of those crd resources have been created. So let's verify then.

A

First, let's verify if the front-based demo set has been set up and running.

A

The film-based demo set looks fine, so go ahead. Let's check the strategy component of flume bait.

A

As you can see from this fluent operator diagram, you might need to verify the status of those crds.

A

Let's first check the status of fluent operator itself and then we can get the status of cluster from base config.

A

Next go ahead: we can check the status of class input next, let's go ahead and check this class input status.

A

Copy and paste it and change it to class filter, and we can see that the kubernetes plugin has been defined in the cluster filter part as we have defined the elasticsearch as a destination in this login pipeline within the output plugin. So the result is looks like as the same as our expectation.

A

All right at this point, we could scroll down to the guideline of how to check the status and configuration of the elasticsearch and to see if the logs and index has been created in the elasticsearch.

A

At the same time, we could also find the front base part is running due to we are using front base only mode here.

A

So next, let's verify the front base configuration that generated by the fluent operator, so we can copy this command line and place it in our terminal yeah. We could find the plugins that defined in this frame base configuration.

A

It includes input, filter and output, as we have already defined the elasticsearch as a designation in the output, you can also change it to kafka or odd designations, as you want all right. At this point, we could check the logs from the elasticsearch, so we can copy this command line and paste it in our terminal.

A

Wait for a second yeah after querying the kubernetes namespace market in the elasticsearch, we could retrieve the key value types of logs and data from the elasticsearch.

A

That means the logs have been collected and shipped to the final destination. You know the last search here.

A

All right at this point, we can also double check the result from the elasticsearch index.

A

Normally it will create a new index for the logs that retrieved from from bait.

A

Last but not least, as we have already deployed kafka as an optional designation at the beginning of this demo, so you can change it to kafka in the output crd component.