Cloud Native Computing Foundation FluentCon North America 2021, 30 Oct 2021

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: From FluentBit Operator to Fluent Operator: The Long Way to Come- Benjamin Huo, QingCloud Tech

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

From FluentBit Operator to Fluent Operator: The Long Way to Come- Benjamin Huo, QingCloud Technologies

FluentBit Operator was created by the KubeSphere community to solve several problems:

1. Collect K8s logs through a light-weighted agent like Fluent Bit.

2. Control Fluent Bit via Kubernetes API.

3. Collect logs and then send them to the final destination without having to go through Fluentd.

4. Enable dynamic config reloading for Fluent Bit to reload its config whenever the config changes without restarting the Fluent Bit Pod.

FluentBit Operator has reached its maturity level gradually after two and a half years of iterations, and KubeSphere and Fluent Community collaborated to transfer it to the Fluent organization.

Sometimes Fluentd is still needed as an optional sink to aggregate logs received from Fluent Bit, then filter and forward to more destinations. So FluentBit Operator maintainers are considering adding Fluentd CRDs and rename the project to Fluent Operator.

In this talk, FluentBit Operator maintainers will talk about the architecture and use cases of FluentBit Operator and the design of Fluent Operator.

A

Hello, everybody I'm benjamin, I'm happy to be invited to uh funcon north america 2021..

A

Today, I'm going to share a session about flynn with operator and our plan to involve it to a more powerful student operator.

A

I'm a senior architect of coop sphere, and I'm I'm also a member of flint and open function.

A

Here is the agenda.

A

We are going to talk about why we created frame bit operator in the first place and a brief introduction, use cases, analysis and, most importantly, our plan to involve current fluent bit operator to a fluent operator and we'll explain the roadmap a few demo and finally, some community stuff.

A

So let's begin why we create a phone with operator in the first place, you know for a logging agent, it's important to be lightweighted and have lower resource footprint and framework is just a perfect selection for the logging agent. So we choose to use fluent base and and then there are a need to control flame bait and its configuration through kubernetes api because we are developing on the native platform.

A

So someone might ask why you create another operator when there are already existing ones. So our opinion is a uh fluently. Filmmaker itself is uh able to send logs directly to the final destinations, a friendly uh shouldn't that be mandatory. I should be optional, so imagine if you have uh hundreds of nodes and all the nodes I will assign logs to the front d first and then finally, forward the lock to the final destination definitely might become the single point of failure right. So it's more efficient and more straightforward.

A

If you have, the flintbed agent can send loss directly to the final uh syncs. So that's the reason we uh created a framerate operator in the first place, and then we have uh some issues. uh Flimbit uh doesn't have a dynamic, config, reloading interface, which means whenever a phone bits config changes.

A

The flimbo film bait port has to be restarted. For the film bit to pick up the new configurations, it's not confident in some cases right. There are already some issues and discussions uh in the links uh below. So we have to show that first.

A

And here uh the uh how our friend-based operator looks like now, you can see that we have uh six crds and the first crd film bait is used to define uh the entire film beta demon side and the rest of the crds is used to uh define the flame bit configurations and you can, as you can see, uh the philippines part will send logs directly to the final six, like yes, kafka, unlucky, etc.

A

So, let's take a look at the details of the crds.

A

We have input, filter and output crds to define our corresponding flipped plugins configurations, and the film bit config crd is used to select which plugins to use by a label selector in the right one.

A

So as soon as the configuration uh the plugins are selected, uh it will generate the uh philippe config into a secret and the secret will be mounted into each frame report. uh Yeah and the front bit. uh Crd is used to define the entire philippine demon site, it's a position db and which configuration to use right and the operation uh the operator will watch for all the crts. Whenever each of these crd changes, it will already cancel and generate a new uh phone bid configurations.

A

So this is the crd and- and one important thing uh issue or very important issue we have to solve- is how to solve the dynamic config reloading problem of flambeat right. So the method we choose is uh uh in the flambe pod and we uh start uh a film with a watcher.

A

uh We create a phone with water component and the flambeed water will watch for the configuration files in report and whenever the configuration port is whenever the configuration file is created or changed, the phone based washer will restart the phone bit process and then the flambe process will be able to pick up the latest configurations.

A

So in this way we saw the dynamic config reloading problem. Yeah, so here below is the link. If you want to take a look at how it's implemented, you can take a look at the code, and here is the docker file, and you can see here. We compile the flynn bit watcher uh into a binary and use the official uh philippe image as a base, damage and copy. The latest configuration and the latest binary of front blade washer into this base image and replace the original film bit entry point with the flying bit washer.

A

So in this way, uh the flying fit port actually started by with a philippine water and the philippine water will start the full blade.

A

So this way we solve the dynamic configuring loading problem of flame bit right.

A

And a user can customize the log loud processing phrases by choosing different plugins for different phases, for example for the output phase. If you have es kafka and forward plugin, you can just enable the es plugin by setting the label enable to true and disable other plugins.

A

I this way you you, you can keep the configuration for kafka and forward without deleting them, and you can enable them again whenever you need it right. So it's a very um flexible and the same applies to uh input and filter plugin. So this is how the login processing can be customized.

A

Then we can take a look at some user cases. Group sphere itself is a good use case for a full bit operator.

A

It's a true flambe for the logging agent, the uh we have needs to control the log settings from the web console, so uh the api server has to uh uh how to call some cube, kubernetes api to manipulate the uh frame bit and the following configuration. So this is a one reason: we create a film bit operator.

A

We it have crds all right. It can be controlled by the api server easily. We will have a demo later.

A

So for uh kubern, uh two client uh kubernetes logs, you have to set up a phone bit crd first and the crg defines everything about the phone bit demon site, including position db. You can set a host pass for it, the request limit and the configuration toleration, affinity, etc, and then you have to define uh which plug-in to use in a full page, config crd right. So this is uh this is done by uh config the input filter and the output selector right.

A

So you can select different plugins for your needs and to con to collect the container logs. You can you have to define the input, the input crd, so I specify where you can find the container logs one buffer limit if there are huge log volumes and, most importantly, you have to define a filter. Crd right, the filter theory will add some kubernetes metadata to the uh to the log messages and another important thing is to actually the metadata contains some useless field.

A

So you can remove the user's field like pod id or container hash, to save your disk space right so and then you can send you can define the output crd such as yes, such as the kafka and fluently to send the logs to these things. Right- and this is how the final log message looks like you can see here, the metadata there are no port id or container hash right, they are all filtered out.

A

So this is how we create uh kubernetes container logs and we have knees. Usually we have needs to not only collect container logs, but also we have to collect the kubelet logs on the node, so how this can be done uh through a flambeat operator, and the first thing is you know: uh cubelet is actually a system d service, so we can use the firmware system d plugin to collect the uh specify, a filter to collect the cubelet logs right and, most importantly, you have to define a filter.

A

What I what this filter does is you know we have a requirement to curate both container logs and kubelet logs in one single login console, which means uh the log message has to be in the same format right. So we used uh this lua script to convert the uh public logs to the same format as the container logs, and this is how the lua script looks like it's. The most important part is here: it add a kubernetes metadata.

A

It will use the host name as the port name and use kubernetes as the container name and it is put in to the uh kuby system. Namespace and final there will be a timestamp will be added.

A

So in this way we can cure the cubelet logs together with container levels in one single uh console right, and the same is uh for the altered d uh logs. You know you have a user's requirements to monitoring what happened on one node, so the other d can will put the audit information of one node in the audit logs. So we can specify the audit the log path and similarly, I have to define a filter using the lua script, to convert the auditd log into the sim format. As the container lock like this.

A

Right yeah: this is the script similar to the previous one and we put the node name to the port name metadata and the container name is set to adhd and it also put the namespace to the qb system.

A

So this is how rdd and kubelet logs are collected.

A

Actually, we have a requirement to do log alerting. There are several ways to do log alerting once as looks like this. The logs is sent to kafka and you can use some kind of gone consumers to analyze uh the logs from kauka in real time and to filter out the logs. You are interested and sending it to some notification channel, but uh this method is kind of heavy right.

A

You have to use a kafka setup, a kafka cluster and set out a few consumers, so a more lightweight way to process the log from the agent side right. So we can. We can also use a filter in the filtering graph in the grub field. We can add the log parting log pattern. You are intrusive uh this. Whenever the log message matches python, it will be sent out to a web hook like this right. In this way, we can do like alerting on the agent side it's more efficient and less weighted right.

A

So this is the use cases, part and fundraisers, as we have uh already know, it's less weighted and it's high performance and zero dependency and no single point of fear right, but it also has some disadvantages.

A

As we know uh it has much less plugins than uh fluently right. It has less powerful filter and parsers than the front d. So in some cases the fluency is still uh needed, so we decided to add fluently to existing film film-based operator and rename it to a flint operator.

A

So currently, the phone bit operator looks like this. We have already discussed and we are planning to add three more uh cluster crds here.

A

Why are we add this cluster crd because uh firmware is actually a demonstration and it collecting class wide logs.

A

So it makes us to have a cluster service to class, uh to collect the clusterwise logs right and, most importantly, we will uh we plan to. I find this apart as a friendly will be act as an optional uh destination of a full embed output is optional. It's important: it's not mandatory, so the front video still can send logs from itself to directory to the final syncs and fullnbc is one of these things and friendly. It can do more log, aggregation and more powerful filtering and passing and and then send logs to the final six.

A

The flint uh front dcrds contains basically five crds and the fluency crd itself will define how the 3d department looks like and the friendly configure fully class config contains information about input and filter plugins and the output and the cluster output plugin defines where to send the logs.

A

So this is the uh with a friendly and frame bit. Crds are all added and we will rename the product to a fluent operator. So it's more powerful and flexible than current operator right.

A

So, and this is the roadmap almost almost three years ago, we created the front beta operator, and it had we created about eight releases uh till august this year, until until august, uh actually um more and more uh contributors and maintainers uh participating in this project uh from all over the world, uh including the u.s, uh australia, the europe and uh and of course, china.

A

So, uh after communicating with uh fluent community, we decided we agreed to transfer this product to the influent hog, so this is happened in august this year and the 0.9 release is the first release after we transfer the product to the fluent organization and the 0.12 is the latest release.

A

We are planning to release the uh we are planning to add differently, support and rename the product to fluent operator in q4 this year. So if you are interested in the development you can, I will share a link to the proposal.

A

You are welcome to participate in the development of fluent appraiser right, and here are a few demos here.

A

You know this is uh this: is the container management platform we can?

A

We can see the log collection configuration here. We have already the es and the kafka absolute plugin config, the yes is actually receiving logs and the kafka is closed.

A

You can change the here to be active if you want to send logs to kafka, and here, if you edited the yamo, you will see. This is a visible here. Right and below is the kafka configurations and for the uh yes, it's a.

A

Actually, it's a enabled right. Of course you can turn it off and you can add more uh things like I had the fluency.

A

So next I will next video. I uh show a demo how to curate the uh uh public logs. You know, as we have mentioned before, the kublas vlog is a post under the kuby system namespace and the the pod. We will use the field to store the node name. Actually, so, if you search master, you can find there are some blocks for the uh google ad right.

A

This is the logos for the google ad, but if you have, uh if you want to see the latest uh terminus log, you can find nothing because currently the uh uh currently the cubelet input has been disabled.

A

So we have to enable it first right here.

A

This add it to true and below you can find. This is the kubelet box right. You can save it you're taking effect.

A

And if you wait a few seconds, the cubelet logs will show up.

A

It will take a few seconds and you can see that the google ad logs are scoring scrolling down right. So this is a demo of how how do we collect? How do we enable the logs and the plugins? So this is done for the demo right and.

A

So if you are interested in participating in this project, you can find the github and slack information here, and here is the proposal for the fluent operator right and finally, I want to thank all the maintainers and contributors, especially uh maintainers from the digital ocean. We are very passionate and professional, so thanks uh other success guys and it's all my uh session, that's all from before my session, so any questions I will be here to answer any questions.

A