Cloud Native Computing Foundation Open Observability Day NA 2022, 28 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lightning Talk: Achieving Unified Observability for Cloud and Edge with FluentBit - Benjamin Huo

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Lightning Talk: Achieving Unified Observability for Cloud and Edge with FluentBit - Benjamin Huo, QingCloud Technologies

With the rising of cloud native edge computing technologies, more and more organizations start to use Kubernetes to manage edge resources and workloads with the help of edge frameworks. Some of these edge frameworks like KubeEdge manage edge nodes as part of a cloud K8s cluster which brings challenges to users to manage edge nodes and edge applications. Observability is one of these challenges, for example: 1. Monitor and alert edge nodes and applications in the same way as cloud 2. Collect and search edge nodes and applications logs in the same way as cloud In this talk, Fluent Operator maintainers will discuss: 1. Use Fluent Operator to manage FluentBit for both cloud and edge environments. 2. Use FluentBit to collect edge nodes and applications logs and forward them to the cloud side. 3. Use FluentBit to collect edge nodes and applications metrics and remote write it to a Prometheus long-term storage in the cloud. 4. Manage logs and metrics in a central place for both edge and cloud.

A

Hello and welcome next I'm going to share how to achieve unified observability for cloud and Edge with Bloomberg and Benjamin I did the conspiracy, observability and Edge Computing team I'm? Also a member of kubaj TSC.

A

In this session, I'm going to give a brief introduction to a fluent operator and, most importantly, how to use it and learn better to achieve a unified cloud and Edge observability.

A

Actually, this project is an open source that has a fluent-based operator uh by a computer in January 2019..

A

We created this Tool uh to address the problem of plant-based configuration, Dynamic reloading without restarting the fluent build, and it is a donated to of the fluent community in August last year and in March this year we finally had plenty support to this project and rename the entire product to a friend operator and release the fluent operator 1.0.

A

This is how our front operator looks like now. We have changed all the fluid CRTs to Cluster wide and we also add namespace and class-wide CRTs for friendly and your user can use friend operator to manage and deploy a full-based demon site as well as following this vehicle site. If user wants to process the logs from a specific namespace, he can Define names based by a friendly crd and recently will allows the user to only process log for from this limbs list and forward forward it to the final destinations.

A

uh User can also use friend operator uh can also deploy a front-based demon, set only use a flambe to collect logs and do some simple processing and forward to the Final Destination.

A

A user can also deploy a friendly only receive logs from Network through the HTTP forward or syslog Plugin do some Advanced processing and forward to the final destinations and, of course, a user can use uh blend date and foreign.

A

D and fluently do some Advanced processing and forward to the final destinations.

A

uh Things uh release 1.0. We have made some big Improvement. For example, we have add the open search plugin for Frontier and frame rate. They have had the Lockheed plugin and recent. In recent few months, we have noticed that uh fluent data is starting to support Matrix plugins. So in release 1.5 we have added some Matrix related plugins, for example the node exposure Matrix, the promise to script Matrix the front page metrics and the promises remote right, plugin, together with open, Telemetry, plugin.

A

Another important Improvement in 1.5 is that we finally add the foreign custom plugin. While we add this, uh you may notice that film beta is having a is having a nearly 100 plugins now. So it's all it's almost uh impossible or unreasonable to modify the code and release a new version of fluent operator to meet the requirement of a new plugin.

A

So uh for, if user want to use a unsupported plugin, he can simply Define a custom plugin section in existing input, filter or output crds.

A

um So the configuration at the configuration of custom plugin and it's just the original uh blend, build plugin configurations. So this way um user can um user can use any uh of limited plugin without change without waiting for a fluent operator release. This is the kind of a wicked type of problems and the three strong tab. Plugins will be added continuously, as we do before.

A

uh We use a quash as our Edge Computing framework, so let's give a brief introduction to kubaj. Kubaji is a single step incubating project for Azure computing. Actually, there are several uh Edge Computing framework out there. For example, case relax is uh actually I will create a single entire cluster in the edge location, uh but actually I will set up several Edge nodes in the Azure location and then it will connect this Edge node to a browser in the cloud also a secure tunnel. So it's a bit different with the k3is.

A

um Just like a regular kubernetes node uh hash node is a have a similar component like the kublas called HD. Rta is a light weighted equivalent. It's a trim, the Russian couplet, most importantly, ID also exposed container matrices just like uh kubernet. It's a important to select Edge application matrices on this way.

A

Let's take a look: how to achieve a unified cloud and age availability with a balloon bed from this architecture. You can see that there is a outside cluster deployed and also some H node in the edge locations for the cloud side Matrix to plug to collect the cloud side metrics and we use produce agent to collect the metrics. From a note to exporter Kublai Kobe State, Matrix Etc.

A

We also deploy a fluent operator to manage and deploy the blend bit demonside to cloud and Edge node for the edge node. um As you know, there is a actually I have limited resources and it we shouldn't deploy money uh component to the original just to collect the observability data. So that's why we use a flame data as an agent. We have changed the promises agent to a fun bit and we have removed notice, water and useful and bait know the exporter Matrix plugin instead, and we also useful embed uh permissions script.

A

Matrix plugin to script matrixes from the address. I have style Kublai to get the container matrixes and, of course, there is a fluid demon site deployed to an edge node to collect the edge blocks for application and node uh you can see the benefit here is a.

A

We only need one uh component, that is a fluid for the ash node. You can use flambeat to collect both logs and the Matrix for Edge side uh for Edge, node and Azure applications.

A

It's a perfect fit for the limited resource, Edge node scenario: let's see how we deploy a will invited to the edge location you can see here. We Define a node Affinity to deploy a separate frame demo site to the edge node, and you can also see that we have monitor field volumes of the actual node to The Container, to The Container, to collect the node exposure. Matrices.

A

To collect the edge load uh container matrices, we Define a permissions Square Matrix plugin uh through a collect, the ACT node matrices. We deployed a node exporter, flame-based, Matrix, plugin and and finally, we created a profuse remote right, a plugin from bed, and this plugin will most write all the matrixes to a cluster site. Long-Term storage- you can you can see here. This is how we achieve unified I got an edge of the observability with a fluent bed.

A

um It's very exciting to see uh of Liberty is starting to support metrics and choosing, because you will have a single tool to claim to collect all the observability data and remember the operator. Fluid operator will support this new abilities whenever it's available, um you are welcome to uh participate in the uh blend operator project as well. So uh this is the mass sharing. Thank you for your time.