Cloud Native Computing Foundation Cloud Native + Open Source Virtual Summit China 2020, 11 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Deep Dive: Fluent Bit: High Performance Log Processor - Eduardo Silva, Arm Treasure Data

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Deep Dive: Fluent Bit: High Performance Log Processor - Eduardo Silva, Arm Treasure Data

Logging in distributed environments is very demanding, specifically when applications generate data at high rates. On this case processing data as fast as possible is one requirement, but also there is another one in high demand: low resources usage, keep your CPU and Memory usage as low as possible. The challenge is: how to optimize for fast data processing and optimizing for low CPU and Memory usage?, this is when Fluent Bit joins the game. Fluent Bit is a sub-project of Fluentd written in C language with a pluggable architecture and optimized for performance. It's becoming one of the preferred options for cloud native environments. In this session you will learn about it architecture, performance designs and best practices.

https://sched.co/cpB5

A

Hello, everyone, my name, is eduardo silva and welcome to the fluence presentation at cloud native, open source, virtual summit, china, 2020. on this presentation. We are going to talk about login data management and a bit of data aggregation in common database.

A

So fluid is a tool, was decided to fix and work around. All the problems and challenges with data and the whole point of logging and data is that we need to be able to extract value from them. But one of the main problems is that data is everywhere. So if we are in to concentrate all this information, we have so few challenges to to deal with. For example, data comes from the operating system system, services, user, space applications and, if you are in the embedded world, we have remote hardware, we have sensors and so on.

A

So how we can solve this problem as just with using one tool or one simple solution. As I said, data is one thing, but if we are not able to extract value from it, it's useless. We have many users just extracting data, but if we cannot perform the right analysis of rate extract value becomes hard when data is ingested or aggregated in one place.

A

It aims to provide easy ways to perform this analysis and, of course, extract this value. So we can gather the insight from it. So you realize that data is important, but if we cannot extracted value is a problem, but from the technical perspective we have a couple of challenges.

A

One of them is that data comes from different network protocols, maybe from tcp udp in different payloads from file system from system d generally in general, and this is complex and a mess, because data is everywhere in different formats and from different channel channels.

A

If you think about just a one, simple generator of log information like a hardware or a software, this generate a record that record gets into a central place, so we can do analysis, but if we have a couple of them, this kind of scalability gets more problems. So how do we inform my analysis fast enough? It doesn't matter if I'm collecting more data, more data from different places, so the login tool here is really relevant, because what one of the things that we want to avoid is latency to avoid that the things get delayed.

A

So data comes from different formats and when I say this, I'm saying that imagine the apache logs apache logs comes with a normal message that you and myself can understand right that mysql I can understand. But if you pass this information to a computer, the computer will just see an array of bytes and that's it. But we want to perform some kind of data processing over this data, so we can perform fastest queries or perform a fastest analysis.

A

So one of the key concepts here is to be able to convert this array of bytes to a structure where we have this notion of key values. For example, apache logs as a timestamp has a method, uri http protocol, and so on data comes from different sources. We need to also convert this data from a structure to a structured way, and sometimes we need to enrich this data with extra information.

A

For example, if your data comes from different servers likely, you want to reach your records with the hostname with the ip address or something that adds more in context to that information and at the end we want to send this data out to a central place for further analysis, like databases or cloud services, and this is fluent fluentmet- is a project that was created to solve all these data problems.

A

All these challenges, where we have different data channels, different formats and so on, fluent, is a proud sub project of the cncf under the umbrella of flue. Indeed, fluent was created around 2000 2015 and originally it was created for embedded linux. On that moment, we we envisioned that we have fluency, which was running on most of the major servers of the world, but we also we we are going to need something more lightweight that can run on constrained environments, but quickly.

A

People from the cloud space started using fluid because they needed something. Light without fluency is for some special cases and fluent bit evolved as a cloud space solution too, but it also can run on embedded environments and, as I said, it's the apache license to that whole fluent bit is fully written in c language. It ties for low memory and cpu footprint.

A

It's running barely can run in 600 kilobytes. But of course, if you are processing more data, you know you will need a bit more memory and it has its own plugable architecture. We support more than 60 plugins between input, filters and outputs, and it also has built in security for tls and network io.

A

When you think about login. You have to think about this. Like a pipeline, we have an input and an output, but in the middle we have different steps or faces for different things. We have an input where we say hey from this place, we're going to collect the data. We have a parser section where we can say hey this part we're going to process the data from this constructor format, to this format right a structure format, for example, instructor to json, or something like that.

A

But note that we don't use json internally, we just use a binary version of it. We have the filtering phase to enrich data or remove any content that we don't want from it. The buffering section for data persistency and, of course, routing to send the data out to any cloud database or service.

A

So right now we're going to jump into demo. So I'm going to switch here just one second, okay, so um right now I need terminal. So the demo will be about how to run a joker container, and this docker container will start a application that generates or simulate apache log messages. This apache dot messages doesn't have a real format, but our goal will be to take them and process them with fluent bit.

A

So first, I'm going to start fluent bit in my terminal and I'm going to I'm going to start with one configuration format file: sorry that I'm going to explain its content right away. This flue embed service is listening for messages on the tcp port, 24 224, and right now we are going to run and docker container, which is called expert, apache locks and basically, what it does is just to generate dummy messages to the standard output.

A

But, as just happened some seconds ago, we were running this with the fluency driver. Flew in this is our parent project and the fluently driver aims to send this data to a central place. It could be a fluency or a fluent bit instance. So, right now, all these container log messages will be routed back to the fluid that is running in the second pane on the on the left.

A

This is interesting, as you can see at the top. We have just an array of bytes, but then we can see a structure like json at bottom. So let's do some analysis on how this is working.

A

I'm going to show you the configuration that we're running. Basically, we are saying collect data from an input which is a forward input protocol filter. This data apply a parser. Do you remember the pipeline and these parts are saying: take this unstructured data apply the apache parsers on top of the lock key and reset reserve all the remaining data.

A

Finally put this data out to the standard output? Okay. So, but this is pretty easy, but let's also apply jq, so we can see how the records looks like in a better way. As you can see our records. If we format the jsons, you can see that we have the host the user, the metal, all the data that at the beginning, we didn't know about it.

A

But after applying a person, we were able to show all of these in a reliable manner, not just show, but also store this data in a different way that can be used later for data analysis.

A

Okay, so that's one use case right: we just took container logs we're shipping it out to flow embed and we are showing the data to the standard output, but we want to do something a more fancy. Yet what about? If we send the same data back to a database like elasticsearch, so I have prepared here a configuration file that is pretty much similar to the first one. But now, instead of sending the data to the standard output section, we are sending the data to the elasticsearch service that is run locally in my machine.

A

The es name means the elasticsearch plugin and we are going to use the logstash format.

A

So again we just start fluent and we just switch the configuration file and then we we just run again our container that is generating logs and, of course, we are not going to see anything on fluid bed output because we are redirecting all the logs to elasticsearch and here in the other tab, we are going to watch how the messages are being ingested to the elasticsearch service. If you can look carefully, we have one index which is called logstash and the number of documents count is increasing.

A

25 27., it's shipping one or two per second, because we are just we just have we just have that rate of data.

A

While we keep this running, we can also explore our data with a kibana. Kibana is a visual way to play with um all the data content that we have in elasticsearch. We are going to start exploring this data.

A

Okay. First of all, we need to create an index pattern.

A

Okay, I'm going to the next step, sometimes elastic on this keyboard, and this part is a bit tricky, but we will see if it led us to see all the content.

A

Okay, perfect and we will start looking at the theorem. We want to visualize this data. Oh sorry, here.

A

Okay, here we go so our data has been ingested in elastic and, as you can see, every record it has its own structure. We can take a look at the json format that is coming in.

A

So, if you remember carefully at the beginning, we just had a one array of bytes, but after a parsing we get a full structure, we got a timestamp and we got different information as an addition. That was like a context I'm talking about the container id and the container name. So this is one of the things that you can do with fluid. Of course, this is a demo. This is a very simple use case, but there are many more things that you can do.

A

So we are going to get back to our presentation.

A

And okay, so after this demo, one we're going to talk about a little bit about the login basics and this how it works internally. Basically, one application generates a record which is a message right, but this record is not just a message: it also comes with a time stamp timestamp because it needs to know when this message was generated, plus the message internally.

A

We do this, but we serialize all the data that is coming in in fluid bit with a format which is called message pack consider message: pack like a binary json, but on asteroids and of course all of these inside fluid fluency also does pretty much the same thing. Actually fluent data processing model like this is an exact copy of fluency.

A

Internally. When we have the input size, we can have many inputs, many filters, many outputs. We have the data that is coming in associated with tax, so with tax we can.

A

We can say this data is coming from this place and we want to create then a natural to say: please send this data out to a different destination. So in the left we can see that we have two sources of data with different tags: right, one for the apache logs or for syslog information.

A

Then we have an intermediary phase, which is like the buffering or storage and in storage is where we are storing our data base. Of course, by default is memory, but also we can put a secondary system in the file system.

A

So if anything happens like a power outage or a network problem, at least we can recover almost all the data that was at that moment being processed after that we have the router. So the router basically takes every input of data that has attack and using the matching rules defined by the output plugins. We can ship this data out to different places, elasticsearch influx dbe or any cloud service provider platform.

A

Right now we have more challenges right: it's pretty cool to take the data from one place and ship it to the other. But what about distributed environments like kubernetes, where every pot can be one or two or three containers, but we can have a many pots in one note, and we can have many nodes, but also nodes needs to be in correlation with the master api server, which provides context, information for every port, so fluent bit comes to the rescue. It basically is deployed as a demonstrate and this demon cell.

A

What it does is to read all the container log files, because all by the by default, this can be changed, but by default most of kubernetes and distribution store the data information into the file system, but also you can configure from a bit to use systemd or any kind of a protocol.

A

You can read the data from the pods and then fluidbit is able to retrieve all the media data from the api server, but finally to be able to ingest data back into the database. So the same demo that we did is pretty much the same thing. That will happen in a cloud environment and now we're going to talk about a different topic which is called stream processing.

A

Maybe it sounds familiar for you, but stream processing is just the ability to perform data processing, while it's still in motion, but you can say hey, we are talking about logging, and now we are talking about stream processing if you think carefully about logging what we did at the beginning, when we are converting data from constructor to a structured format, we are doing data processing, but also we went beyond that. I implemented string processing capabilities on top of flambe, so think about this. Every record or log message is an event.

A

The events already has a structure and we have two things: timestamp a message like this right: this is an example temperature and a timestamp. Those are keys and values, and now you start thinking about hey. This sounds like internal representation of data in a database could be, but also in stream, processing. There's a couple of challenges: all the data has different formats.

A

Even in such a structure that schema is not fixed right, it could be different. You can have records with um 10 keys on this with 8 and maybe not all the records will have the same name for the keys right, so how we can do stream processing or provide the ability to process this data in a more powerful way, which is not just filtering, but something that we can run queries on top of this data that is flowing through fluid.

A

So the stream processing goals are just fast data processing trying to do it without the notion of tables no database, no indexing or index free, and ideally we want to do it just in memory. So it's just ability to process your data. What your data is in motion, meaning it's still in memory flowing in the pipeline from one point to the other.

A

So how do we do this? This is like traditional, a way to do stream. Processing. You have the hardware, you have the events, the data gets into a stream processor and then you ship the results out to a database okay. So this is really good, but if you have a stream processor, you have need to deal with the events the records.

A

You need to expose a query language because you want to do key selection, filtering aggregations ever wrote in and all that things the traditional model is that if you think about the edge of the cloud, when I'm talking about the edge I'm talking about where the data is being generated, it could be anything kubernetes, node, an embedded device right.

A

The normal approach is to have on the edge something that kind of centralize data, send it to the stream processor to the cloud, and then we centralize the results in a database, okay, but what we are doing with floom bed, it's more fun than that. Actually we are doing logging on asteroids. We are doing stream processing, but on the edge in the same place. Where fluid is running, you can defer what they used to do in the cloud.

A

Maybe you can do 50 or 60 percent of that processing, but on the edge in a lightweight mode.

A

This is the original mechanism that we have right from the edge and send the data to the out to the end, to the cloud to do all the stream processing. But now what we are achieving in flowing bed is a street internal and built-in stream processor that can do analysis, data forecasting, create streams of data, and maybe for you this sounds familiar to kafka, okay, sequel and yes, it's pretty much pretty much similar and what are the goals and features with stream processor?

A

Well, actually, we can offload computation from the servers to the edge the were delayed. The data has been regenerated. You just can send the results that you care about to the cloud and not send all the data. Imagine that you have a 50 servers.

A

You don't want to send all the data just a fraction of them and with our query, language is a sql and I'm sure that you are familiar with sql and yes, there's no tables, there's no database, it's just one bit with our built-in sql processor integrated in our core and what you can do with the string processor.

A

We don't have enough time to make a demo right now, but actually the things that you can do that are really fun is like create a stream for data that is flowing, for example, from an input, a plugin tail or forward. You can say, create a new stream using a query, meaning like select all the keys that matches a certain criteria and create a new string of data and that new string becomes a new input of data inside fluid, and you can wrote that everywhere. So you can imagine that you can create other things.

A

You can create a special functions to set to try to detect anomalies, which is quite powerful, the same pipeline that we had now, but instead of normal buffering, we have the format, storage and after storage we hit the stream processor and the stream processor. Then we hit back the information as the input, because that data that you are generating is new data. It needs to go through the pipeline and can go to a different destination.

A

We support aggregation functions so we can create aggregation, windows and calculate aggregation and apply the common functions like max means, um and also we support the basic of machine learning functions. We have to support time series forecasting.

A

We did a webinar a few days ago, so we will share the links during this presentation where you can watch more about this, because this is really good, and I think that the future for stream processing and fluent beta is playing a major role on it.

A

Now an example how to create a stream. Imagine that you're running the input plugin, which is called cpu, which collects cpu metrics in fluent, you can say, hey, create a select the average cpu consumption from stream cpu during a window tumbling of 60 seconds. So during 60 seconds we are calculating. What's the average of the cpu that has been reported and after 60 seconds we just ship. One result right that is created with a tag called results.

A

So we don't you don't keep all the 60 samples out you just ship one per minute and flowing beat. Nowadays, I will say: that's getting a lot of traction and the team. We are really excited about this nowadays. In last week we hit it just more than 100 million deployments just on the half of this year. So most of our users, I would say, 90 percent are kubernetes users.

A

They use a fluent bed with fluency in conjunction or sometimes it's just fluent, and but the adoption is just crazy and also we are pretty happy to be working with many companies that are using fluent bed in production like arm aws, google cloud lock, dna, sumo logic, lift and so on, and I'm sure that many companies in china are using it too, and we welcome every company, every user of the community to join us and help to make fluent bet better.

A

And now we are going to open at the time for questions. So I'm going to stop the recording for now for this presentation thanks so much for attending this session, and then we are really happy that we can engage and receive feedback from you if you are a fluent bid user or if you want to jump into it. Thank you enjoy the conference.