Calyptia Fluent Bit, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Fluent Bit - Swiss Army Tool of Observability Data Ingestion - Michael Marshall, Neiman Marcus

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Fluent Bit - Swiss Army Tool of Observability Data Ingestion - Michael Marshall, Neiman Marcus

In this session, Michael will show how Fluent Bit became his Swiss Army tool for observability data ingestion, how he was able to add dynamic log metrics extraction by writing a custom Golang plugin, and as an added bonus, help lower his organization’s AWS costs by selecting Fluent-Bit. Neiman Marcus needed an architecture of both open source log and metrics ingestion projects running in AWS. Michael built out an enterprise quality, fully scalable, performant, solution. All teams began lining up demanding Michael ingest their system and application logs, as soon as possible. On top of this, each team had their own formats and data schema challenges. Michael will share how he overcame those challenges using Fluent Bit. This session is complete with code examples, a brief demo, and lessons learned.

A

Hello and thank you for joining me for this footbed swiss army tool of observability adjustion.

A

My name is michael marshall, and I am a senior sre with newman marcus, I'm also an avid cloud architect and developer.

A

Before I begin this talk, I would like to do a shout out and thank you to my management team who sponsored the project alec nelson and centos mott hicker.

A

During the talk I will cover a little bit what it is why I'm so excited how I added log derived metrics functionality.

A

How I use it as my swiss army tool and then I'll? Let you see it in action, so I came across a challenge.

A

After building out my data observability platform, with grafana low-key logs with prometheus and cortex metrics using grafana ui for dashboarding and ad-hoc queries, I set out to find the best tool to use to ingest the data into the platform it needed to be low complexity and it needed to be inexpensive.

A

What is it fluent bit by definition, a log processor and a forwarder.

A

It's a pipeline made up of plug-ins for extreme agility flexibility, a few of those I've listed here to ingest to massage to extract the drop records, as you can see in the diagram below, and it's the best thing since sliced bread.

A

So why am I so excited about it?

A

It's fast, it's developed in c, it's developed to run in an embedded system, environment, it's open source. You can look at see how it runs and contribute to it. It's container friendly. It runs great inside of a container. I run it on fargate, it's tiny, it's very low resource usage.

A

Along with being fast and container, it's very cost effective. I can run multiple instances of fluid bit each doing its own thing without impacting any of the others at a in a very reasonable process.

A

It has wide cloud provider adoption. I first came across it when using aws and I came across fire lens fire lens is a side car implementation of fluid bit that runs in fargate for sending logs to directly to foot d or in this case for a bit bypassing the need to send it to club mesh logs.

A

I also discovered that google cloud has standardized on fluidvis, for its uh logging infrastructure fluent is extremely agile and flexible, which is what makes it so attractive. It has a very active community for its open source project.

A

It makes releases often and always welcomes additional contributions and is extremely extensible, as I will show later in my talk so checking the boxes.

A

It checks the box for speed, agility and cost, but it was missing a critical function that I needed. I needed to be able to derive metrics from my logs and send them to prometheus.

A

So what do you do now?

A

Well, if it doesn't exist, create it says henry voice, so I did. This is how I added log derived metrics to permit I mean to flip it. I use the golang plug going sdk and I built output, prometheus metrics, plugin.

A

This is the plugin architecture that I used. I wrote logs through fluent bit using the tagging. Then I route them to the output module the prometheus metrics output module.

A

I define the count, the metric type, the metric name, the job name and and then I push that information to prometheus push gateway prometheus then comes and scrapes the prometheus push gateway uh when it's configured to do so now I wanted to address a couple of questions.

A

Why go away fluent bit is written in c you can do native c plug-ins as well. I use scotland because prometheus push gateway and prometheus are both written in going and made the handoff easier.

A

Now anybody that's run prometheus knows that push gateway is frowned upon.

A

I decided to use it so that I didn't impact the tight speed at which flint bit is running and instead of holding the metrics waiting to be scraped, I chose to push them to push, gave way as a fire and forget and then allow me to come pick them up. This model works very well.

A

This is a sample configuration section for my plugin.

A

What I wanted to do here was create a metric, a counter metric to count the number of stylwatch log records processed by this instance of flipbit and index them, as the log group name and the source aws account.

A

So the first few here are standard output parameters, the name of the print of the plugin, the tag your matching on the log level, and then we have the job. The job is a prometheus job label that will show up. When you do queries.

A

The url shows where to push the metrics where the push gateway is. I have the metric type at this point counter the metric name following prometheus naming convention.

A

And then I have the metric constant labels and variable labels. I want to explain the difference between these constant labels are set static in the config file.

A

At the time you configure flip bit, the variable labels are extracted using regex from configuration above for further up in the pipeline. This is the true key to this plugin.

A

I can then create truly log derived metrics from the log lines passing through the pipeline.

A

The id allows me to have multiple copies of the prometheus metric plugin in use within a single fluidbit.com instance. They must be different.

A

Now I'm going to go over a few of my go-to plug-ins that I use quite a bit for my production installation.

A

An input plug-in is how you get data into flipbit.

A

I have a couple of external feeds coming in from upstream vendors. One of them is syslog based and the other is a stream of raw json. So I'm using the tcp input for the raw json and the syslog input for system.

A

My onstream data is being I'm using the tail input to tell logs on application servers.

A

There is a special input plug-in, which is also an output plug-in called forward. This defines the forwarding protocol in use between fluid d and flip it or flip it a little bit it's for passing records between instances.

A

I leverage this by using tls and pushing my on-premise logs to my cloud-based footbed.

A

I also use it for multi-region passing data between regions, there's a second use which I've leveraged and I'll be demonstrating later in this talk is the programmatic input used using the sdk that I've listed here?

A

Basically, it takes the logs that you want to push to flip a bit and packages them into the forward protocol and assigns a tag and pushes it in to flip it.

A

These are the parser plugins that I use quite often.

A

Json, so you can pass it a json string and it'll return. The set of cumulated pairs and regex regex uses ruby regex to extract content using named captures you'll notice that I'm using the extracted fields later in the demo. In my output, plugin.

A

The filter plugins that I use quite often is grep mainly for discarding records that I don't want ingested into my system and the modify which I commonly use to drop keys prior to registration.

A

There's an additional one for tag, manipulation called rewrite tag now. Tagging in fluid bit is: is that one of the key uses for for the system?

A

You can reroute records through the pipeline just by changing the tags, or you can split the record by using rules assign a new tag and when you re-emit the tag it will re-start at the beginning of the pipeline.

A

You can do this. This allows you to do many different things. uh Flexibility. Now these are my most used filter, plugins, the parser using regex and then there's lua lua influent bit was a game changer.

A

There are many built-in plug-ins in fluid bit, but when you face a situation where you just don't find support, write a load function and your goal it uh it runs very fast and it's well integrated into the product.

A

Here's a few tips and tricks. I like to say: if I can see it, I can troubleshoot it in this case, I'm creating a t for the pipeline.

A

You can add a little filter between pipeline sections, write a function in lua that will dump the current table with standard out. This will display all the tags and keys in play, as that goes through that t now. This is very helpful for just getting a good feel for what's going on.

A

uh The second tip is basically for when you're looking at examples of configurations that people posted and documentation you'll see fields in play, that you don't see a definition for and therefore they don't make sense.

A

Having been bit by this several times, I will tell you to check the parsers.com file, uh where you'll probably find the definition.

A

Let's see it in action during this demo, I'm going to show you the set of configurations for my internal programmatic, fluent forwarding in production. This is implemented as a serverless project.

A

I'm doing a lambda subscribing to a log group and pushing the lamina gets fired every time the law group receives a record and the lambda takes it puts it into an susq.

A

On the other end, I have a lambda routing, a flint router that takes the message, decrypts it and puts it into fluid forwarding protocol and pushes it to loki and prometheus as configured. I will show you what it looks like in graffana.

A

A

This is the foot bit comp file. Follow that I'm using for my flatwash logs.

A

Each of these sections is is basically a part of the pipeline.

A

In this case it starts with forward. The forward is receiving the logs from the lambda, which is calling that sdk that I showed you and receives them on port. Two, two four, two, two four notice: there's no tag matching here in forward inputs. You always tag the messages prior to pushing them to the forward.

A

Now I know that they're tagged with cw logs cloudwatch logs, so the first part of my pipeline is a parser and it I'm going to push the tag and I'm going to decrypt it with a regex called cw tag.

A

What I'm doing is when I send this data, I am adding the cloud. The law group, cloudwatch log group and the source account to the tag, and then I decrypt it and use it in fluid.

A

Bit, you can see those names there. Those are the extracted labels.

A

The next phase is, is the parser on the record, I'm going to search for nested jason. If I find this at json, I'm going to return it as a named field called message.json along with pre and post data. I'll show you that in a moment and then after that, I'm going to clean up the nesa json because it includes backslashes and uh non uh I'm sorry non-json compliant formatting. So I'm going to fix that using lua.

A

Here's my nested json parser, you see here, I'm capturing the nesa json into a field name message: json and the post and prefix data.

A

This is the lua I'm calling the function it passes in the record. The record looks for the next json key runs through cleans up the json formatting and reassigns it to the record code. 1 says change it. When it comes back and code 0 says ignore it.

A

You'll see here the commented out section that I mentioned for diagnostics.

A

That would be helpful to dump the pipeline that's flowing through at this time.

A

The next parser decodes the message: json that's cleaned up using a built-in decode field, that's built into flight, it I'm passing it a string, called message, json and telling it it's in json format and it returns the key value.

A

Pairs, the next one I'm using the filter, modify I'm dropping option and tag keys, and then I get to the output section.

A

One way, I'm sending it the low key, sending the raw log to low key.

A

I'm sending static labels as set here, and these are my dynamic labels which I've derived from using the rig axis from above the output. Here, I'm using the prometheus metric plugin.

A

Setting the job to cloudless logs, pushing it to my push gateway, defining a counter a metric. These are my static labels and again these are my dynamic log derived metrics.

A

I'm going to quickly show you what it looks like in grafana. This is a ad hoc query. Looking for electric logs there's only one log we pushed in at the moment.

A

This is a dashboard I put together real quick.

A

This is showing the rate of change.

A

And this is showing the actual counter. The counter is always increasing. The rate of change will increase and decrease, as the rate of quantity shows up over a period of time. A higher quantity can indicate a possible issue in your network, and then these are the wrong laws, and one last thing I wanted to point out was what it looks like in push gateway, so I'm doing the cloudwatch logs. I have one log record at this time and you'll see the counter and you'll see what it's indexed by this down.

A

Here is also a syslog job, that's running, and it has all the different combinations with status logs counter. You could update it, and these this is being scraped by.

A

A

As far as lessons learned, I say provision a new instance for each endpoint they're cheap.

A

Also, if you need a new feature, edit, there's plenty of sdks and examples I'll do that uh the tagging and tag manipulation is one of the keys to agility and warren lua I'll. Thank you later.

A

Thank you for joining my talk. I hope you enjoyed.