Calyptia Fluent Bit, 30 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Getting More Out of Lua in Fluent Bit- Pranav Marla, Fidelity Investments

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Getting More Out of Lua in Fluent Bit- Pranav Marla, Fidelity Investments

Fluent Bit has many useful plugins and features but by far its most powerful feature is its support for custom scripts written in a programming language called Lua. Unfortunately, it can also often be one of Fluent Bit's most underutilized features, with most users limiting themselves to minor variations of the sample scripts provided in Fluent Bit's documentation. This is because Lua is not as well-known as other programming languages, and its unfamiliarity can make it seem intimidating to people and prevent them from taking full advantage of its capabilities. This talk aims to help users become more comfortable using Lua outside the bounds of those sample scripts by walking through some novel scenarios, as well as sharing some tips and tricks, to help users take advantage of Lua to unlock Fluent Bit's true potential.

A

Hi there thank you for taking the time to attend this presentation. My name is pranav marla and today I'm going to be talking to you about how you can get more out of lua in fluent fit. Specifically, this is what I'm going to cover today.

A

After briefly, introducing myself I'm going to talk about the motivation behind this talk go over the basics of using lua influenbit walk through some example. Programs provide some general tips and finally wrap up with a list of helpful resources, as well as my contact information to start a few words about myself.

A

I am a senior software engineer at fidelity investments where I build solutions that allow other teams a fidelity to monitor their applications running in the cloud over the course of my work, I've gained experience with a variety of open source tools, including kubernetes, helm, fluentd and bit, most importantly, the question on everyone's minds. No, sadly, those are not my cats.

A

Next up. Let's talk about the motivation behind this presentation. As you all know, fluent bit has many filters that allow you to extend its functionality of those filters. The most powerful one is the lua filter, the lower filter lets you call a custom script written in a programming language called lua on each matching record. This means that, unlike the other filters which all have very specific predefined functionality, the lua filters functionality is essentially limitless since just calls whatever program. You write in theory, you're constrained only by your imagination.

A

In practice, though, in my experience, this incredibly powerful feature is criminally underutilized. If people use it at all, they tend to limit themselves to making minor tweaks to the existing sample programs, provided in the fluent bit documentation to get an idea of why this is. Let's look at the most recent iobi index and redmonk rankings.

A

They have two things in common one: they are both measures of programming, language popularity and two lua doesn't appear in the top 10 or even the top 20 of either of them. In fact, if we want to find lua in the tiobe index, we need to go down the list all the way to number 39 below ada, lisp and cobalt, and I believe this is the crux of the issue. The average user is unfamiliar with the lua programming language and because of that, finds it too intimidating to stray outside the guardrails of the sample scripts.

A

Now, over the course of my work with lua, I had to do just that and I ended up spending considerable time and effort drawing through manuals and just going through a lot of trial and error. So my hope is that, by sharing what I've learned, I can spare you that effort and make your lua journey smoother.

A

Let's start by going over the basics, the lua filter in fluent bit takes two main parameters: the name of the file containing your lua program, as well as the name of the specific function that you want fluent bit to call on each record, matched by the filter. For those of you who don't know what a record is, every input log is represented in fluid bit as a structured collection of key value pairs, which we call a record.

A

The lure function takes three arguments which are automatically supplied by fluid bit every time. It calls a function on a matching record. The first argument is a filmmaker tag associated with the record. The second is the fluent bit timestamp associated with the record formatted as an epoch, timestamp with nanosecond resolution, and the third and final argument is the record itself formatted as a lower table. For those of you who don't know what a lower table is. It's an associative array which is essentially also a structured collection of key value pairs.

A

So, for example, let's say we have the following input: log represented in fluent bit as the following record, associated with the following record, timestamp and the following tag. If this record matches our lua filter, then fluent bit automatically calls the specified lua function on that record and supplies the associated tag, the equivalent epoch, timestamp and the equivalent lua table as arguments.

A

This function also has a specific format when it comes to the return values in particular, it always returns the following three values: starting from the right. We have a lure table representing a record, an epoch, timestamp and finally, a return code.

A

This return code is an integer which tells fluentbit what to do with a record that triggered this function call and which effectively determines whether or not the other two return values play any role at all. Specifically if the return code is -1, fluidbit drops the entire record. If it's zero fluent bit does nothing if it's one fluent bit replaces both the record timestamp and the record itself with the second and third return values from the function. And finally, if it's two fluent replaces only the record itself with the third return value from the function.

A

So now that we're all clear on the basics, let's move on to some actual programs. Let's start with one of the canonical examples provided to us in fluent bits, documentation replacing the record timestamp. For example, let's say we have the following record with the following record: timestamp, as you can see on the right. The record also contains another timestamp within its mytime field, and we would like that timestamp to replace the record timestamp on the left.

A

We can accomplish this fairly easily with a lua program. First, we get the value stored in the my time field and save it to a variable which I'll call new timestamp. Next, since you want fluentbit to replace the record timestamp with new timestamp, we return 1 as our return code. We return new timestamp as our replacement timestamp and finally, even though a return code of one tells fluentbit to replace both the timestamp and the record since in this case, we don't actually want to change the record.

A

We just return the same original record as our replacement record all right. So now this function does what we want, but let's take a closer look at it and see if we can improve it. First of all, I don't know about you, but I can never remember which return code means what and I'm definitely not gonna. Remember six months from now, which means that maintaining and debugging this code will be that much harder.

A

So my first tip is to replace this hard coded integer with a well named variable in this case, I'm going with replace timestamp and record, and this applies to all programming languages. Really, your life will become a lot easier once you replace all the mysterious magic numbers in your code with well-named variables.

A

Now looking at this code, you might notice that, since this variable is defined outside this function, it's a global variable. But what you might not know is that, in fact, new timestamp is also a global variable. In other words, even though I've defined it inside this function, its scope is not limited to this function. In fact, in lua, unlike in other languages, by default, all variables are global variables, and this brings us to my next tip, wherever possible, use local variables instead of global variables.

A

This is because global variables can make your code harder to reason about, and more error prone and in lua, it's slower to access a global variable than it is to access a local one. So how do we do this in lua? The way you declare a local variable is by adding the word local in front of it, which takes care of new timestamp.

A

As for the variable above it when you want to reference an existing global variable inside your function, a common pattern in lua is to declare a local variable with the same name so that it shadows the global variable. So now, when we reference replace, timestamp and record inside the function, we're actually referencing the local variable, and this is good, because not only is it faster to access, but even if another function accidentally changes the value of the global variable, we now have a copy of its original value preserved in the local variable.

A

Now, let's consider a more advanced example, let's say you want to generate a new timestamp every time, fluenbid processes, a log and then inject that timestamp into the record as a new field. One reason you might want to do this is if your fluence collector is just one part of a distributed log pipeline, in which case debugging. Any delays in that pipeline can be tricky when the only information contained within your log is when it was first created.

A

However, if your log contains another timestamp letting you know when it was subsequently processed by fluent bit, then that can help you narrow down where the delay is coming from.

A

At first glance, this seems fairly simple to accomplish with lua for every record. We create a new field, let's call it collector timestamp and we set it equal to the current time. So the question then becomes: how do we get the current time in lua?

A

Well, there are two functions that can help us here: os.time and os star date. Os.Time produces an epoch timestamp, whereas overstar date produces a human, readable timestamp. Unfortunately, as you can see, they both share the same problem. The timestamp has only second resolution, and that is definitely not precise enough for our needs. Ideally, what we really want is a nanosecond resolution timestamp, but lua is not currently capable of producing that. So, where do we get it from well?

A

The solution I eventually came up with was to force fluent bit to generate it for us to understand how this works. Let's take a step back and see where fluent bit gets the record timestamp from it, obtains it in two different ways, depending on the structure of the incoming log.

A

The first method is when you have a structured log with a well-defined time field like this, when fluid parses this log, it produces the following record, with the following record: timestamp note that the record timestamp is the same as the log timestamp, in other words, for structured logs fluentbit extracts the pre-existing log timestamp from the log and uses that as a record timestamp, which here represents the time when the log was created before it reached flow and bit.

A

The second method is when you have an unstructured log without a well-defined time field like this, when fluent bit processes this log, it produces the following record, with the following record: timestamp: note that this record timestamp doesn't match anything in the log, in other words, for unstructured logs since fluent cannot find any pre-existing log timestamp to extract it generates its own record. Timestamp instead, which here represents the time when the log was processed by fluent bit so for unstructured logs fluentbid, already does essentially what we want.

A

The problem arises when we have structured logs since there, the record timestamp represents the time when the log was created, not when it was processed. So what do we do?

A

Well, let's take another look at our structured log setup notice that at ingestion time we configure fluentbit to parse the log with an appropriate parser, in this case a json parser, since it's a json log, so all we need to do is get rid of that parser and now, even though we know it's a structured log fluent bit thinks it's an unstructured log, which means that it's going to generate a new record timestamp.

A

So at this point, we've been sure that fluentbit is generating a new timestamp for all our records and is setting that new timestamp as the record timestamp. And if you recall, when fluenbid calls our allure function on each record, it automatically passes that record timestamp in as the timestamp argument.

A

This means that if we return to our initial lua function, all we have to do is set collector timestamp to the timestamp argument. We then return the appropriate return code to let fluenbid know that we made it change the record and return the modified record as our third return value. In this case, the second return value doesn't matter, so we can just stick with the default.

A

Now, let's see this in action, we have the following record, representing our unparsed structured log, with the following record: timestamp generated by fluidbit fluidbit then calls our lua function on this record and passes in the tag, the record timestamp and the record itself.

A

And finally, our alua function returns a modified record with the new collector timestamp field, containing the same value as the record timestamp. So we're done right. Well, not quite if you look closely, although the nanosecond resolution record timestamp has nine decimal places, the collector timestamp only has six, and unfortunately this represents a fundamental limitation.

A

When that floating point record timestamp is converted from c to lua and then back to c. There is some unavoidable loss of precision which means that the original nine decimal places of the record timestamp are not preserved in the collector timestamp.

A

So now we have a microsecond resolution, collector timestamp, which is better than second resolution, but it's still not quite what we want. So what do we do now? Well, in response to this issue, the focus of fluid bit came out with a new parameter for the lua filter called time as table when this setting is enabled flunebit no longer passes the record timestamp to the lua function as a floating point number.

A

Instead, it passes it as a lower table with two keys sec containing the integer part, in other words everything to the left of the decimal point and nsec containing the fractional part, in other words, everything to the right of the decimal point.

A

Although this takes a little more work for us to parse and manipulate, it allows us to preserve the original nanosecond resolution of the record timestamp. Let's return to our lure function and see how it works. First, we extract the integer part and the fractional part from the timestamp argument.

A

Using the string.format function, we then put the two pieces together to reassemble the record timestamp as a string, and finally, we inject that string into the record as collector timestamp. If we return to our example, we can see that our modified function now returns a collector timestamp containing all nine digits of the record timestamp.

A

This looks good, but there is a subtle bug in this code to demonstrate. Let's try this again with a different record timestamp just like before fluenbit calls a lower function passes in the record timestamp as a table, and the lua function then injects it into the record. As collector timestamp, however, notice that this collector timestamp does not match the record timestamp specifically, it is missing the leading zero in the fractional portion.

A

This is because, up above when the timestamp is passed in as a luau table, the fractional portion is passed in as an integer, which means that any leading zeros get dropped. So, let's go back to our function and fix this bug, instead of including the fractional part in the timestamp string right after we extract it. Let's first modify the fractional part such that, if it contains less than nine digits, we pad it with leading zeros.

A

We then include that padded fractional part in the time stamp string, and if we go back to our example, we can see that the collector timestamp now includes the leading zero and is once again a perfect match for the record timestamp. So at this point we have successfully injected the record timestamp into the record. As a field named collector timestamp, however, collector timestamp is still formatted as an epoch, timestamp and especially if you want to use it for debugging, it would be better if it was formatted as a more intuitive, readable timestamp.

A

So, let's return to our function instead of including the integer part in the timestamp string right after we extract it. Let's first use the os.date function to convert the integer part from an epoch format to a human, readable format.

A

We then include that human readable integer part in the timestamp string, and now, if we go back to our example, we can see that the collector timestamp has been converted from an epoch timestamp to a human, readable timestamp. Let's zoom in and take a closer look. The collector timestamp says september 5th 1pm, but note that there is no indication as to what timezone it's set in.

A

In fact, this turns out to be the local timezone, because that's what the os.date function defaults to now in general, I strongly recommend that whenever you're dealing with timestamps, you never set them in the local time zone. It makes interacting with systems located in other time zones very confusing, and it can even cause confusion within your own system if your local time zone has a concept of daylight saving time. Instead, I strongly recommend keeping all your timestamps in utc, which is essentially a neutral time standard which all time zones are based on.

A

So returning to our lua function, we can force os.date to give us a timestamp in udc, instead of in the local time zone, by adding an exclamation mark in front of the format string argument.

A

In addition, the convention is to add the letter z at the end of the timestamp to let people know that it's in utc returning to our collector time stamp. In my case, my local time zone is four hours behind utc, which means that instead of the local time of september 5th 1 pm our corrected, lua function will now return the utc time of september 5th 5 pm.

A

So at this point our log finally contains a human, readable, collector timestamp that clearly tells us when the log was processed by. However, we still have two issues left one. Our record timestamp still measures the time when the log was processed by fluentbit, but now that we've successfully saved that information in the collector timestamp, we would prefer it if the record timestamp went back to its default behavior of matching the log timestamp, thus letting us know when the log was first created and number two. Our structured log is still being treated as an unstructured log.

A

We can solve both these issues by parsing, our structured log. As you recall, we purposely chose not to parsage at ingestion time. So what we want now is a way to retroactively parse it, and we can do just that with the help of a fluent bit parser filter.

A

Once again, we configured to use an appropriate parser, and finally, we get a nice clean, looking record with a record timestamp that now matches the log timestamp and with that we are finally done with the world's longest example. So now, let's go over some tips. We've already mentioned the following tips: use variables instead of magic numbers, use local variables instead of global ones, and for your timestamps use utc.

A

Instead of your local time zone, let's go over some more starting with truthiness, in other words, whether or not a particular value evaluates as true when used in a conditional statement.

A

Let's consider the following: lua values: the boolean value, false nil, which is a reserved word used in lua, used to indicate the absence of a meaningful value, zero, an empty string and an empty table.

A

If you're used to other languages like python, you might expect all these values to evaluate as false, but in fact in lua only the first two values that is false and nil, evaluate as false everything else evaluates as true. So, for example, if you're checking whether a table is valid before you make use of it, make sure you're clear on whether or not you want your program to consider an empty table valid since that will affect what conditional check you choose.

A

Next, let's talk about the contents of the main lua function. As you recall, from the beginning of this talk, the lower filter takes two arguments. The main lure function that gets called by the filter and the lua file where that function is located.

A

Till now, we've dealt with relatively simpler programs where the lua file contains only the main function and that main function contains all the lower code. However, as your programs get larger and more complex, this might not always be the most efficient approach. For example, let's say you have the following main lua function containing code that needs to be executed repeatedly, that is for every record as well as code that only needs to be executed once because the lua filter calls this main function. Every time it matches a record.

A

If the filter matches three records, then both these pieces of code will be executed three times, even though the first piece of code only needed to be executed once, however, if you move the first piece of code out of the main function, then, although they will still both be executed for the very first matching record for every subsequent matching record, only the repeated code within the main function will be executed.

A

Thus, I recommend restricting the contents of your main lua function to only that code. That needs to be executed for every record. On a similar note, let's talk about the contents of the uf file till now, we've dealt with relatively simple programs where we want to execute the same lower code on all the matching records.

A

However, what if we want to execute different gluar code on different subsets of the records? Initially, this might still seem fairly straightforward. We create three different lua filters with three different tags and three different lua functions. But now the question arises: what about the lua file? Do we store the code for all three lower filters in one big file, or does each filter get its own separate file?

A

Let's try out both these approaches and see what happens. First, let's specify the same lua file in all three filters. Next, let's take a look at the contents of that shared lua file. It has the main function for the first group of records, as well as the one-time code for the first group of records. Similarly, it also has the main function and one time code for the second group of records and the third group of records.

A

Now we already know that the main function will be executed every time the corresponding filter matches a record. So let's focus on the very first time that a filter matches the record. Since we know that's the only time that the one-time code will also be executed, for example, the first time that the group one filter, matches a record, both the one-time code for group 1, as well as the main function for group 1 get executed, which is what we want.

A

In addition, the one-time code for group 2 and group 3 also get executed, which is not what we want. Similarly, the first time that the group 2 filter matches the record. Not only does the group 2 code get executed, but also the one-time code for the other two groups and so on for the group 3 code.

A

So as we can see when all the luer filters share the same lua file, we end up needlessly executing unrelated one-time code meant for other groups of records. Now, let's return to our lower filters and this time specify a different lua file for each filter.

A

Looking at the contents of those files, we can see that now each group's code is located in its own separate lua file. Now, let's run through our example again the first time that the group one filter, matches a record. The only code that gets executed is a one-time code for group one, as well as the main function for group, one. No other code gets executed, which is what we want. Similarly, the first time that the group two filter matches the record.

A

Only the group two code gets executed and the first time that the group three filter matches a record only the group 3 code gets executed so, as we can see this time only the code that we wanted to be executed got executed.

A

Thus, when you have multiple lua filters operating on different subsets of records, I recommend creating a separate lua file for each filter, such that each file only contains the code relevant to that particular filter.

A

Moving on, let's talk about some helpful resources that you can use to build upon the stock and to dive deeper into the areas that you're interested in from the fluid bit side, you have the lower filter documentation and you have some sample lure programs from the lower side. You have the official lua manual, but personally I recommend, starting with the programming in lua book, written by one of the creators of lua, as that provides a much friendlier introduction to lua than the manual does.

A

One thing to keep in mind when consulting lua documentation is that, although lua itself is now on version, 5.4 fluent bits of distribution lower is limited to version 5.1.

A

That's because fluent bit doesn't embed native lua; instead, it embeds luagit, which is essentially a fork of lua that is frozen at version 5.1 with a few newer features back ported so before you incorporate features introduced in newer versions of lua into your code, make sure that they're supported on fluid bit.

A

Finally, if you'd like to reach out with any questions or comments, here is my linkedin profile, my github page and my email address, and with that we are done thanks again for giving me your time today. I hope that you found this talk helpful and I hope that going forward. It makes the prospect of using lua to build in whatever additional functionality you need in fluent bit less daunting.

A