Calyptia Fluent Talks, 25 Feb 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Fluent Talks | 004 | Fluent Bit + Lua

Description

Please join us for Fluent Talks! Our weekly webinar and office hours, on Fridays at 2PM Central. Streaming live on YouTube.

This week we'll be discussing Fluent Bit + Lua. We'll include some cool scripts and real-world examples to help you get the most out of this powerful Fluent Bit filter.

#lua

A

A

B

You're good to.

A

A

Hello, hello, hi. Everyone thanks and welcome to this fourth episode of fluent talks sponsored by kalitia. Today we have one of our engineers joining the this session. His name is tiago he's joining us from brazil. I think that he can introduce himself later and today we're going to cover a little bit of knowledge of lua, scripting and lua scripting.

A

Sorry, I got some return on my oh. You just lost everything, so we're going to do some intro about a lua, how to use lua tables a and why this is important, because, when you're doing um managing pipelines with fluid, sometimes you want to modify your data, but actually sometimes the filters are not enough, and sometimes you need to have more custom logic like conditionals or remove kit or nest some keys at a different level, and that is really important, and since we have a lower filter, that is the the right choice for choice.

A

For that, that's what most of the users are doing so yeah. So I want to introduce to thiago who is going to take over this presentation and please enjoy thanks.

B

Hi, my name is thiago. I've been working for calipte for the past few months, mostly on fluent beat, but also on other related projects like sigmatrix, and today I'm gonna show a bit uh how how you can extend flintbeat using lua.

B

So I've prepared a little presentation. Let me share my screen.

A

B

Okay, I try to share when I try to share a chrome tab. It seems that I get muted.

B

Let me try again.

B

B

Can you hear me yeah, you're, good, all right, so I'm going to start with the brief introduction to the word: it's a dynamically typed language, very simple, but at the same time it's it's powerful. You can do a lot of things with it. You can do object-oriented programming, functional programming, uh it's very efficient.

B

uh I've never done any performance in benchmarks myself, but I've read that it's it's a bit faster than other similar language like python and ruby, and it's basically it's the main use cases to embed and extend uh other problems. It was designed to be easily embeddable very lightweight.

B

The api is is very good for embedding.

B

If you try to embed another language like python, because if you know that it's it's a bit conventional, you have to manage a garbage collection. Reference and lua has clean enough api. That makes it simple to that and it's even even though lua is not very used uh like as and use a programming language like you. Normally, you don't write programs in lua, it's very widely used as an extension language in many games, uh world of warcraft, for example, very famous game.

B

It's it uses lua and many other games. I think uh roblox also uses lua, so it's very widely deployed in our district and since the goal is embedded, it's also uh the target platform for who is the ncc.

B

So it's uh on purpose, it's a very old, it's an old version of c. That's, I think it's the most portable platform uh that exists today. uh So when you, if you can have a c compiler for a platform, then you can build lua for it. It's uh for that reason, there are very few batteries included. Basically, lua is the standard library of lua is mostly a high-level wrapper around the c standard library.

B

It doesn't have much operating system specific calls and apis.

B

So there is legit. uh It's an alternative implementation of lua 5.1.

B

It's fully compatible and it uh has a cheat compiler that that translates uh lua bytecode to the the machine bytecode, and this is very optimized. It's I think it's uh some might say: that's the fastest fastest scripting uh runtime uh that exists. There are many benchmarks that show it uh having greater performance than many uh runtimes that are created by big companies like javascript v8.

B

Also, it's very lightweight compared to those runtimes and it's it's. The version that's used by fluent beats with embedded legit uh and not the standard word.

B

So uh let's have a look at some of the data types of some basic blue code here, as you can see, uh it has the standard types like boolean numbers. It doesn't have integers so now, lewis numbers are basically doubles.

B

It's floating point point with double precision strings you can have single quotes, double quotes and multi-line strings.

B

So here you can see some examples of how you can write conditionals, it's a very clean, syntax, very readable. uh It's easy! It's meant to be easy to understand, to read very explicit too so, as you can see, it also has support for c a format format strings.

B

So it's very it's very. It can be very familiar, very easy to use by c programmers.

B

So the main uh data structure of it's actually the only data structure of lua are tables tables are basically uh hash tables. It's implemented as hash tables. It's it's an associative array with tables uh you can uh out of the box with standard library. You can use tables as lists and as maps.

B

So you can see an example here we have a list of three items and you can insert items you can iterate uh to the the items with the index and you can also use it as a map, associative array. You can add arbitrary keys and in value street and with tables you can. Basically, uh you can basically implement any everything else that you want.

B

It's the only way to structure data, creates advanced data structures and is by using tables.

B

So here you can see a function definition.

B

One thing to notice that if you look at the I I've created some examples here uh when you do an assignment like that, when you assign to to a variable like that, it's actually assigned to a global variable so to to create a local variable inside a function to ensure that the variable is not defined.

B

You have to precede the the variable with local. It's how you declare a variable to be local to the function. I'm going to show there'll be some examples later. uh The exceptions are parameters, of course. So when you have a function like this, the whatever you define inside the function is only visible inside the function.

B

So here is a simple factorial function, showing.

B

Very simple way to define a function, if else basically functions of of the language uh so loops you you can.

B

The main way to iterate is by using iterators, which is a it's a construct of I'm not going to go into much detail into how it works. But uh basically you call a certain function. That's that returns on it iterator- and this is this- makes it compatible with the for loop, and you also use. If you want to do a plastic look like uh iterator to arrange there is a function that returns a range iterator.

B

So that's basically how we do simple for groups.

B

So that's basically it's very some very basic things. I've shown about the language. I think that it will be uh very extensive to cover everything. There is supports quite a few extra features.

B

It supports cool routines, it supports uh while loops and many other features- and I think the best place to learn more about that is the reference manual.

B

I'm going to show some examples later, but for now I'd like to talk about the ecosystem package acquisition for work. So this uh this program called ruler, rocks you can install it on many linux distributions, it's very easy to install and it allows you to install lua packages. It's I think it's similar to npm for node.js and pip for python.

B

So anyone can create packages for modules and publish to lure rocks registry and it will be installable like this. You can install modules and then you can require you can use the modules like this called the require function and that's basically it's very familiar. It should be very familiar to javascript node.js programmers.

B

And this is all uh easily accessible through flintability.

B

So when we let's talk about blue in the context of flexbit, basically you have the full power of a programming language to do any kind of processing on fluid bit records.

B

Normally you have you, you can use 20 bit configuration tags or matching tags to apply certain plugins to to records, but with lua you can also use any kind of criteria to draw to modify records. So it's very flexible. You can split or remove records and you can install any pack. You can use any lower rocks package.

B

Like I've shown before, and I'm going to show some examples of that.

B

So this is a very simple example filter. It doesn't do anything except print to the console.

B

uh Assuming you install this inspect module, it's a module that allows you to stringify any newer object, including tables. This is not something that's available on the standard library, so you need to use a third party module for that, and this is what what's being done.

B

It's basically, uh this handler function is going to be called by fluid bit with this. The following parameters, tag, timestamp and record record is a table that contains the fluent bit record keys and in this example, we don't do anything. We return zero as the code, this coded it tells what tells fluid what to do with the record, uh with the record that you're returning, if you return zero, it means uh flint beats not going to modify the record that was fast.

B

There are a few, other values can be returned and here's an example how to run this. I'm going to show later running here.

B

uh I've created these in uh obviously this into a repository, so anyone can run this later, but uh this is basically how you run it. You in this example we're using the input plug-in for the cpu cpu data, we're passing through the lua filter, matching all tags and we're telling that it should run this. This script called log.lua and call the function handler and for output we're just using no because this is just an example we're showing how to just how to lock records.

B

So if you run this you'll be seeing uh every record passed uh uh sent to the from the the cpu input, but it will be printed using the lua, inspect module.

B

So here's another example: it's a very naive, csv password it's uh so it also shows a bit uh lua patan matching library, lua doesn't have regular expressions. It has a very simple pattern: matching uh it's it's not that it's similar to regular expression but less powerful.

B

uh It allows you to have some of these constructs that are similar to regular expression. Basically, here we say uh this assuming that we're receiving records that contain csv data on the on the log key.

B

What we're doing here is we're matching all the pieces of x that are not a comma and then we're inserting into a field stable, so it we're creating like a list of fields and and then we modify the record.

B

We pass fields like this. I'm also going to show this running later, but you can see uh below how it it. This is called uh very similar uh in this case we're also returning one, because we want the flint bit to modify the record uh in.

A

B

Case we could also.

A

B

Written two because we're not interested in the time step. But since we returned zero and we written run as the first value, fluid bit is actually going to replace every time by zero.

B

And so there is more uh information in this. You can learn a lot more about your fluent bits in this resource resources.

B

I'm gonna share my let me share my another window now that will uh show a few examples. A few more complex examples of a bit working.

B

So this is uh the git repository that contains uh source material for this talk and I've created a docker file and some some things that allow make it easy to run uh so. The first example that we saw that was login cpu uh data to using lua inspect model.

B

So this is how it it looks like using the spec model. We fluent bit creates this record with a bunch of cpu data and our script receives as a wood table and it's printing to the console.

B

uh This is the file. That's been run, let's show the the symbol, csv asap.

B

B

So this is, uh let me show the file.

B

This is a example: csv file, that's downloaded, it contains many cities of the world in csv, so flint bit by default, does not have csv support.

B

So what we're doing here is basically we're splitting uh each row and assigned to the this view uh key on the on the record and, as you can see, it's zero on the timestamp, because we replace the timestamp just to show how it works.

B

I've written a few more complex examples.

B

uh It's a more complex, csv facet, written using pure rule, and in this case we actually.

B

We actually passed the header.

B

You see we we take the first row, it's some extra data here, so the function we're actually using a config file. For that this is the config file.

B

We are passing the csv dot script and we're calling the process record, and this is the function that's called by fluency. uh As you can see, we declare uh headers uh variable in the module and the first record that we received are actually going to be the headers of the csv file.

B

See it here contains the column names and the first thing we do is we call the split csv function and we return -1 to drop this record because we don't want point b to process direct the record. It's just uh the headers and we store the headers into this table and after that, all the next record that we receive we're going to basically call the split csv function. And then we create a table that contains the record data and we assign each header uh key from the fuse.

B

And this is the result. We get a map.

B

Each column name is a key, and each row is as a value here.

B

So I have one final example.

B

This is basically the same uh as the csv example that they wrote, but this one is, you uses l crack opec. Is uh it's a powerful library do a library for patent match it's actually a full busing library. You can pass even a full programming language using lpeg and you define the grammar uh using language. Constructs I'm not going to go into much detail, but basically allows you to override operators to do whatever you want.

B

So the library auto created uh this language that allows you to define languages grammars and then you can use it to pass anything here. We just took this grammar out of the object website and we use it process csv in.

A

The same manner as.

B

As the other case, but in this case just to illustrate- you can see that we're returning two and we're changing the timestamp to zero, uh but this is not going to be used because uh we're done in two. So let me show how that works.

B

So, as you can see here, all the time stamps are left intact, but the the records are the same. We practice the records in the same way.

B

So uh all these uh material, uh all this source code is available. Anyone can run I've created a docker file that will set up a flatbeat, a docker image based on flatbeats official image, and it will install all the dependencies. So all you need is docker and there are some instructions on on the repository.

B

That's basically it I hope you enjoyed.

A

Thank you trevor. That has been really helpful.

A

I know that not all users are aware about the capabilities, and the intention of this is like how we can also provide more knowledge on managing records across the fluent bit pipelines, and we experimented also with other ideas on how to improve experience with scripting with lou and fluence, and we might have some news. You know in a couple of weeks, so well, lua is rather used. If he used to manage, I don't know nginx web servers right configuration has conditions with lua.

A

If you used to, I don't know, develop games and roblox also is purely lua based. So this is a very interesting scripting language, with just-in-time, compiling functionality that, when you said the the performance penalty is very, very minimal. So actually we found use cases where people has a. I don't know five to seven or eight filters, and actually all of that was a bit more expensive competing time.

A

They just have one single lua script, doing the same job, so the invitation for the the audience is just take a look at the lua script into the filters. We're going to have um all the information listed in this uh this session and then the youtube channel and feel free to just leave your question we'll be happy to to answer them and if you want to contribute to deploy, you know that we hang in the fluent website or github.

B

So yeah I'd like to add that we created a community discussion in flint, beats a repository git repository uh on how we can improve the user experience. So anyone in interest can go and and can give suggestions on what we can do to improve.

B

uh As as as I'm working on this presentation, I realized that we can use blue rocks package manager, so it's perfectly possible for us to simply create, as like a standard library of fluent many uh common tasks and then publish to rural rocks, and then anyone can install it and use it.

B

We can also embed a standard library, so we're still learning, but the best approach is to improve the experience to users.

A

Okay, yeah, that's a that's a really good idea and actually for the discussion. Would you mind if you share the website where people can take a look at this discussion.

A

So when they watch the video they can, you can see where to go.

B

Do you mean the the material that I've shown for the presentation? No.

A

No I'm referring to the discussion about fluent bet and lua.

B

Yeah yeah yeah, I'm gonna, I'm gonna share it. It's going to be in the description.

A

Okay, no, no! I just wanted to ask if you can do it like right now, open the browser.

B

Okay: okay, okay,.

A

Not everybody is using github from our users, so that gives some more perspective.

B

Just give me a second.

B

So this is a freight github page.

B

If you go into the discussions tab, uh I think.

B

Let's search for you.

B

Okay, uh he did it here, it is so this is the discussion.

B

It was created uh last year, but uh anyone can come here and contribute give suggestions.

B

We're going to consider it.

A

Awesome so people interested in this can go to github discussions and do a search for the lua keyword.

A

A

Well, I I think that we have and reached the end of this session, thanks diablo for your time on this presentation, and we hope to have you again presented here- maybe a different topic or more advanced lua, stuff and yeah. We just left the invitation open to the audience to take a look at the fluency project and yeah. Follow the fluent talks, we're going to share different uh technical topics, not just about flambe but others about.

A

Oh we're going to have special guests from all a custom projects like prometheus, open telemetry, so we are going to discuss and show some integration across these projects. Okay, that will be everything for now. Thanks. So much for attending have a happy.

B

A