Calyptia Fluent Bit, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Noise Cancelling Headphones for Fluent Bit - Powered by Lua - Matt Lehman & Guillermo Menjivar

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Noise Cancelling Headphones for Fluent Bit - Powered by Lua - Matt Lehman & Guillermo Menjivar, GreyNoise Intelligence Inc.,

This session will demonstrate how to use Fluent Bit to rapidly prototype IP address enrichment and filtering techniques. You will learn how to write a Lua filter plugin for Fluent Bit, how to use 3rd party Lua modules, and how to add a Lua test harness. Anyone that has plugged a device directly into the internet and checked their logs has wondered “what is all this activity?”. Matt runs a global network of hundreds of sensors at GreyNoise, so he sees this “internet background noise” daily and thinks about how to deal with log volume, enrichment, and routing a lot. If you work in a security operations center, site reliability engineering, or data engineering, this session will show you how to reduce log volume within your organization by tuning out the noise.

A

Hello: everyone welcome to fluentcon, I'm matt lehmann, I'm an engineer at graynoise.

B

Hi and I'm guillermo manjuvar and an engineer great nurse as well.

A

So today, we're here to talk to you about noise, canceling headphones for floatbit powered by lua, and I'm going to give a quick outline about what we're going to chat about today.

A

Just kind of set expectations there first thing is guillermo, is going to talk about what is gray noise, I'm going to jump into what you know great, how grain always uses fluid bit internally for our framework guillermo is going to talk a little bit about a hell world, how to write a hello world, lua filter and then I'm going to jump into the gray noise, lua filter and a real quick demo and at the end, we'll have a q a session. So uh camera's going to tell you a little bit about gray noise. Now.

B

Great so yeah, so what is gray noise so greatness tells you what not to worry about, and the best way to think about is think about any server that you have that it's publicly facing and it has a ssh open. And then you jump onto this, your the terminal and you open that file, and you look at all these random ips probing your server and then you ask like well: are they attacking me?

B

Are they a threat to me or they're hacking me, and there are many things that happen in the internet right they're in a super noisy place, so you have opportunistics uh any inner scanner, scanning devices or common business services like think about google bot bing, so they're not really a target to you. So what greatness focus on like what to tell you what not to worry about so it helps you focus on what matters most by filtering out the noise.

B

So that is what greatness does it allows you to um yeah, look at look at data and look at your logs move away. The noise focus on those targets, those threats that actually do affect you and uh yeah.

A

Great, so I'm going to tell you a little bit about how we used little bit at grainois um and so fluent bit plays a pretty big role for us here. We have about a thousand sensors and growing every day um for us and for sensors, that's uh largely just a honeypot framework for us, um and so we have a global footprint with that framework, uh we're in just about every country you could imagine across every provider.

A

You could imagine, and we have a hardware agnostic approach there, so we might be in a cloud provider in a vps in a dedicated piece of hardware in a data center, or it could be even as something as small as like a raspberry pi and so on.

A

All of those different pieces of hardware in all of those different places of the world, we're running a collection of open source and closed source software, and we have various different profiles of those sensors that we run and the one thing all those things have in common, both with the open, closed source software. Is they all output to log files, and so what we needed to be able to do is figure out?

A

Okay, how can we combine all of these various weird log files into some common schema that has meaning to us um and that we can use to provide value into some sort of data set, and so the way we looked at that is, we were like okay. Well now we need to figure out.

A

How can we, um at you know, on each one of these centers sensors, filter, transform and validate this common schema and the logs coming out of those various open and closed source software and then finally route the events out to our data lake, so that we can do various different types of processing, data science etc and get those into a format where ultimately, they land in a data store where we can put an api in front of them and provide value to our users and so guillermo's, going to tell you a little bit about uh fluid filter.

B

Great, so, in order for us to really understand, oh go capture cache the idea of a flumed filter. uh We need to understand kind of like the ecosystem of the pipeline. The fluent makes available to us right. So in its context, we have like an input, parser, filter, buffer and then routing right, so and so in that pipeline. The filter plays the role of transformation right.

B

So you can transform an event that comes in or you can drop an event, so it allows you to do uh intelligent decisions uh based on some criteria that the event has or meets now that, while floom bay allows you to do basic filters that its power really comes by used by combining fluent with the newer jet that it makes available to you. That's where you know we can write a bit more advanced uh logic to make determinations of certain criteria.

B

The event does and then maybe drop it or maybe transform it and pass it down to the buffer and eventually the routing mechanism.

B

So cool, so with that in mind, we we're gonna, now write the very simple hello world lua filter, but before we start writing our code, we actually need to configure fluidbit to actually use our lua filter now. So you remember we mentioned about that pipeline in the context of the flume, we're going to look at three steps, which is the input, the filter and then the output. Now the input in this for for our example, we're going to use this dummy, really super handy mock uh input. The fluid allows you.

B

It allows you to use and, as you can tell in line four, you just generally have some random message, and then you actually can tell it how the frequency of it um being a minute per. Second. Now, for example, it's important to catch the we it's a key value right, so it's first value hello, second value, we're all uh what we're really interested is the filter, stanza, and here the name is actually the type of filter that we want from bit to use, which is lua.

B

The match is the the flag, the tag that we actually assign in line three um line. Nine tells you tells fluent where to find the script and then line ten tells it within the script. I need you to call the function handler and then the output, just it could be sender output, for you know, for our example, we're just going to use the unlock file.

B

Cool, so now now that we have plumber configured right so now it's ready to be used. This is the this is how simple a lua filter looks like if you written any aws lambda. This would look very very familiar and the reason is because there's a predefined set of inputs that you take in predefined set of outputs that we expect from you. So the tag is a the tag associated with the record. The timestamp is the time sum of the arrival of the event in the fluent framework and then the record itself.

B

Now, if you remember from that sample from the input the dummy input, we got first value second value. So what we're going to do we're going to transform this event through this through our hello world, lua filter. So, as you can tell here, the new record, full value will be the first value plus some spacing and then the second value which should give us the hello world and then now we're gonna return. It now there's something to say about the return as well. Now, okay, perfect!

B

So it's key to know the caveats of the code that we return. This is how our lua filter communicates with the greater fluid framework. So if you return a negative one, that means that you're telling your lua filters telling fluent b to actually drop that record. If you return to zero you're telling it well, you might have peaked at the record. You actually didn't modify it and then, if you return a one means that you actually transform the event. Something actually changed, as we did ourselves here.

B

So the cool thing here is that there's a very simple yet elegant way for your your lua filter to communicate with the greater um record. I mean the frameworks right so now, there's because we're taking this usually for if you are going to write up into a filter, you are going to deploy to your production servers. So we like men uh uh in gray, notes we actually have tests for free plugins that we have written. So we also want to show you how to write a very simple test using the busted lua testing framework.

B

Now this is bdd, so if you ever written b test, this looks very very familiar if you've written ginkgo and go super familiar, it's the same and more or less looks like it uh with some like nuances now, here we do the same thing as we uh they actually lure. The flambe would do, which is in line three would load our code.

B

We set some inputs in parameters eight to eight to twelve, and actually then we just exercise our handler in line fourteen and notice that the three events we for the sake of this example, we actually didn't care which one uh inspect the record, which is the v in this context, and then we just assert that basically, full value equals hello world and that's it right. So now we're testing it and you can see the bottom, how simple it is to test to run busting.

B

Now you can, which means that you can configure it with your configuration management uh uh flimbit. You can write your lua script. You can now have tests, so you can actually wrap this into a really nice ci workflow.

B

They promotes they actually not only ships your code, but you can actually have so much version that with confidence that it will work so definitely buster is a really cool framework that complements the lua filtering setup.

A

Cool, so I forgot to mention uh earlier with the busted framework: we're also leveraging that already in green noise for a lot of the ecosystem, that we have, we've got a lot of pretty rigid tests for the various open source projects um and and closed source stuff that we're building on top of we've got a a robust test framework that we've leveraged busted for and it works really well for us. So the next thing I'm going to jump into is talking about the grain noise filter.

A

So this was you know once we started gaining that experience with lua and learning how to leverage flint bit in our ecosystem. We realized hey. We could, you know, write our write on a fluent bit plug-in in lua.

A

That would actually call the grainois api itself, and so we started walking down that path to see you know how we could do something like that and what it would look like, and so the the reason here that we wanted to do this is so that we could remove or de-prioritize noise from your logs and so for us. We break noise down um in this specific plug-in into three categories, which are really noise riot or bog on and so noise is you know that passively observes scanning someone passively observes scanning or crawling the internet?

A

In this case you know we're going to be looking into an ssh log on a linux server and what happens a lot of times is.

A

Somebody just is using like an ssh brute forcer, where they're scanning the entire internet, um with you know this collection of usernames and passwords the ssh and it isn't a targeted attack and it generates a lot of noise that might look like some sort of attacker when in reality, it's just you know: they're they're, blasting, the entire internet, with this brute force and so riot's, the second category which stands for ruled out and those are predefined, address spaces like google, bot or bing bot, where you know, google or bing themselves, are saying: hey look, you know these slash 24s or these subnets.

A

These are dedicated to our bots or our crawlers, and so grain always internally has um a framework that goes out and collects all of those sorts of resources and constructs it into what we call our riot data set. And then the third category is the bog on category which is private, ipspace, and that might be you know, 192.168 or it might be like a multicast ip space or any ip space that doesn't that doesn't um turn out to be public ipv for space according to the various rfcs.

A

So the last part is that we want- or the second part here is that we want to route these events based on tags injected via the filter themselves. So those categories that we're getting back, we want to say, hey, you know: we've extracted this ip address from a log and then now we want to see like how do we route this specific event or this log line based on these tags that we've injected via the filter and the last place.

A

The last piece is to talk about where the best place is to use this sort of filter and so for us we're using it on a single server just on an ssh log or the offload analytics server, because it's a good place to talk about it's a simple enough thing to implement and put into a small ecosystem to test, but really anything. That's some sort of public-facing asset.

A

Where you have um you know global inbounds allowed and it might be an api or vpn or firewall itself, where you have a lot of noise there and you want to try to filter that down and determine uh you know what are these things that are targeted and how do I remove that or de-prioritize that and it's important to point out on the deep prioritization that that is not that it's not just dropping necessarily, and it's largely up to the user, to determine well hey.

A

You know I just want to send these log events to some cheaper storage or put them in s3 and keep them out of some. You know more expensive data store and so for us, for this great noise filter config. One of the first things we started doing is you know just like guillermo mentioned earlier.

A

We need to build this config, and so we started with the input section and we uh used a tail input plug-in and we used the ssh log that we pointed at here, and this is just an example line showing one line of log here, and here you can see you know: somebody's blasting, um a login for user wordpress with some fake password from this specific ip address and probably doing it.

A

You know across a bunch of different usernames and so we're able to take a common regex that you can find off the internet to be able to parse that message, whether it be syslog or the authlog or any sort of specific regex part routine. So we were able to put that into the config and able to chunk this up, and the part that we really care about. Is this last part in red here where it starts with invalid user?

A

Because that's the thing that contains the ipv4 address, and so we extract that out and we call it message and it's important to pay attention to that label, because it's going to come up here in a minute so uh with a quick prayer to the demo gods, we're going to transition over to a real, quick demo that I'll kick off and then be able to talk in a little bit in depth about the code paste there.

A

So I won't go into the exact folder layout because it's pretty small here, but I'm going to go file by file. But the first thing I'm going to do is go ahead and kick off the demo, which I have make targets for. I'm just going to kick off the make run rewrite because it does take a few minutes to get through and we'll. Let that run in the background. While we're talking about some of the files here.

A

So that's just going to kick off the docker file and it's going to start running the fluid bit configuration based on this rewrite conf.

A

So what we did is we built a docker file that contained all of the various lua resources, whether it be busted or lua requests, or the caching framework itself to be able to build that in isolation. So it was really easily easy to extend various libraries and continue to grow that lua script itself. That we'll show here in a minute uh we we threw together a quick make file and you can see the run rewrite target that we just ran here for this test.

A

You can see we're just you know calling into docker here and we we are targeting the rewrite. Config file looks like I had a small error there, but we'll chat about that in a second um and then, if we go into the rewrite config file, we can walk through this. Just kind of like guillermo did earlier, and we can see up here at the top.

A

We have a service block that designates uh the port that the stat server itself is going to listen on and we'll talk about gathering stats here in a little bit and the parsers file where that regex is located. I mentioned earlier the next chunk again is the input section where we're using the tail input configuration we're.

A

Looking at that authlog, which is you know, a log file, an ssh log file containing 10 000 logs little 10 000 log lines, and then there are a few other attributes here that are available for config that we won't go into. But you can look in the details on the flipbit documentation page and then next is our script. This is the important piece.

A

This is us jumping into lou at this point with uh something pointed at a grey noise lua file and then specifically calling into the gn filter function, so we'll jump into that here in the green noise lua file here and look for gn filter and here's our function right here. This file has a bunch of other helper functions.

A

We won't go into the details there, but just stick to this gn filter function. So again, this is just the handler we're not using tag within this handler, but we're using leveraging timestamp and record. So the first thing we're going to do is extract the ip out of that message field and we're using ip ipfield here, but we actually set that via an environment variable so that we can say: okay, let's just look at that red portion.

A

If you remember earlier and we're going to try to extract the ipv4 address out of there, which we do with a simple lua match here.

A

The next thing we do is, then we take that ip address and we say um you know. Maybe this isn't the first time we've uh come into this section of code. Let's first check the cache and see if we've seen that ip before. So we look at an lru cache, which maintains, I think, it's uh ten thousand the last ten thousand records and also has a time based roll off. And so, if it's in the cache great, we go ahead and return.

A

There's no need to call out to the api and there's no need to even check if it's a valid ip or not, because we have an exact record in there and we only set valid records in the cache.

A

If it's not the next thing we do is we go and we run a check ip routine, that determines if it's a valid public ipv4 address and if not, um basically, if it's a invalid address or if it's a bog on address meaning it's. You know one of the rfcs that covers the private ipv ip space or um like multicast or one of the other ones that wouldn't we wouldn't normally see on the internet.

A

So if it, if it is a valid ipv4 record, then we say: okay, let's go ahead and reach out to the grainoise community api. So we can see this function here that we're calling and real quick I'm going to hop over to gray noise. The grain always community api documentation just to walk through that, and so you can see here.

A

There's we have a response body that contains noise and riot booleans and those are the only uh fields that we really care about in this. But uh if you're interested you can hop on over to developer.greenoice.io and check out the free community api and some of the other, uh some of the other information you can get out of it.

A

But for this purpose we're just going to talk about those two fields so jumping back here, we can see that we get those two booleans out g noise or g and riot, and then we just return the rec or at that point we only return if it's invalid and then the next. uh The final step is we set that in the cache so that it's available for us the next time we come around.

A

So it looks like our our test finished, but before we jump into that, I wanted to jump into how we leveraged busted tests within with this guillermo talked about this a little bit earlier.

A

But this is just me coming through and essentially running that check ip handler with a valid ipv4, making sure that it doesn't come back invalid and then checking it with a bog on address making sure that it comes back bog on the same thing with link local and a few other like multicast, addresses, etc. And this is just a nice example for leveraging busted um to get some consistency in our lua code because, as you start to write it, you know the code base grows.

A

A little bit it's good to have test coverage in here to isolate things so that you know when your issues are in lua versus when they're in fluent fit themselves.

A

So, finally, if we hop back over to our um our test, uh it's finished now and I can go ahead and jump over and run the stats endpoint, which curls back to that port 2020 just runs a little bash script to generate some metrics out of it.

A

So I'm going to run that just to show that it's live, and then I'm going to jump out of this and actually go back to our slide deck to talk about the results there, and so with this one we can see, we had 10 000 records, seven thousand seven hundred and four uh those were noise. Six percent were bog on or invalid, uh which yielded about seventy seven percent noise.

A

So a lot of the traffic that was hitting this specific ssh um and the auth log was largely noise took about 134 seconds and we were hitting about 75 records a second on average, which you know we're not winning any races here. This is obviously just for a prototype and really just to prove out all the different moving pieces and how to go about writing a plugin to do ip enrichment or some of the things that we're doing here and then guillermo's going to talk to you a little bit about where we're taking things. Next.

B

Yeah great, so, as you guys see, we were extremely excited of finding out that our thesis that we had uh of taking gray noise, putting it with fluid, uh made sense. The 77 of real noise from a real ssh log validates our hypotheses now. So what we're looking for now is actually, how do we reduce the per ipip request? How we actually can do offline noise, caching and we actually went back and started looking at some of the good um filter, fluid filters that they are out there and matt's mind.

B

Guip has a really nice model that we actually are tailoring our production, uh great noise filter source and the other thing is because we want, because we all use fluent in production and we trust it and it's reliable scale performance we're also looking at how do we make this this plug and date? Do you guys the magic's demo as a native plug-in, so you can also easily not only if you find yourself interested on what grain noise is doing, but also the need of reducing the noise in production right.

B

So we want to make sure that it sounds something that the community can use um that is performing and scalable. So we're actually really really excited about this part of things and and for the people to actually see how core great knows it's um and okay, there you go perfect uh and the last thing the the repository the map mentioned there. It's already a public repository matt mentioned the dockerfile file. We actually have that image in docker hub, so you're actually free to use it, not only it brings all the dependencies.

B

They mentioned the caching but also brings busted already. So if you want to just play around and use as a means to get yourself acquainted with busted writing some lua and testing. That's actually a good image to use um the greenhouse api. So, as matt mentioned that uh the input that we're using for this demo is actually a free community api, which is you don't even have to authenticate to actually play with it.

B

So you just can strap curl if you just want and more importantly, if you want to look more of the other options that the ap the greatness api, uh it's closest to you, that's a great source for you to okay, it's a great resource for you to learn more about it, and also we found this blog post. That was super helpful and it I and when matt- and I were like doing the slides we we needed to put this in- do a shout out.

B

Oh, how valuable that resource was so definitely one blog post for you to go and check it out. So.

A

Awesome uh so I've got some links here at the end uh just to our greenways website, our github repo, which has more things than just this. In it, we've got a bunch of integrations and other tools that we publish and hopefully more to come there to engage the community, um our grey noise twitter here.

A

uh If anybody wants to reach out or follow us on grey noise and then finally, our community loss or less our community slack here at the end, uh you can sign up for there uh sign up uh for grain, always slack access there. That's our community slack guillermo and I are both on there uh and a quick shout out to our community manager uh sapria.

A

uh She can help you out on there too uh kind of get you engaged um and and maybe help answer some questions, even if it might be about this flimbit plug-in um so anyways. I appreciate everyone's time, uh especially the fact that this may have been a little bit. Clunky guillermo and I are very far apart today, um and so it was awesome. We were able to synchronize on this and be able to get through this. This chat will be available in the q. A section.