Cloud Native Computing Foundation FluentCon: Cloud Native Logging Day with Fluent Bit and Fluentd EU 2021, 14 May 2021

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Customized Cloud Native Logging: Fluent Bit in Open Service Mesh - Sanya Kochhar, Microsoft

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Customized Cloud Native Logging: Fluent Bit in Open Service Mesh - Sanya Kochhar, Microsoft

Fluent Bit is a powerful tool for filtering and forwarding logs, and this talk will explore how it can be easily integrated into CNCF projects in containerized environments, be customized for users’ logging needs and be the right solution for your project. It is used by Open Service Mesh (OSM), ⁠an open-source cloud native service mesh, to help users process and send logs to various destinations. So why did OSM choose this particular CNCF log forwarding solution? How can those learnings help projects in the community inform their choice? In this talk, we will go into a practical example to demonstrate how to support multiple user scenarios with a few quick toggles; more specifically, filtering on log level, custom annotating logs and running traffic behind outbound proxy servers. This will showcase how OSM leverages Fluent Bit’s functionality and plugins for its log forwarding needs.

A

Hi everyone I'm sonia kocher and I'm a software engineer at microsoft.

A

I work on the azure container upstream team, which focuses on a number of open source projects and I specifically work on open service mesh today, I'm here to talk about how we added logging to open surface mesh using flint bit and how you can use it to create a really customized logging experience for whatever product or service you're working on.

A

So real, quick, I'm going to give you a little intro to what open service mesh is and then talk a little bit more about why we chose fluent bit as our solution. Then I'm going to dive deeper into how we actually integrated fluent bit into open surface mesh and finally, I'm going to go in into a number of demos, a three-part demo to show you just how much you can do by changing a few toggles in fluid bit and hopefully that will give you some insight into what you can do in your project.

A

So, just to start off what is open service, mesh or osm so think service, mesh, but open source open service mesh is written in go and it's a cncf sandbox project, which is what it has in common with flint bit. It's also super lightweight and supports the smi spec. Now, if you're not familiar with the surface mesh, what basically happens there is, if you onboard a number of microservices onto your mesh, you can then control ingress and egress traffic, as well as access base control with uh between the apps in your service mesh.

A

In our case, we use an envoy based control plane that can inject an online proxy into every pod to enforce policies and routing rules.

A

So why did we choose a bit so fluent bit is a lightweight solution and that really aligned with osm's goal of keeping things lightweight but, more importantly, it's super customizable and pluggable, which means that we're able to offer our users a pretty vendor neutral approach to logging, because there's so many inputs filters and output plugins available in fluent bit that users can really configure it to their own customized logging experience, there's also tags and matching that allow users to control how data flows through the logging pipeline, which means they can send logs to various outputs and change the nature of the logs in the in their pipeline.

A

It's also important that fluidbit is cloud native which made integration very seamless in a kubernetes environment and then, most importantly, it's governed by cncf. So, as I mentioned before, open service mesh is also a cncf project, and so we can offer our users a commitment that both projects are vendor neutral and quality gated. So they will uphold the same standards and we can also rely on long-term support supportability because of this, this governance, uh and so those are some of the reasons we had when we chose the solution.

A

But now, let's go deeper into what's going on under the hood like. How did we plug open service mesh and fluent bit together? So we have a fluid bit exists as a side car to the osm controller. So it's basically added to the osm controller deployment in the pod spec we've added the fluid container and then also um outline a number of volume mounts that ground fluent bit. uh The fluent bit container read access to to the log files.

A

um The bigger piece in this is really the fluid config map and I would say that's where the magic happens, because this is where users can change a lot with really minimal onboarding from their end, because it's really really easy to configure all of the different parts of the fluid bit config map. Without you know, spending too much time on the actual configuration and focusing more on just figuring out what pipeline is best for them themselves.

A

So, let's take a quick look at what the controller deployment actually looks like. So, as you can see here, this is just a piece of the osm controller deployment, um and this is just the piece that adds in the flip bit container. So the first line shows you that we have a fluent bit as an optionally enabled sidecar and that value to enable a disabled. Fluid pit is uh tied to a cli flag that we added, so the osm cli has control over whether fluent bit gets deployed or not.

A

Next up uh we have the environment variables, so this section is really great because it, um firstly, it allows outbound proxy support. So we were looking at a scenario where, if a customer is running flint bit behind an outbound proxy server, they should still be able to send their logs wherever they want to send them, and so fluent bit makes it really easy to do by just defining these values in a secret and mounting them as environment variables um for the container to have access to so these http and https proxies can be proxy.

A

Values can be set up um with a url and optionally. Also, a a hostname, a password and a username. The other piece of this is, if you want to export any other or set any other environment variables for your config map to have access to. You can set them right here and then reference them, and we will see more about how those values are used. When we take a look at the config map.

A

The most important piece of this are all of the volume mounts, and this is where the fluent bit container actually gets access to the bottle logs. uh So these are a number of volume mods that are required um based on the different clusters that that flow this could be deployed on.

A

There could be different sim links uh to to the log locations, and so these are the volume ones that we have defined and- and I haven't shown this- but there are the corresponding volumes to these volume- amounts- uh also included in the controller deployment.

A

So now going into the section that I said holds all the magic, the fluent configmap itself. So, just to note, I am not going deep into the service and the parser sections here. Those are two other sections that aren't um as customizable, because uh they're pretty much set up to what we want them, how we want fluid bit to function in this situation, so I'm going to focus on the sections that users can really uh plug in and out. So, firstly, there's the input section.

A

The input section in our case is uh the tail plugin and so that uh tails a custom bot in the kubernetes pod. um So here is also where you tag the logs with a certain tag and that really defines where what their journey is and I'll show you. What that looks like in just a minute.

A

um Next up, there's a filter section and the filter section is really great, because this is where you can decide what goes into your logs. You can add or remove keys. You can modify the logs that already exist, and the best part of this is that you can really stack filters together and that allows you to to do a lot more with your logs than than the information that they naturally contain.

A

You can also refine whether or not you want to forward a particular log uh and then again you can use the the tags that match the tags to match the tags you set in the input section. You can also use the tags to reroute where the logs go, which we will see in our demo.

A

Lastly, there is the output section and there are a lot of outputs and, as you will see in a minute, this really speaks to how truly vendor neutral fluent bit is because there are just so many available outputs output, plugins and that list is consistently growing. So in the output section you define one or more outputs that you would want to send your logs to and then use the tag matching to route your logs to to the different outputs available.

A

So this is just a quick look at the fluent config map- I'm not going to spend too much time here, just because we will be looking at this for quite a while in the demo. um The two things I do want to point out, however, are that the path is picked up in the input section, and this is where the log file actually uh resides and then- and this is based on the volumes already mounted in the in the osm deployment.

A

Secondly, there is the environment variable piece, and so uh here right here you can see the modify filter is adding a key value pair with the environment variable that we had set in the osm deployment, and this is absolutely optional, but we just I just wanted to show that this is possible to do so now, let's get into the demo, so just to go over what this demo is going to show you. There are three pieces to this.

A

Firstly, we're going to take a look at a very basic pipeline, so here we're going to use a simple input. um That's just the tail input we're going to use one filter that adds one key value pair, um so that is the modify filter and then finally, we're going to output our logs to the datadog plugin and- and so I have set up already a datadog instance to um send these logs to just wanted to stress that these this is not the only route possible.

A

This is just one of many possibilities, but it does uh give you a basic demonstration of what um you could do with your data pipeline after we've done that we're going to add a little spice to that pipeline. So we're going to add an additional filter that adds a bunch of kubernetes metadata and then also sends the same logs to a second output and then, lastly, we're going to make things even more fun.

A

So we're going to read out a certain portion of the tags to a different output plug-in. So if you wanted to look at two examples of a data endpoint versus one where you know you can send messages, slack is a good example of that. So I'm just going to get started with the demo and we're going to make things more complex as we go on so before we dive in. I just wanted to take a quick look at the fluent bit documentation and right here you can see that there is the data pipeline.

A

That shows you all of the available inputs, filters and outputs. As you can see, there are a lot of available inputs, but we're going to be focusing on the tail one, because that is what's most intuitive and makes more sense in a kubernetes environment.

A

um You can feel free to explore any of these based on your use case.

A

Next up, there's the filters and it looks like they're with your filters than inputs, but these filters are very, very powerful and they each of them have a lot of functionality within them and then there are like, I said, a lot of outputs and- and this really shows how vendor agnostic fluent bit, is and and we're gonna look at three different output plugins in this situation.

A

So, let's get started with this datadog pipeline that I had shown you in the slide um so, like I said we're going to use the tail input, we're going to add one key value pair. So this is the type so imagine our type is test data versus you know, you may have production data and so that that kind of just delineates, the type. um So let's take a look at our fluent config map and keep that documentation up with us at the same time.

A

So, like I said there is the service section, uh the service section, all I'm going to point out here is that we've turned daemon sets off, and so one other alternative. If you wanted to set up fluid bit in a kubernetes environment, is to use a daemon set, and we decided that it made more sense to stick to. uh We decided that it made more sense, in our use case, to use uh to detail logs from just our one controller pod, but you can actually get logs from every single every single pod.

A

If that's what you need. um The next thing I want to point out is that there is a tag being set here, and so this tag the scoop store tag. Basically, um it is used as a map with a matcher in all of the other sections, and it is effectively like a regular expression that includes the entire pot that the log is actually coming from um and then the path referenced is from based on that volume mount.

A

So now, let's take a look at this modify filter, so the modify filter um just requires uh in our case there are there. There are quite a lot of things you can do with the modify filter, but we're just going to add one key value there, and so this key, the key here we're going to add, is type and then the value is test logs um and then next, let's move on to the data datadog output.

A

There are again a lot of defaults already defined that we can continue to use, but there are three values of interest to us that we need to set for our instance. So one is the api key. That is something you can get once you set up datadog. um It's automatically kind of set up for you when you enable logs and then there's the datadocs service and the datadog source source. These are just human, readable values.

A

um Extra keys to add on to tell you in your output, where are these logs coming from so something I want to do behind the scenes is replace this api key with my api key. um I obviously don't want my credentials out there, but that's something that you should know is happening in the background and I will be replacing so now. Let's um go and check out our datadog instance. As you can see, there are no logs there and we can get started with osm.

A

So, let's just install osm and insta enable fluent bit with the config map that I just showed you so now that osm is installed. uh Let's see what is going on back in our data dog log viewer and so right here you can see pretty quickly. All of the logs have started flowing in and all of the keys and values that we had defined, including the original values like the message and the log level, as well as the file that the logs came from.

A

All of that is visible here and then all of the extra stuff we had added like the source, the service and then the type key. Those have been added here as well, and those really make it easier to query your data once it's already in the log output. So now, let's get back to work pipeline.

A

um What we saw a second ago were pretty simple logs, and so what we're saying now is we want something more robust. We want to add kubernetes metadata to our logs to know more about the pod that these logs are coming from.

A

Secondly, we want to use a second output plugin, so we're going to be using the log analytics of the plugin to just show you that the same data can be sent to different places um right now, we're not rerouting or anything we're just using a pretty straightforward pipeline with a second stacked filter.

A

So, coming back to our config map, you can take a look at the kubernetes filter documentation. It is actually pretty in detail right here, but we can rely on the default values that are defined in this documentation.

A

There's a lot more to learn from the kubernetes documentation, including how exactly this filter is able to pick up. The information it needs from um is able to pick up the information it needs from the logs, but we're just going to add the defaults for now.

A

Secondly, there's the azure log analytics output: this is actually pretty simple. It only requires four values to be defined by you, and two of these are actually just values. You need to pick up from azure portal. When you take a look at your agent's management in azure portal, you should be able to get your customer id and shared key so right here, I'm just ensuring that my matcher matches the tag that I had set in the input and then I'm going to add in the customer id and shared key.

A

So these are these correspond to the workspace id and primary key in your log analytics workspace again, I'm going to make those changes offline, because I don't want my credentials on the internet, but um definitely make sure to get those and add those in here.

A

So now that we have that set, let's just take a quick look at our log analytics instance right now. We don't have any data um appearing there. It's pretty empty, but we're gonna just go apply that configmap that we just created and that should have the data flowing pretty quickly. So I'm gonna apply that config map and then the other thing I'm gonna do is also delete the existing uh osm controller pod so that it restarts, and we can make sure it picks up all of the changes in the config map.

A

um This is not necessary, but it's a good safeguard just for this situation and, as you can see now, the logs that are in datadog are a lot more robust. They have all of this kubernetes information that has been added on as an annotation, and you can learn a lot more about the environment that these logs are coming from. um So these logs just got a lot more useful and then, as you can see, the original um key value pair that we had added with the type is, is still a part of this.

A

Let's now also go check out log analytics and run that basic query again.

A

And, as you can see, all of the logs have shown up and all of the kubernetes metadata that we saw in datadog is also present here, cool. So now that we have that going, um I want to envision another scenario now, looking back at datadog, you can see, there are some errors and emergencies and warnings. um You know some different logs, and so what we want to be able to do is say if we have error logs, we want to alert someone whether it's yourself or somebody um on your team.

A

You want somebody to know something's wrong right, so so, let's branch out our existing pipeline. So what we're going to do is if normally, this scoop star label existed, we're going to use a rewrite tag, filter that will allow us to check for a certain condition.

A

So the condition we're going to check for is the word error and then we're going to re-tag that as a page or pager tag and then send those logs to slack- and I just want to note that this doesn't stop those logs from being sent to log analytics and datadog, but it just gets set to sent to three places instead of two so coming into the rewrite tag bit. um The really important piece here is just the rule, so the rule has four parts um the key, so it looks for a key.

A

Then it checks for this regular expression as a value.

A

It then rewrites the tag to new tag, and then it asks you whether you want to keep the originally the originally tagged data and send it to the original outputs or whether you just want to discard it, and so we're going to stick with keeping that data so that we're not um getting rid of anything that could be used in our databases as well. So, let's quickly add that rewrite tags section right here, um so it's a new filter, it's fairly basic.

A

All we're going to do is add name we're still going to match the same coop store tag and re-emit these tags, these logs with a new.

A

A

Great so now, let's take a look at the slack output, um so the slack output plug-in requires you to set up an incoming web hook for your stock channel, which I already did. It took me three minutes because of great documentation.

A

We love that, um and so all I need to do is add these three lines and I will be good to go um so I'm going to add the output and then I'm also going to change the match, and so it we need to make sure that we're now matching the new label, which is the page or label that we just set and then again behind the scenes.

A

I'm going to change this web hook to the web hook that I'd already created for the the slack channel that I have set up, let's go through the same steps again apply the configmap and then delete the.

A

A

Cool so now, let's go back to datadog.

A

Oh actually, let's check out slack first, we can already see some of the logs coming into slack and you can see that they are fatal or contain the word error in them and we'll come back to those in just a second. But let's check out what's happening in our other outputs, so we still have output coming to log analytics. But how do we know that it's the correct output, um I'm just going to quickly check using the metadata that we already added.

A

So as you can see this body, identifier matches the pod identifier in log analytics and let's take a look at datadog.

A

And you can see now that the the metadata is pretty useful for making sure you're getting logs from the right place. Our slack channel is now pretty full. We have a lot of blogs going in and probably somebody getting really disturbed if um they're the person who has to deal with all those logs so coming back to this pipeline, um as you can see, we were able to send. We were able to both stack the logs and send them to a variety of outputs and change the root of the logs along the way.

A

So, looking ahead, um I hope you were able to gain something from watching this talk today.

A

What I will say is that there are so many output, plugins um constantly being added, and the rapid development cycle of fluid bit really makes sure that the new features that we have needed along this way have been consistently added. A great way to engage with, what's going on with flintbit is to join their slack, and if you want to know more about, what's going on with osm, we have a monthly community call that you can definitely join in on.

A

I've also included links to the the osm and fluent bit docs, so that you can also take a look at some of the code I referenced today and just wanted to say huge. Thank you for watching today and definitely feel free to reach out. My twitter handle is included at the bottom of these slides, and this has been fun.