Calyptia Fluentcon Europe 2022, 19 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Monitoring Fluent Bit in Production - Pandu Aji, Microsoft

Description

Monitoring Fluent Bit in Production - Pandu Aji, Microsoft

Fluent Bit comes with built-in features to allow you to query internal information and monitor metrics of each running plugin. But how do you decide which metrics to use for measuring the reliability of your pipeline? How do you leverage the storage metrics, and what are the challenges? How do you detect if your logging pipeline is unhealthy and if there is any congestion? This talk will describe our Fluent Bit monitoring story. It will briefly touch on some of the design choices but will primarily focus on how to monitor Fluent Bit in the production clusters.

A

Good afternoon, good morning or good night, if you are attending this, virtually my name is bandu aji.

A

I work for the azure machine learning infrastructure team at microsoft, so we built we run and maintain kubernetes clusters for the azure machine learning team and we build tools to improve uh the speed and the reliability of the deployments and we and to also increase the observability of the application running in the cluster.

A

About five years ago, we started to explore logging solutions for kubernetes.

A

There are two open source projects that caught our attention fluently and fluent bit. We went with fluentd at that time because fluent bit was missing, some key features, files file system buffering and then the scripting blocking, but fast forward about a year ago our business is growing and we are looking for more efficient logging solution.

A

So we revisited fluent bit and we found both. Our missing features have been supported.

A

And also we found that many people have updated the fluent bit for their kubernetes logging solution. So we made a switch early last year and overall, we've been in fluent ecosystem for about five years.

A

So this is our production environment, the current state of it. uh Some of the interesting numbers are the overall log volume which is somewhere between 750 to 850k per second in our pcs region. The log volume cluster is about 80k per second, and the log volume per node is varies between 1 to 2k per second.

A

So today I'm going to briefly talk about our logging architectures just for a little bit of context, and then I'm going to show you how we monitor our the health of our logging pipeline just by using one single metric. So let's get into it.

A

We have two logging solutions. One is the sidecar pattern. It's fairly straightforward, the application right logs to the login agent sidecar through through forward or message back protocol.

A

The logging agent sidecar then sends that logs to the storage backend, the logging agent cycle is automatically injected by the mutating admission controller. The application can enable the cycle injection by adding annotations in their deployment spec.

A

In our case, the locking agent sidecar is microsoft, internal agent. But technically you can replace the logging engine cycle with fluent bit uh to send the logs to the login agent cycle.

A

We instrument the application with the logging library, so one of the downside with the logging library is that when you have applications written in different languages, then you have to support multiple logging library, but we needed the the logging library for two reasons. One is that we want uh our.

A

We want all our uh application logs uh to have free defined schema, just to simplify the login queries, and to uh is that we we want to have the application to have the option to either uh to be able to be uh to be able to write to standard art and as well to the uh right directly to the ah cycle, so the logging agent will when, when the login agency hey, there is a cycle on my bot.

A

I will write directly to the sidecar and if the application disable the cycle injection, then all the locks will go to the standard out, which is then will they are handled by the second solution, uh the forwarder and the aggregator pattern.

A

It is the most common pattern out there. I believe so the the forwarder is is a very lightweight demon set that tells all the container locks on the note and then route them to the aggregator surface, which is then subsequently the surface will distribute the logs among the aggregator parts.

A

So so the aggregator uh then write those logs to the azure uh storage backend. uh In our case, it's azure and the.

A

And the aggregator actually uh does all the business logic to transform the logs and then send them to the sidecar.

A

We also for the data safety security. We enable the file system buffering on the forwarder and then also for to to adapt to the load. We enable the hpa on the aggregator and next.

A

So, let's talk about fluent bit metrics, uh so the http server on fluent bit, they ex it exposed uh multiple endpoints.

A

Two of them that we are going to talk about is the matrix endpoint that is in prometheus format. This matrix exposes the metrics for each running block plugin. The second endpoint is the storage endpoint. It exposed the storage information, but they are in the json format, and this is a output of the storage endpoint.

A

So we see chunks. What is chunks chunks is how fluent bit groups and store the log the logs in the file system. So there are two types of chunks: the up chunks there. They exist in the file system as well as in them as well as in the memory. They are chunks that are currently being processed and you can configure the maximum number of up chunks allowed in the memory the down chunk. The dongjeong only exists in the file system.

A

So, if your, if your logging pipeline, if they are healthy, typically the down chunks, is usually zero or very close to zero, and if your pipeline is congested or slow, then the up chance then starts uh starts to grow and when it reaches the maximum limit, then the down jump starts to starts to accumulate.

A

So this is actually the metrics that we use to monitor our uh logging pipeline before moving to the next slide pay attention to the json path of the fs chunks. Up and down it is storage, layer, chunks, fx, chunks, up or down.

A

uh To export this storage metrics, we use json exporter deployed as a sidecar as well that you can find the json exporter in in prometheus community github.

A

So this is uh the example of our config, so the name here flow and b storage layer is the metric name. Prefix and the value of the values field uh is the value is the metric that you want to export. So in this case, uh is the fs chunks up and fs chunks down and the exporter uh to get the metric uh value.

A

The exporter will follow the json path, uh those those are the one that is in the curly braces and if you look at it is the storage layer, uh the chunks that fs chunks up or down.

A

We use the cubes prometheus stack to deploy our prometheus, and this is our surface, monitor configuration.

A

So does anyone use cube from how many people are using cube, promited stack here, okay, cool, so service monitor is a crt that is basically tell prometheus where to script. The metrics, the first endpoints here, is just the matrix endpoint on the fluent bit container.

A

The second endpoint is the probe endpoint on the json sidecar container, so the so when, when prometheus script, this probe endpoint it will fetch the json metrics from the target, convert them into prometheus format and then send the result back to the response.

A

And to just visually test this you can use keep cattle forward, for example, this, for example, you can see on the screen.

A

If you look at the url is the prop endpoint and then the the target query parameters and the result you'll get the fluent bit in the prometheans format. So it's the fs chunks up and fs junction.

A

So let's take a look at our live dashboard.

A

Let me switch for a second here.

A

So this is our.

A

Still, loading.

A

A

So while it's loading, okay, it loads now uh the top panels here uh these are the global summary of the down chunks across all the cluster. So it's kind of nice to be able to see your the overall health on a single uh pin and um you'll see that the down chunks are pretty much zero.

A

There is mostly likely five and then they drop back down to zero, and these are from japanese and you can see this one is from germany, west central uh so because this is global, the drop down here doesn't work on this panel.

A

Ideally, you can put this on on a separate dashboard with the drill down, but we just consolidate this together.

A

So the panels on the file storage uh group, there are the down chunks and the up chunks for uh by pot, and you can. Actually. This is our busiest region which is in east u.s, so we can maybe switch to brazil, south and change. There is no downtown, it's super healthy and if you switch back to east u.s,.

A

And then in the il group, the the panel on the left is the input versus the output rate, and, if you look at it, they are kind of like overlapping with each other which what you expect from a healthy pipeline.

A

uh Oh actually, why does it show? Can you switch excuse me.

A

Can we switch the camera to the.

A

Can we switch the camera to the like uh my?

A

How do I switch this.

A

A

Since we cannot switch this, can we switch this to the uh the other and okay, never mind I'll, just go ahead and use the pre-screenshot thing.

A

Can we get back to the presentation? Please?

A

Okay, oops yeah. So this is the dashboard. The top panel is the global view, down chunks up chunks and then next is the input and the output rate.

A

So if you, if you remember, we look at the overall log volume- uh that's actually taken from this graph, so it doesn't, it doesn't include the volume from the uh logging agent cycle, so it's only from the forwarder.

A

The next two panels, on the I o is, is the input rate and the output rate by part. The last is the errors and the retries we seldom use it. But it's it's a nice, uh it's nice to have for an additional diagnostic tool.

A

Next, we'll take a look at of uh take a look at couple of studies.

A

So case studies number one: you look at here, the down chunks, they are growing, but they are growing on only single part.

A

The input and the output rate are pretty stable. But then, if you look at the input rate by part there, there are kind of like two outliers, with significantly higher input rate than the rest.

A

uh The last panel uh on the bottom right there it's actually taken from another dashboard. I just shuffled it here and took a screenshot, is the cpu usage of the folder and similar to the input rate.

A

There are. There are two outliers and if you can see the forward part that has the highest cpu sets and the highest input rate, they are the same part as the one with the accumulating down chunks.

A

So what happened here uh so there are, there is a one up. There is one tenon that generated too many locks, and was it has a very few replicas though, so it only happened on one or two notes: uh it's it generated two money logs and it's holding resources. So as as a result, our our forwarder is kind of like struggling and failed to keep up.

A

Our mitigation is basically just add, annotation to the app and on bottom to the sidecar solution, and this is case study number two.

A

Now we see that the down junks are growing too, but they are growing across all parts.

A

Now, if you look at the input rate, there is a very big jump like around maybe 16 45, like maybe from 20k 30k to 800k, and then, if you look at the aggregator replicas panel, uh the the aggregator seems to try to scale up, uh but then it's scaled back down again uh and then, if you look at the crashing parts most of the forward bots are crashing. So this are actually. This is pretty, was pretty bad x uh incident.

A

So what happened here is that for some reason the applications like uh scaling up rapidly, uh while while the aggregator is adding replica and waiting for the bots to get ready, uh the existing replicas were overwhelmed and bots are crashing and your when your parts are crashing, the hp is not gonna, it was it's not going to work properly, so the mitigation is for us at around maybe 1845 there we manually scale up the aggregator and then you look at the down chunks. It's the forwarder starts to recover.

A

So what- and there are a couple fixes- one of the fixes- one of the fixes- is to make the hpa more aggressive, so the the aggregator will will scale up earlier, uh but this is just to alleviate. The problem doesn't actu, it's not actually bulletproof, uh because there is always a time delay from when the time the load starts to increase until all the new aggregated parts are ready.

A

So, theoretically we still can hit this problem, so we also make the second changes. uh The second change is to actually on board on board few applications that both very very chatty and they then- and they have the tendency to scale up rapidly to the sidecar solution uh and what are the key takeaways here? uh There is actually no perfect solution for everyone, so choose solution that works best for your environment.

A

We have the forwarder and aggregator pattern as our default login solution, but we also have the sitecar solution to handle the uh heavy heavy application so be. Creative.

A

Second point that I want to make is that kubernetes and fluent bit: they continue to progress and your production environment change over time.

A

So your solution must evolve and adapt um to the changes and kubernetes and also, for example, we we added this sidecar solution just about a year ago, and and also that we are exploring uh kind of like adding persistency to the aggregator on or also the idea of merging the uh forwarder and the aggregator.

A

So we only have a demon set without the aggregator and finally, last but not least, although kubernetes simplifies deploying containers in the cloud, it also adds infrastructure complexity so to having feasibility on the infrastructure components such a flow and bit and also choosing the best signal for your monitoring is, is crucial and fundamental in production.

A

With that I'll open up for a question.

A

So we have about maybe six minutes per question.

A

A

I couldn't hear you.

A

Finally, we have a question hello.

B

Sorry I wanted to ask you, said you're currently using the forwarder aggregator approach, but you're switching to demon sets.

A

Not switching, we are exploring actually kind of like mercy, forwarder and aggregator.

A

The aggregator is it's. If you merge that together is it's expensive on the note right, because all then your note then will have to uh do all the business logics that aggregator does so it takes resources from the node, uh so we're still exploring it nice. Is that your question? Do I answer your question yeah.

A

A

But I also want to mention that the uh the aggregator, although it is expensive, I think it it works in certain scenario uh like like when you're like for us in the second uh second case study, because when you have the logging cycle inside your port together with your application, if your application scale up you're logging, it you're logging, your login agent, also scale up together at the same time. So I think that's the advantage, but it's it is expensive.

A

Okay, uh thanks for attending my session, if you have any further questions, feel free to talk to me afterwards I'll be around today. Thank you. Thank you.