Cloud Native Computing Foundation FluentCon: Cloud Native Logging Day with Fluent Bit and Fluentd EU 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lightning Talk: Parsing CRI JSON logs with Fluent Bit - Bart Robertson & Joseph Fultz, Microsoft

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Lightning Talk: Parsing CRI JSON logs with Fluent Bit - Bart Robertson & Joseph Fultz, Microsoft

With dockerd deprecated as a Kubernetes container runtime, we moved to containerd. After the change, our Fluent Bit logging didn't parse our JSON logs correctly. containerd and CRI-O use the CRI Log format which is slightly different and requires additional parsing to parse JSON application logs. This session will detail those differences and present an end-to-end example on how we resolved the issues.

A

Good morning, I'm bart robertson, I'm a software engineering manager at microsoft and we're here today to talk about the log changes between using the docker sim and using container d or cryo as your kubernetes runtime.

B

And I'm joseph fultz, I'm one of the engineering leads at microsoft and I'm going to open with talking a little bit about our repo you'll, see it posted up there. You find it with a short link of aka dot. Ms slash fb cri.

B

In that repo. You can find uh some examples and a walkthrough of what we've done. uh The the problem is that when we switched from container from docker runtime to container d, the log formats didn't match up, and so we had to do a little work there and you'll see all that in this repo, as well as some additional mappings, where we lift values out of kubernetes and kubernetes labels.

B

So with that being said, let's take a look at our setup here on the left. We've got a docker runtime running you see at the in the terminal window. There there's a describe and you can see it's the docker runtime version and in the top window you see that we've got the parsers app as docker the configuration for it's pretty bland, there's nothing special there, no regex or anything like that. So, let's take a look at how that log output looks as it did initially prior to us changing to container d as a runtime.

B

So if you look at our app, this is the log output from the app you can see. There's a few fields, there date status codes on so forth and we want to see how that maps out without any any other special mappings from the docker runtime with the docker parser.

B

So when that gets picked up by flintbit, you can see there at the top line. There's the container log and in fact, there's a log field there to which our information is mapped. You can see it as a string with the different fields differentiated there and followed by that you see the stream output type, which is standard out, and then you see the kubernetes log information, pod name so and so forth. Map to there as well.

B

Now with that being said, I'm going to head over to bart to walk through what we've done, with on the cry parser side to get everything matched up and looking uh looking good before we add additional lifts and such mark awesome.

A

Thanks joseph so the first thing you'll see is that we're running container d um as our container run time and let's just go ahead and take a look at our logs, real quick and I actually have two instances of fluent bit running one that just runs the standard out of the box, exact same configuration that we have on the left side and the other one that runs uh that fixes it. And so the first thing that you'll see is that our log format is is a little bit different.

A

And so here we have a a log field, uh but you'll notice that, unlike over here where our log field just contains our json that was written by our app, we actually have additional fields in here, and so we have the the daytime stamp. We have the standard out. This is a full log which is this is the log tag field which is new in the cri format, and then we have our content and obviously this doesn't parse, and so our back-end parsing system broke because it wasn't getting what it was expecting.

A

And so let me show you how we fix that. It's a really easy fix and uh but just realize it does break your logs and so here's our new version that we have a little bit of change to and you'll notice that now it looks more like this one, so we have the stream just like we do over here.

A

We have this new feature called log tag and the f just tells you. This is a full log. So it's not a multi-line this. If it's a multi-line log, this will actually have a different value and then the logs will be written out on on multiple different lines.

A

One of the reasons we chose to log with json is to avoid the multi-line logs because it makes parsing so much easier and then here we're back to our log format, and you can see it looks exactly like it does over here and then here's our kubernetes map, and so uh it was really pretty easy to fix this. And so let me just walk you through the two changes that we had to make.

A

um So the first thing is, you will see these files are, are pretty much identical um with two exceptions, so uh this input section is uh is where we made our first change and over here you can see we're using our parser, that which is the docker parser, and this is probably how you have yours configured today, if you're using the dr shim or you're using dr d as the runtime, the change that we made is we actually use this cri parser, and this is a new parser that we added it's available on the fluent bit website.

A

And if you go down to the bottom, you'll see this parser is defined down here and all it is, is a reg x that pulls out that time, the stream, the log tag and the log value, and so those are the only two changes that we had to do to make this uh to make this work. The other thing you'll see is that we have our merge logs turned off right now.

A

If you turn these on, it will lift the individual log item to top level, uh and you can also lift the kubernetes to the top level as well. So it's a really simple change, but if you don't make this change, your log processing is most likely going to break because the cri log format is different than the docker format, and so with that. That concludes today's lightning talk and thanks for joining us thanks.

A