Cloud Native Computing Foundation FluentCon: Cloud Native Logging Day with Fluent Bit and Fluentd EU 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lightning Talk: Fluentd as syslog drain for CloudFoundry - Karsten Schnitter, SAP SE

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Lightning Talk: Fluentd as syslog drain for CloudFoundry - Karsten Schnitter, SAP SE

CloudFoundry can ship logs of deployed applications to user provided syslog drains. This talk will show, how to configure Fluentd to receive logs messages via syslog. Furthermore, parsing the log messages will be explained with special attention on the structured metadata from CloudFoundry and parsing of the access logs generated by the gorouter. This results in a basic configuration, that can be enriched by custom parsing rules for the log messages.

A

Hi, my name is carsten schnitter. I work as a development expert at sap business technology platform. My team provides logging services for applications running on that platform. Today, I want to share our experience with fluency when providing logging for cloud foundry applications.

A

Let's have a short look. How logging in cloud foundry works? An application can write its logs to standard out and standard error. Those logs are collected by the cloud foundry lobby gator system running in the container runtime it can forward the application locks, as well as the access locks coming from the go router via syslog.

A

Any component that receives such a syslog stream is called a syslog tray. I want today to show you how to set up fluentd as such resistor tray. That means I will explain how to register a system training cloud foundry in general and then how to configure fluency as such as the stock endpoint, how to pass the application logs and finally, how to pass the access logs from the go router.

A

So, let's start with configuration of the system tray, we assume that fluid d is already running and we know the url to fluid d. What we now need is the service binding to forward the logs that can be a service binding to a user-provided service which we will use here or any other managed service.

A

So the steps for user provided service are first register and create a user-provided service and then bind our application to that service instance. So, let's have a look at an example. We say: fluentd is running at the url fluentd.example.com and we also have an application running on cloud foundry by the name of my app. We will create a service with the name fluentd logs, for the forwarding we can choose between three protocols. The first one we will use is plain syslog and there are more secure options with tls support for syslog or syslog over https.

A

So let's use the cfcli commands for this thing. So first we create a user provided service with the name, fluid minus logs and then with the minus l switch. We give it the syslogtrain url that points to our running fluency instance.

A

Then we bind our app maya to the newly created service instance fluentd logs and that's all. There is a word of caution. You need to secure the fluency instance. You are binding here against unwanted excess.

A

So what do we see at the fluent design? Incoming messages will have the following format and you will get them by configuring, a source type syslog. The message format is ifc5424 and the transport will usually be tcp depending on the cloud foundry configuration can also be udp. So let's have a look at the parts of this message. The first is the priority that is default to info by cloud foundry and next thing is the ingestion timestamp.

A

That means the time stem when the cloud fountain logic data received the message, and then there is information on the origin of the message. So this is the awk name, the space name and the apnea. Unfortunately, all three kind of information a little bit unhelpful. If you really want to distinguish the log messages, the first really useful thing is the app id. I use the placeholder here for the uuid. That is usually there, and the next thing is the source type.

A

I will explain about the source type later on, so all of this you get just from the sister configuration to the left, but, as you can see, there are more parts. The first part is the so-called structured syslog data, and this contains all the cf metadata to attribute the logs to its source. I will explain how to pass this part in the next slide, and the final part is the log message itself and, depending on the source type, there can be different formats in the log message, so it can be the application log messages.

A

It can be the assess logs from the go router or it can be run time. Events from cloud foundry- and I will explain that one as well, so let's have a look on how to pass the metadata, so the extra data field will look. Something like is what you see below.

A

This contains a lot of source identification.

A

So it will tell you the app: the message originated from the space the app was deployed in and the order the space belongs to and also if the application is running in multiple instances, you can distinguish the running instances by the instance id also contained in this metadata, and then there is more information, and this is information about the cloud foundry runtime. So you will get to know what the ego cell the deployment was running on what the ip of the cell was and such things so to pass the extra data as below.

A

We apply a three-step process. The first is the pre-processing that just removes the brackets at the beginning. At the end of the content, then we pass the metadata fields you and for this we use the labeled tab, separated value filter, because then we don't need to know how the fields are called and finally, there's a post-processing step, you will notice that cloud foundry quotes all the values and that one we want to remove so in configuration. This looks as follows: removing the brackets is a small ruby code. Snippet.

A

Then you see the configuration for the lts me parsing rule, so also very basic and finally, in the post-processing step, when we remove the quotes, this again is done with a small, ruby expression, but here you need to know the name of the fields, so this is no longer dynamic, but still you specify all the fields you want to have, so you don't need to take all just the ones that are interesting to you.

A

So now we have the metadata. We know where the message came from, let's see for the message itself, so there are different message formats, depending on the source type. The first part everything that starts with app slash. These are messages that come from the application itself, it's jobs or the sidecast that are deployed by the user by the application developer, and this is user generated format, and here is where you put your custom routes depending on the application you are looking at.

A

Then there is rtr that stands for go router, so these are the access logs. This is plain text and it's basically an extended apache log format, and I will show how to pass this with the general reg x pattern.

A

And finally, there are a lot of source types like api staging loggregator, ssh and cell, and these are runtime events from cloud foundry. They are in plain text and we leave them just as is so. If you have this kind of different formats, there are two approaches you can take.

A

The first one is to use the multi-format parser plugin and that's the one I will showcase and the second one is to really switch the parsing by the source time. So how would this configuration look like so here you see the multi-format parser in action for the application. I assume it would write a json format. So that's the first part. The second part is the go router logs.

A

I left the expression out here because it's rather lengthy and I will show it on the next slide and the last one is just the format, none for all the then runtime events. Okay, now how do we pass the router logs? Of course this is an assess log. So it has a lot of fields.

A

The first fields are fixed in format and they are very similar to apache logs. So you get for every request: the host, the user, the method and the protocol, and also requests and response sizes and such things and then there is an additional optional fields in and they depend on the go router configuration, so they may differ from different cloud foundry installations, so there might be http headers, for instance, for tracing like zipkin or open tracing. There might be response times or again, some metadata about the application.

A

So if you want to pass such a router lock, the expression that I will give now needs to take care of this. So again, we can use two steps. The first is to pass the fixed format you find the expression down below.

A

So you see it is rather lengthy, but it's just taken from the apache two filter and extended a little bit to cover more fields and also to take care of the quotes that cloud foundry once again emits and once you have this, you notice at the very end, there is an additional data field and that will contain all the optional periods, and this is the second step you can pass the optional fields, just as we pass the cf metadata, it's basically again key value pairs, but this is now a very cloud.

A

Foundry configuration specific thing, so this needs to be tailored to your landscape. So summarizing all of this cloud, foundry can forward lots of applications via syslog brains.

A

Fluently can easily receive those messages as a the syslogtrain, and if you want to pass the incoming messages, you will require an advanced ruleset to first pass the syslog part and then take care of the metadata and the log messages and the three kind of formats of log messages you will get. So I hope this presentation is interesting to you and thank you for your intention.