Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2017 (Berlin), 6 Apr 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Keynote: Fluentd Project Update - Eduardo Silva, Treasure Data

Description

Keynote: Fluentd Project Update - Eduardo Silva, Treasure Data

We are working towards Fluentd v1.0, the ecosystem is rapidly evolving and new exciting tools are coming in as part of the Fluent ecosystem. Very exciting news is coming!

Eduardo Silva
Eduardo Silva is a principal Open Source Engineer at Treasure Data Inc. It currently leads the efforts to make logging ecosystem more friendly in Containers and Cloud Environments. He is deep involved in Fluentd and Fluent Bit projects among others such as Monkey HTTP Server and Duda I/O. He is an active speaker at international conferences like LinuxCon and CloudNativeCon events.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Without further further ado, please welcome to the stage Eduardo Silva, representing the fluency projects.

A

Hello: everyone thanks for coming this day and my name is Eduardo Silva I work with the fluency team at pressure data, and now we have five minutes to give you an update about the project.

A

What's going on, why it works and whether what is the future for the project, but I cannot start talking about about zoondy without talk about login, when you think about so Indy, you need to think about just one word: login login exist because at some point you would like to analyze how your applications are behaving and in order to accomplish that, we need to figure out how to get an information from the applications, because we want to analyze that kind of data and the usual way to work with.

A

This is like we make our applications to generate some messages and saying hold. It is working if we have some error. If we get some warnings about something and most of this information is stored on logs logs can be any kind of a file in the file system and also a log can be sent over the network, so the login mechanism that exists nowadays.

A

There are many, but if we want to make some analysis over those logs there's a full process that we need to go through it, because we want to take these logs and insert these logs into a centralized storage, and that is not so simple. So how do we make this happens? How do we take different logs from different applications from different nodes and centralize them to perform our own? A analysis for this?

A

We have a login pipeline, which means that if you want to analyze all this data, you need to take the data from any input. You need to parse the data you need to filter the data and then buffer the data, because you doma, you don't want to loss it, and then you want to send this information back to a storage. So if you want to analyze your logs from your applications from your own clusters, you need to go through a login pipeline, and that is why I flew India is a full solution for logging.

A

That allows you to take your any kind of log information process, this information and set into any kind of storage.

A

So now that we get some general overview about flu, Andy I will I'm very excited to talk about what new sends a our inception after the CN CF, but right now the flu NT team, a main developers, one here with you. They build multi-process support. They build built-in security for log forwarding. Sometimes you need to forward the messages from one point to other in a secure way: plugging helpers for developers, the community is quite huge.

A

You will see right now and improve it tested testing across the whole platform and right now, flu Indy is not just one service. It's not just one program. Flu Indy is a full login ecosystem. Think about that we have more than 600 plugins last night. I just did account about how many plugins do we have, and there were seven had with something so I just thought that a hundred they're kind of obsolete or not maintainer or maybe not working and most of this work comes from the community.

A

I will say that the main core developers of Fundy may be do they maintain 20 plugins. But what about the others and, of course a when we started flu indeed years ago? We wanted to make it work with all applications. So if you write your own application single language in Python, you have your own, a connectors for flu in D, so you can send your application logs to flu in D, and now flu in D also runs on Windows. That's unexpected! For most of people.

A

That's here, I see a lot of smiles, but you can imagine that Windows is a serious system for many people.

A

Now it is, it is a it was honestly. It was hard for me to understand it until until I was involved in a project where they were doing some game development and gave development happiness on Windows, no you're quiet. Now you get it, you know, and also we have many windows services running any ATM in the streets. They runs Windows and you need to collect the logs windows even log system, your applications running on windows. Maybe people don't like it, but does it true?

A

So we were on top of these trying to see how we can scale not just a service but also cross-platform, and also we talked about cross flat platforms. We need to talk about cross system, curved clusters and in terms of docker a recently, the team come up with new docker images for full Indy for people have works with docker and the last Debian images has more than 3 million pools, and this image is quite new.

A

I will say that in the has no more than two months, so this is a really really good stats and the fluid ecosystem also jump into kubernetes and for kubernetes. We're trying to make login simple and easy, as I said at the beginning, is not like that. It's pretty hard, but we trying to make this happen. How creating a new diamond set. That's the way to distribute fluently in a kubernetes ecosystem, which has the generic a system unplugs for different backends, for example, cloud watch, elasticsearch loudly and fluent ecosystem is growing.

A

We think we think a lot about performance. People say that some time had performance issues because they had too many logs and, of course, you had a lot of computing time, so how we can improve that. So we created a new log forward which is called fluent bit, which is fully written in c language. It has a really small memory footprint and can do most of the air lock. Forwarders features that you can expect like fill a collect data area, filter the data parse the data and send that data to some back-end.

A

So how do we make this happen? We have looked for words and log aggregators Phoolan D is both, so we can. We say that we can make a login cheaper if we make a log forward more lightweight, and the fluent D right now is that we have two main branches. The one is the seer that 12, which is the long-term super very own, super stable, and we have the student 14. That's like the next generation of Lundie. That will be the base for fluent e 1.0.

A

If you want to try su that 14 under your own risk and I cannot finish this session. These five minutes without embed you to everybody to the fool in this salon, which will be on Thursday at 11:30 a.m. is not too early and just before lunch, which is perfect. So please come with us and we have the full court developers of Lundie. We will have sessions about filling the water zero, the internals of Lewin D about full impede, and we would like to have you there to have some networking and interchange our interests well.

A