Calyptia Fluentd, 30 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Keynote: Accelerating Adoption of Fluentd/Fluent Bit With Confidence- Hisatomo Takana, ITOCHU

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Keynote: Accelerating Adoption of Fluentd/Fluent Bit With Confidence- Hisatomo Takana, ITOCHU Techno-Solutions (CTC) America, Inc.

Fluentd/Fluent Bit CNCF projects have been accessible to everyone as we embrace the community effort to bring vendor-neutral, unified, and robust approaches. Through our Fluentd community survey, we have discovered some challenges that community members have when using Fluentd/Fluent Bit.

In this session, we will discuss our approaches on how to reduce operational challenges. We have been contributing to "Diagtool", which is already available in td-agent, allowing you to automate troubleshooting. In terms of security, we developed the "Sanitizer" plugin which masks sensitive information on the fly and helps you to solve security incidents such as data breaches by making data meaningless for attackers.
We, ITOCHU Techno-Solutions America, Inc, are a part of the community to accelerate the adoption of Fluentd/Fluent Bit through commercial services and contributing plugins/tools for our community.

A

Hello welcome to the fluentcon north america. Thank you very much for joining this conference today very excited to have you and people connecting virtually as well.

A

I am hisa one of the fluent community members working with our colleagues and friends to accelerate adoption of fluenty and explore where the new adventure next adventure is to bring industry to the next level.

A

It is my privilege to share what we've been working and then discuss what we can work together as a community as a community.

A

I also run a city office of a company called ito to techno solutions, america, silicon valley, arm of a publicly traded global idea, solution. Provider people sometimes call us a ctc in this session. I will talk about our effort on some of the operation and the security challenges, as we've contributed a tool called diag tool, which is allowing you to automate some of your troubleshooting process and in terms of security-wise.

A

We contributed a plugin called sanitizer which mask sensitive information in your fluent d pipeline to bring another level of data security in your organization.

A

Okay, let's get started first of all a little bit about us. We are one of the I.t solution providers in california, driving technology enablement around hyperscale and cloud native technology, such as open compute projects initiated by facebook, allowing people to operate infrastructure at the scale addressed by data center hardware, designs and cloud native patterns of operational practices made available from companies who run the application. The infrastructures at scale and driven by cncf.

A

Many of you already know, but fluency is, I would say, a defect standard, open source software for your very first mile towards the entire observability journey, which provides a unified layer for logging and now supports metrics. It is a graduated cns project.

A

There are more than 7k plus contributors across 1k plus companies and the community is still growing.

A

Well, we see why people start using fluency and fluent bit or where people think logs to be distributed and used in multiple tools. Around observability data generated by machines are not only for application infrastructure monitoring, but for more evolving use. Cases such as a machine learning utilizing the data across different applications and security tools more instantly and storing for a long period of time for compliance or sometimes a forensic purpose, also because fluency has out-of-box capabilities around data processing such as processing and filtering more people.

A

Look at this as a tool to manage data volume by removing fields and records. However, although those are very important I would say, operation is much harder. Job and people spend more time than just selecting a new software.

A

Matter of fact, there are the some of the things that we hear more from our friends and colleagues in the community.

A

Scalability is where we see there are more considerations and practices need to be incorporated to run fluentd at scale tk. Who is my colleague we'll talk more about it in the afternoon?

A

Configuration is also another element that does not only matter at the initial implementation, but how you would maintain and make sure how properly fluenty is running in your infrastructure is another topic which our friends in caleptia talk today.

A

Now, let's talk about diag tool. First, through the course of activities, we've identified a few typical problems within a troubleshooting process. First, one is a configuration error. One example is a file descriptor in a corner parameter which needs to be increased.

A

Second, is collect the necessary logs itself. Lastly, information over a specific system such as fujidn ip addresses name of cos, applications in your fluent logs or logs that you're delivering to destinations shouldn't be exposed for troubleshooting process.

A

There are what we addressed. Those are what you addressed in the with the diag tool.

A

Diag tool is a tool to simplify troubleshooting process and make it more secure. This tool delivers the functions you need to automate your manual checking and the data collection processes in order to accelerate the troubleshooting activities. The tool does three things.

A

First, one is collecting parameters configuration and logs. Then validation allows these parameters to be checked if those are configured properly. Lastly, masking sensitive information such as fqdn ip addresses, and you can also specify your own keywords by using regex before sending those configuration logs to experts for troubleshooting.

A

Another area that we addressed was security.

A

Huge amount of data has been collected from machines. Collected data is consumed for several purposes. However, not all operational security members do not have to see raw privacy data or confidential data within the collected logs.

A

That's where the idea of sanitizer plugin came from and this tool that masks specified data in a fluency pipeline as one of the filter plugins.

A

This will allow you to maintain data security and privacy policy within your organization and potentially prevent a risk of data breach.

A

This slide shows you how sanitizer plug-in can be utilized for those use cases where we need raw privacy. Data need to be masked.

A

This plugin does it within a fluency pipeline and if you would like to store all privacy raw privacy, data for compliance or forensic purpose, sanitizer will give a flexibility to allow you to configure what destinations you would like to turn the masking on and what destination you like to send the raw raw data as well.

A

Now, let's take a look at how the how this works in motion, I showed a video that demonstrates the sanitizer plugging, so let me play with it now. Let's first see the example is to mask ip and hostname.

A

Let's first see the config file.

A

Okay, all right and then you can now specify find key. Ip and host rules are configured with patterns. These are preset masking rules, so you don't have to write a regex.

A

Right here and then.

A

Let's see we send the logs there.

A

This case we are using elastic here. You will see data already masked.

A

And now let's go to a regex. Second, I'm utilizing logics.

A

A

Configuration here this case is utilizing a phone number as an example plus social security, number type of format, data, and then we configure it writing regex there and then send your data to elasticsearch.

A

And then you see those logs to be sent.

A

Data in elastic are masked.

A

So that's basically the uh how sanitizer works, as we already embracing the community effort to bring this vendor-neutral unified and robust tools and community to be accessible to everyone.

A

We would like to also make sure your journey of fluent different bit has maximized value for your operation and even by working together, we will have many people, an organization to bring it was observability journey to the next stage.

A

Also to make your production use to be very confident with the confidence we have a commercial services around full and difficult bit, which is called fluently subsequent network. Please contact us if you need any help.

A

Well, this is uh for my presentation. uh If you have any questions or comments I'll be around and then hope you enjoy the rest of the conference and have a great day. Thank you.