Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2019 (Barcelona), 22 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Intro: Fluentd - Masahiro Nakagawa, ARM Treasure Data

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Intro: Fluentd - Masahiro Nakagawa, ARM Treasure Data

Logging for cloud-native applications and environments is a continuous challenge from an operational perspective. Fluentd offers a full logging layer than can be accommodated and extended as required to solve any logging need. In this Fluentd session, you will learn about its administration and log processing from a general perspective.

https://sched.co/MPhm

A

Okay, can you hear me, okay, okay and so coming into my system, so decision is our friendly pollutant. Also I will talk about the fan to the basis for our entry new users. We friendly project as a one more throat- or this could be point you so there are tomorrow. Yeah Eduardo will talk about the tea plantation. The plantation will talk about our friend to beat. So if you are interested during the event, a bit project, please join to the tomorrow system.

A

In this system, I took basis, so I talked about the friendlies overview and architecture and I use kiss as containers integration. The first I talk about stuff entity overview.

A

Country fatigue, a designed to foster leading that correction. This means a friendly continues to lead. Overly ship data from the data focus sends data to the any destination in Australia manner, so, unlike botch, Madera, so like transfer with SCP so and the country is a consist of the core on the problem. Part of entities core provides important features for the data correction. The fertility probe's covers actual systems, so call api or write data to the databases and what we is a little in Ruby and see. So.

A

The home of intensive paths is between CC or the data data station or network wondering and with little a core or province between Ruby. We use all three gems for the problem. Distributions regimes is a standard library path home for Ruby, so we can use the whole Ruby's ecosystem or develop things of friendly proteins.

A

This is our fourth preferred to redevelop. The women's friendly project provides a similar approach for the installations friendly duration, so one it's always using the OS package manager right like yeah and aft, so friendly project provides the packages for the open to Arabia, and the RPM package holder, led to hot on the scent oasis on the ferry project, also provides MSI packages for entry. So you can easy to install the friendly and Lobby on the you at windows Saba. So you, camera friendly ons, are glad to serve windows service together.

A

This is one example of for immediate Corrections. This model is using the Intel, probably said to destinations with several problems over the MongoDB, probably in device a problem, so in this model currently is interpreted. Local files. Same application rates are locus to the local files. Fred with intel probably reads: the data immediately anderson de souza destination in this model is a local server, so this model has a married with a low rating sheet to inter observer.

A

So you can see the live data quickly on your data generation, food related databases, so stream data collection is now important because currently the protocol using the air time arises, but but geologists and the combination, so you need to get to the data immediately when drug offenders are generated so freely fit. Sergeants made a platform architecture.

A

Under 50, Isis affinity has a problem, so fatigue has a private architecture, so we can be to the I unified regulation without 20 frequently becomes a grew into a data pipeline. So we can correct our I needed any data. You can refer to data from any the forces under send data to any destination with the province. So you need you don't need to write any code for the. You are destinations.

A

This is our overview elephant released, open to its platform model and how to distribute in the province. So next topic is a currently at picture. So in this topic, I will show that how country works, I.

A

Said operative consists of a core part under the probe impasse. This this table shows details of each eight paragraph, so twenties Park covers common consent of the data collection, so, for example, who entities coop provides the portfolio under literal mechanism of Latifa correction, so you're, probably don't we need to write each by firing a little mechanism. So this is this simplifies your protein code, so horrible. If problems arise, is an error for the network area or something friendlies automatically litter eyes.

A

The errors until Santa succeeded and the free school provides a similar help ask for the friendly, probably develop path, for example, long till a network server or timers, or creating paths or more so you probably developers hope acids on the quality over the proteins. So this is quite friendlies. Had a lot of province may be near thousand proteins in the wider community.

A

So when the proper focuses on how to read data from the API API or how to write data to the your databases, the fridge.

A

This is in separately so friendly event. This is a structure, Florida heat, one events, rental. The event consists of the time and the type of Records time is now second delete. The disease comes from the our drug generated arose under countries event as a tug caused the event eluting, and this toggle uses of order identified of these are forces. So you can know the where this data culled from in the using distant and the very important point is our report.

A

So free winter, liquid is not raw story, the friendlies combative into the low string into the JSON object and in problem inside. So this is because JSON object is easy to mutate or a little filter or easy to transform to the other home at recently, so middleware or web services can accept over JSON Oracle database or elasticsearch or a device or DCP services. So this is why friendly uses Jeff level, this object or event to record.

A

This is a data pipeline of the 3ds infinity internal data pipelines and fifties right. Data pipeline is consist of our problems and the start is imposed at the filter bar fund outputs, so incoming the data is a pass-through from left to right. So if the problem generate even offer to the event under, you can apply the filter and the data are solid buffer and preservatives our output.

A

And I will talk about each components, the interpreting, so it probably is a starter points of the federal data pipeline, so in probably receive the data or little data from the leaf forces, for example, Kafka or HTTP API or local files, and the friendlies improperly parties are logs into the combat into this in its structure. So.

A

After after province Philippines, it can assume that is even two structures of not pro story.

A

The filter problem- this is a very simple so because this is a whole filter or mediator robes. So, for example, if we want to add the forum to the event reports, you can easily hold the transformer and over if we want to check the condition to ignore unnecessary rules. For example, Oliveira was something so you can it's a great filter to avoid it, of course, few ties that could be change, so you can.

A

You can operate a much prefer us to your data streams, so adding equal to transform and filter to only are to same data streams.

A

Next is the buffer, so buffer protein is not standard on protein. This is the user inside loud to Praveen, so internally, with the data and Emma to the data estado into the buffers before right up to the destination. This is because the improve is the throughput or improve the reliability for the litterin buffer problem. So this is the perfect remains a core of the low bustin running on Twitter I knew holder, gated rush and Federico ah provides that to buffer types.

A

If you use the file buffer incoming data as protons opposition to disk, this is avoids that are lost of the data and even friends, effectively crushes. So we recommend to use the file buffer on the production. Every River fights for mainly mainly for moisture meter, expand, Netflix or some from no important data.

A

Internally, so this is a continue to buffer proteins, so internally, with this buffer is a cube chunks. So this is this: is the three three chunks so buffer have some options in the buffer Praveen, the child has lots of L'ecole waiver up to ten thousand or a million or something, and we can configure this size so chunk size or frustration table. So these parameters so present our world, but size of the chunk is effect to the our ratings yield of throughput.

A

So if you want to a low rating sheet for the data transfer, you need to set to the smaller Frostine double ins up out of the proteins, and he wanted a lot. If we want to the throughput right, the s improving or very probably you can. You need to set a large large, Eva's interval or larger chunk of chunk size.

A

This is very important, for the fruit leaves home.

A

Not clubbing so input on our point, importance, buffering or Detroit is a done by the buffer protein. So Auto provide very simple, so just to write is a buffer chance without to destination, so Wesley plugging a three or a starch, or something.

A

Now this is an excellence is how to eat rice. The fifty Isis, the fir-tree is a streaming model, so it's streaming model is made to hold the try and literally on the rescue. So friendly is still a treat to the data to limit into the many chunks, many small chunks, so this means friendly little right, cheaper, because no elevated, large data, so under this small chunks is it can easy to process that concurrently.

A

So we can write the multiple chunks at the same time, so it also improves the latency, because we're hiding the artists are writing data at the same time. So no need to wait previous chunks to frases accomplished complete.

A

This is why friendly uses of timing model are the affinity fits the data into the small chunks.

A

The Brittany has to province, or so this is our welcoming by committee province, so of Capri or services or more.

A

The other problem is also has a mood, are lots of problems because there are lots of services or databases in the world, so not so I see you're in fact, DB or morals.

A

I said I just talked about the picture, so next is I use cases. If it's a configuration example.

A

Hey this is a very simple. This is very simple: our use cases so friendly correct the data under local files. The digital data from the application friendly is in depth in the inning data to the earth. To this initial, or this example is a MongoDB.

A

This is the conquest example, so this is a I'm sure this is a familiar with you or not. So this is a very similar to the apart, like Apache, like format in its compression force directive, I was a pro probably multi detective. As for the flag, winds and much much the active as a pattern in argument, so we can use this folder or this pattern for event. Looting with touch I see the factories amino structure has time for a bit blue team. So this confusion uses a terrific start.

A

Captures deserve this in developing and it's settle to the MongoDB and trade open probability. Plotted was event, so this is a plot apart. So this approach tab is a little too demography output.

A

And the Pinta deal also supported are much more destinations to our metadata. This is very simple. Proper, simple configuration so friendly has to be probably in the periodic deterrent province. So with copy. Probably we can send data to the brought to destinations in the same date with the same data. So in this computation data is such an elaborate 80 degrees.

A

The abbreviated effects, flag, wins, oh here's a ho today, Tahoe direct such and you can Hadoop over something who the data and if we want to add the more storage like the Kafka or more, you can use just a disaster detective into the copies of ravines. So this is very easy for adding the new destination.

A

The country also has a complex data Wadi. So this is a merciful. Our gt-r, who are the sofa tree, has a original forwarding protocol it's to given to delivery semantics. So what most of us, or at least ones and the faculty, has a TA or a load balancing or the multiple entity aggregators, so you can use you can processes the more traffic with this multimodal and I want to talk about this model, but this has pros and cons, and we on the 0 several lots of discussion point.

A

So this is hard to talk about in this show. So if you, if you are interested in that this model, please check the might write to that before previous repair. Previous to the cone.

A

This is a lot. This is largely prevent, so many a lots of large company you that, if HTML for thousands who others ten or hundred thought the Gators.

A

He, the is Cassidy the continent abilities so because this is a good episode. I talked about this case.

A

Country is a now widely used in production with the containers, so, for example, by the electricity's currently is a host product on running on the dope car. So friendly is very running on the mini companies and minidisc systems and I will talk about the how to work to data from the doctor containers or occupy discontinuity seas, environments.

A

Before are before how to collect data, I mean I wish those resources holes are how to use a friendly on the your container. What's environments in a container friendly project provider. President, okay, images for the that we are your container environments so on the 50 projects, provides the RP and otbi images. Both our peoples of small containers and the IBM is for products or environments to be recommended to use the API image for production, because these divya images works with the system D and J Morrow on the high traffic environment.

A

Sometimes memory consumption is a problem and we use the J mark to mitigate the definitely program, so I really committed to use the tibia images. I want to share for RP images, doesn't work with the J mark, smoothie, so apparently Jamie's off on developing images.

A

Alt top image is now official images. So this image learn on the inteiro MV or are more efficient or more so. You can live literally on the road top a lot of environments.

A

The kubernetes III project also provides the demo set image other users, so this is only the via images, because I say those are so RP mates has a similar program on memory consumption. So quite a similar map is only only tibia. Images on this images has several beautiful destination, so you can use the popular. The first data services is that this images, so elastic search or Africa and gray roll card abuse drive a lot more.

A

But this is this: image is not the culture images, so this means lands on only X, sub, n, bar x86 environment and home chat is also provides stable images for the country. So you can also use fancy with ahem packaged.

A

I talked about the how to collect data from the containers the first. This is a basic approach, dr. Hoffler, looking adorable mechanism to send the application data to external systems for examples of JSON, Joker, fire or sister or Morris Luka supports with reading the driver, it's a by default, so you can use this same view driver for your collecting that you are application ropes and this is our options- are linking tribal, so Robo driver asked affinity.

A

Robot you can specialize off into the address address was a quick I rented instances and we few photos are employed, probably in the friendly configuration you can receive data from this the car instances. So this is very easy because they, you don't need or to register our support in your application, but I said so. This is aa specific approach, so this is we reach and we come to this approach on the caballo. So as a countenance.

A

And second approach is using the relatively low girls, so frontier yoga is a client library over for the friend OD. So this approach is mainly for collecting the data metrics or application specific data.

A

So we see this approach, so you can see that this also, you need to add, was a login code in your application, but you can collect any data of the applications. So this is the merit of this approach.

A

Friendly is yoga, it's implemented in the many languages, so Ruby, Python, Java or C or last or go muah. So I assumes that you are application, easy to use the Buddhists in raga for collecting data. With this approach, the pre 3 5 is a same odds friendly doctor engine driver because this is Fantino 20 log is used to in the top column a doctor in the tribal. So this is the same protocol. So UK is the same filtration cosa, rowing driver and raga approaches.

A

Next, it's a shared data point approach: this is a menu Hotel, middleware images, so sometimes we can't modify our load. You mekinese almost equally single table car images, for example, if you, if we were either way our generator logs to the local files, we can't access these this local files outside of the container so hold dot cases. So we can use the share data volume and we can use the Intel protein to collect data from the solution volumes, roper files.

A

Mountains are MLT, Ruta and free access to, though it's wrong.

A

Now this is a very early approach, but the kubernetes yeah. We use the similar approach because Copernicus doesn't provide any API folder for it, access loads from the application container and on kubernetes coupon its generates the rows into the borrow containers directory, so in this case, friendly user Intel, probably in wildcard pattern with this configuration really automatically expands. Our directories directories files and the leedle leedle fund on the log files and immediately understand the data to the these nations.

A

So I said those are friendly communities demo set uses this same configuration to data from that you are various containers.

A

And what demo set this is our provided. Images also also included as a metadata filter for automatic environments, so this filter at the kubernetes metadata to the data streams. So this is an example for namespace or a container name, but the name or its filter of this metadata to the path. The data is a path through this example. So we can, we can use that is metadata for you, our data process inside so group in the data.

A

Erotica was something.

A

So this is a summary of container container working approach. So, if we're using the top Vinayaka emphasis, you can use the logging driver mechanism with a friendly driver. This is very easy.

A

If we, if we can do studies the drive approach, why he wants to correct the complex data was a metrics. You came is the friend Drogo approaches with additional report refused similar way of you can't modify the argue mechanism and you can share data poems and on communities. You get a similar approach, so correct font arose from the borrow containers with Intel Prague wins and, of course you can use the files metadata pilita to add to Quercus api's based metadata to the locus. So this is NDG that your data is a various metadata.

A

Now this is our last chapter of the rental of cap table this session and the flat width. So it is a input of echo system, most affluent the project.

A

This is a simple composition friendly and the front of it so I, just a friendly is written into B and C, so C C part is for this elevation on data lab hungry, but also sometimes we throw but wait whit is who really fully implemented into C. So this is a 50 B trees, a very fast and lowest points.

A

So an English young friend bit is also has a pluggable mechanism, so you can add the you a destination or input. Probably are the sources into the thread bit with the problem? Api so currently currently is targeted for the whole other aggregator, but fit, which is a hope, a season's for other eighty use cases.

A

So maybe this pointer is a total of a trailer. There is a tomorrow so currently so with container working with the friendly under Fitbit. So friend, with its you week, you can use the Fitbit on the whole other side to reduce the newly usage or the CPU usage and went to the retreat. Also has other protocol so friend of it careful what they talk to the twenties. So you can use the fifthly from the aggregator site to freak shiver or robust, robust suffering and data hoarding under UK mr.

A

little bit on the whole other side to the minimum with minimal resources. So this is a very popular approach in the current. Currently this popular approach on the container logging with a friendly project this this net week in Colorado and tomorrow tomorrow tip session, talk about the deeply. So please join us tomorrow: system, okay and you're loading thanks.

A

B

C

B

In terms of collector, what do you think that the benefits of fluent bit versus file bit yeah, you.

A

Means rustic, yeah, yeah, I'm sure the details compiled John up to I hear the value dissociative of the file bit to the recipe for the private, sometimes CPUs consumed with a lots of CPUs and a lots of memory. It's on the plane to beat sofa tweet is written in C and variability written go. So sometimes sometimes we go how to to manage the memory management in go because I go in GC base, though language, so high-performance environments sometimes 2006 met with it. If it turns out lower with lower lower CPU and memory resources, I think.

C

Hi first, thank you very much for the talk. As far as I understood, you have a son like input, plugins, okay, one of them as far as I remember was the HTTP. Is that correct? So so my question is: is there any output plug-in for HTTP or.

A

C

Not is any plan to to have it so.

A

You want to even you want to the output, ultimately probably in the core, so after country, so it doesn't have that attraction is all so currently we have a prompt order of hte problem in the core, but we are now concerning the replacement to the networking start with new new library. So after that we can support the HTTP through or efficient or HTTP access. So after that we have front order to operate really, probably in the core.

A

A

A