Cloud Native Computing Foundation KubeCon + CloudNativeCon China 2018 (Shanghai), 21 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Intro: Fluentd - Eduardo Silva & Masahiro Nakagawa, Treasure Data

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Intro: Fluentd - Eduardo Silva & Masahiro Nakagawa, Treasure Data

Fluentd and its ecosystem logging projects are a fundamental and critical piece for observability in Cloud Native environments. On this introduction session, you will learn the basics of Logging, the pipeline applied to Kubernetes and overall features associated with filtering, buffering and log centralization/aggregation in general.

To learn more click here: https://sched.co/FuL8

A

So for the rate, so I can't offer those tried to lie in it site. So this is a my travel. Sorry, okay I started my system. Okay, coming thanks for covering this my sessions. This is showing talks about the friendly project introduction. So, in this conference we have a to say shows all the friendly projects. So one is this front three project to introduction and the second is a tomorrow's deep devastation.

A

The PlayStation it's this Eduardo will talk about the plan to beat holes are pretty bit for introduction. So my talk is a read: 2d, not red meat, hey red talk for my system, so in fast I talk about the overview of the friendly, so that concept of the friendly okay. This is a list of the friendly contexts. So what 20 so 20?

A

It's a design to full streaming data collection, yeah, not too much money, so it means a friendly continues to Lidl office rooms, are any data sources and continue to continue to write data to the base station, for example, pourings HTTP no points and the correct data understand Atossa for F iterator to tiny data stores, for example, s3 or our DBMS or other local file over files.

A

So this is inaudible, continue, continue, continue processing, so the friendly is consists of our core on the problem path. So friendly is Cora. Our provides to our input of the futures for the data correction. The province covers are used, retrieve, use of the use cases, for example, reading data or writing data, so core provides are important feature so probably focuses on the quality of the leading data or at writing data, and the friendly is our little in do B and C. So are you heavy or other cpu? Heavy pots?

A

Is the vitamin C the fatigue, poor soil to prevent or Allah Klopas is between Ruby. So this is for flexibility. This Ruby is for flexibility or the C is suppose our horse and lava tree uses our regimes of our problem distribution. So we can use the whole ecosystem of the lighting, or this will be a probably distribution.

A

Under definitely has the are several is relation ways for the your deployments, so one is algorithms, because the tree itself is all real library. So you tell you: if you are familiar is Ruby, you can use the friendly Jim library directory and other ways is a arm TV. So this my company provides the packages holder operating systems so one it's an RPM for repertory necklace or wasn't always until package for our Debian and Ubuntu and the country.

A

It runs on the windows, so we also provides the MSI package or the Windows servers and you can choose installation weights or your knees. Oh of course, friendly provides container images or doctor and the kubernetes. So you can also you cannot retrieve under your container a long time.

A

The countries are, the people taught it with drizzle log in between to CN CF, so kubernetes or a very active the product is law uses separately for that the data data collection.

A

Okay- and this is our one example of the asteroid- Negro correction, so in the old days we use the article other transporters that the problem is. Our G of this approach is needed to wait. All data are logged in the local price, but in streaming a new collection called appropriately is monitored log files and if application writes the logs to the local files with leader logs the immediately and the friends that did send the data to the destination, this can be configured so by popular cases.

A

You descended to data to ii ii ii ii, so one second was 10 seconds to central servers.

A

So we chances are, we can see a newer data is quickly on the central servers on the Hadoop or the query, or is 3.

A

And rice, it is a friendly consist of the quando path, so pretty awkward, Kwanzaa program. So by province we can practice any data sources from the Ridgid faucet understand data to the any destination. So friendly is a group in the data pipeline. You don't win the turret or any reflection, demos or your own use cases so friendly, probably about your case.

A

Okay, this is over your entries and exits. Our picture this chapter are shows the house with three walks.

A

Okay, this is design your entry. Isis are friendly, consist of corn proteins. This list is a detail of the each each path so with the core hundreds even to buffer England's are littering. So this is a very important for the stable deployment. So twenties cross supports the buffering of the drawing and later hunting. So it's probably developer. Don't we need to consider how to deter either how to perform before data?

A

Writing so particular part can focuses on the how to read data from self the forces on how to write data to the technician so focus it on the quarry. This is tree. Has our water problems in the worldwide?

A

Maybe country 500 province in the Ruby Louie are the regions.

A

This is also important architecture, so currently, this is a friendly event to structure, so it means in currently one event are the ds3 compose are three elements so time and time on the record. So this is a one friendly events, so, for example, if you have a Patronus so on the one reigning is combative use at these ailments, so time is coming from the Apache Louis type.

A

The tag is identifying the data 40s, so we can use the tag for where come from where this event come from for the destination site under friendly uses, records, depositor JSON object, not lost or incorrect. Our patrols, because currently there are many systems, can accept the our JSON object. Natively solderable ask search or other web services and the gist object, easy to mutate on the data filtering or the hard to are easy to come back to the other format. So what we use is this object. Holes are able to record.

A

So Nepenthe is hundreds of data is a data streams. This is the data pipeline, simplified image of the data pipeline, so I incoming the incoming events is a pass through this pipeline from left to right right to left. So did I, sir? It is the current running. We probably send data to the filter and the past. Our output proposed the event which consists of these three events. I.

A

Took about each component.

A

We have two problem, so this is a very important because this is a start point of the are cradle is data pipeline. So, if the probably receives or little data from the data 40s, for example, look our leading local file or pourings HTTP, API or it DiNapoli's, is through as a long demons and very important point is put probably parties that incoming growth or the structural in so in probably competitors are our loss Turing into the fretless, a better structure.

A

So in this result, currently a filter probably and other plug, probably can easy to mutate or a combative. The event to three there is this new destinations format.

A

Okay, the Philippines very simple, so this is useful for the mutating. The turtle, for example, adding of adding the Sabbath or strain to the event record or D will be unnecessary liquids from the bio based on the field value example. This rule every it's not read it. You can use the triple filter to remove events from the data streams.

A

Philip Ragini could be changed, so you can operate the multiple logic to the data streams.

A

Is it a problem, but how progress did not stand alone Praveen? So this you can use a buffer problem directly from your configuration, but this problem improperly it's used or inserts the output, because this problem is very important for the Lobos buffering. Are we trying so buffering the data so algebra? We can improve the performance because sending one decoder to the destination is very slow, so mahaprabhu towards the data, because output and was it implemented a coma so or example- support to the right destination.

A

This is mainly holder, elasticsearch, all about homebody, probably who is a pro green.

A

Internally for 20 leads, Bertha is a cube chunks, so pakoras are numb several chunks and the child has article events, for example thousands records or about how the two labels. This is a you need talk the one fresh one fresh herbs out the proteins, and, of course this is a can be configurable, so how many numbers in the one one chunk or how her own falls out the brush, so this configures Mac size of the chunks or frosting tower of output, so you can cradles, are latency or throughput.

A

If we want to Loretta see you can see to the small rusty in trouble so once upon you get into the immediately, sir, are the thermos if you want of the throughput through a large light, larger body in the junk size? So you can get the maximum throughput without put a steep ravines.

A

So what the problem is very simple because of probably hundreds there are many concerns of the Romans, so just right up to use out for hollow chambers to the output.

A

Now this is our internal data model of the Freddie's data streams is the friendly, is our soybeans data progression but apprenticed under the data micro patch? It means so I see tada mahalo chunks, so in the batch model, this is a very hard to retry, because I we need to listen to the whole data to the destination. Finn's error happens, but in microbots model, so we can eat it to the dry, because chunk is small. So this is easy to write. I was a weak. Young protesters are small that small data chunks concurrently.

A

So this is improvable rating. She so called we don't when the two waiters are asleep right in this channel. So country has are several with the features for the final inning reflection from the low rating see.

A

This is the example of the our existing third party plug-in so reading data from 30 stop command or a Kafka or other messaging to services or DBMS.

A

Doctor provider is more warm up existing widgets because there are lots of data stores in the world, so you have TB pro materials or how do we provide ask such go over, but you can touch the rubygems? Maybe you can't find the right proteins for your databases.

A

Okay, so next is our I showed our values casing for the fluently. With actual configuration example,.

A

This is a simple forwarding example. So, in this case, when three correct data columns are two in to the forces, one is a local fire and the wines application practice. Data into MongoDB or DBMS.

A

This is the configuration who that is yes, disease in the friendlies configuration it's a Apache like robot, so we uses the particular delectable syntax, the source defaults into probably so this means in produce atop the tail, if probably under the forum praveen's, and the much detective is order, probably in if this this friendly instance uses a MongoDB for the data stores and much deliberative has a pattern so I see the friendlies event consists of the time and time and records and talk with use the whole event eluting.

A

So we can use the data pipeline to the runtime, so this is for the performance.

A

So in this case, intel commands me to data with the product apart tag and that this much pattern, except to the product star star, means any pattern. So take months are tales. Data is stored into the MongoDB in this case.

A

So this is very simple about the basic configuration for sovereignty and the next risk is. It is a multi-modal. This is mainly for the high traffic environment, for example, if you have a lots of instances or the whole hawala side, so many databases can't accept a lots of data collection from our clients. It is very high traffic, for this is not too good to hold a structural material, as are several databases in this case, tip-top applicator is mitigates. The problem of the data collection was the drive mechanism.

A

I want to talk about this this topics, but this talk is topic. Guys need more more time. So, if you want, if you are, wants to know the details of the disk watch a whole other module, so please check the my talk of the previous Kubik on I. Talk more deeply about this topic, so aggregation who are the side aggregation or the distance and aggregation and I, talked about the pros and cons for that. Several patterns of the original urged here.

A

The first country also supports there are multiple destinations so country, one data story, it's not popular, so a user set up the multiple databases or data stores for the data list platform. So, for example, lot of data is stored into the ask such followed. Recent analysis and settings are all stored. All data in the huddle poor, three answer period, people, data repository.

A

So in currently, we can use a copy from me, although these are deceased cases, so in copy probably send daughter data to the multiple destinations, for example, this case for P sin theta to ask such problem and love a great discovery. So this data is our same underdog struggling to the eg set of trough.

A

This is only tool to this mission, but we you can easy to. Others are our motivation. It's just out of the store we go after this point, so you can see the more data to the more destinations.

A

This is all of these cases, so consider logging. So currently it's our. Why do you use the whole chart on teleworking, so logging data from the doctor and the data under cooperate kubernetes? So this is a little easier use cases and Athiya our approach to collect the data on the container, so I will talk about the each approaches.

A

One is that this is a country she approaches. The doctor has are looking the driver mechanism, so this driver mechanism is sending the container locus to the external systems, so not toast and adult or local files. So this doctor neighbor has a several types or entity or get four more so friend, dr. cadra boster, boster friendly. So you can use this system. This approaches or collecting the container locus without the writings additional reporter in your application.

A

Arriba click the rows from the top card standard, auto Sandra Aaron data on the Senate Tuesday Fred of the instances. So this is a computation example of the doctor being a driver, so under indifferently you can only set up the who are the purple one of the protein. So this is very simple, so in this result, friendly instances can get to the top columns from the auto calyx directory via the loading driver. So this is dr. trick. So this approach can two stones: are quiet, Isola the container?

A

But if we use the doctor, you can choose that this approach.

A

And next it's our friend Ryoga, so friendly as a 50 by the community. There are lots of Flint Eluga library into each chip, each providing run which so Ruby Java Python, Goldust's or German or Morse. So you can use this free, local library. You cannot into any data to the friendly. So this is a very application, especially to data or metrics. So this approach is useful for very deep inside.

A

You can get deep insight for the data analytics or politics or machine learning, or more so about the matrix that you need to use of interrogates the your application. So you need to operate the your applications code from the side of the thing, so you can use the same configuration as w driver because a local regional driver and the interleukin-2 Roga is the same protocol between the friendly, be friendly notes.

A

Hey the necklaces are using the shared data podium using Ridley's interpreter into proteins. Some applications can't you just some rear wheels. You can't modify the lingo mekinese, so in this case, so how's that, for you, Olivia, like the lobsters, are fixed up off the rocker fine. In this case, you can use the share data body for accessing the data from the friend D in this case, and the X is stored into the mount to inject SLOS directory.

A

The friendly access to the dogs via the shared data volume and the using material, probably currently due to the local fighters of the collecting data. Now Intel probably is a very popular province infantry because this is very useful, the on-premise environment. So this is general approaching the friendly world.

A

And kubernetes demo set so friendly provider to buy us demo set images for the want to say environment. This demo tests is also uses a similar approach for the the share data poem because on Cuban it is come on. It is collected along with containers to the bar containers. So all container rugs are starting to the bar container, so I see da da da cunha tribal was approached. Chant is the homes kubernetes, so red cedar that is programmed by using the Intel, probably to collect the old container rules under the power containers.

A

So if you set up a friendly demonstrates friendly automatically leads the all rules under the bottle containers on the sender to the destination copyright, iris Archer as our destinations.

A

So this is a familiar container organ approach. So if you are using a dope car, you can try the doctor in a driver. This is easy to use and if you notice the more data or the application, specific application, specific data or metrics, you can use the half-frame draaga, so I'm just up less than your port and if he uses a mirror wheels or was that kubernetes you can use Intel proteins. Collecting the data from the local file system and the system matrix is a means of for CPU usage or little poetry.

A

No penalty can present this data by into praveen's, but I'm I think primitives over al-amin salivating. The data agent is better for the beta, because this is our. This agent focuses on the still metrics.

A

Okay, this is our last chapter and I took up those it's open to it.

A

So this is a comparison of the friendly and print bit. So itit definitely is a returning to B and C. So the beholder who are under the reactivity of the proteins using the sequel's, are you happy or deserialization?

A

The transmitter is fully implemented. We see so Fred bit is very small on the fast. So this is a it means this was friendly in in the container error. So a lot of sites resource it is very tight, so it's sometimes fiddly it's not of it for the whole other use cases. So in this case phantom it is based fit to holder over ice cases and that which follows the friendly design. So when Twitter is also probably so you can indexes are. Are your province in the builders of retweets?

A

You can easy to use the third-party probings, such so in the large deployments, are all odd rows from the frame to be 2 to the 50 and it's RPI use cases so I. This is a change of the, so you can depress who all's hydro, Fred T with a friend bit. So this is very. It was friendly on the water side and you can do so freely flexibility on aggregate aside, so you can easy to argues the or friendly province. So you can any. You can easy. 2.0 tools are any destinations.

A

So if Mohammed is our disturb our still needs of affirmative talk with that tomorrow. So if you like the rest, doing the friend meets the peace during the tomorrow's Eduardo's talk hey, this is the heart at all. Thank you.

A

B

Transcription layer, my first green, is actually was the day and the correct directive from the machine rock server we want to avoid. One thing is, we may have several copies for that right.

B

I understand that this question may not may not be the problem.

A

B

C

His question is that.

A

And this this sorry it don't I can't share the between the each destination. So so this is our problem interpreter, but each problem has their own format for the destination, for example, sometimes jz3 format towards data restoring the rest touch needs to the json format, so we can't share data between the destination. So this is the probably country we can't share the data between out problem copy.

B

C

So the data is a store life in one place right if you're going to forward from one place to a different part. Look like in your note, server you're to also put some history, and so there some changes over the data. So you can think that you have the same day and both sides, but it's different because they played a takeaway from, for example, is when about the container.

C

You have to contain information, but you also, if you bring in communities to meet the pot name, labels, adaptation and that informations have ended and there is ship it to a different place. So, ok, he's kind of similar information.

A

D

Wendy and associative are so sometimes multiple. My identity is not her very nervous and always posting time. So, in the times of way, brandy container will have inaudible memory. We know, Caesar is memories, memories, hi and.

A

D

Think there is something I know the ins and all of us, and it said as a.

D

A

You the question, sir, how to suppress the anybody brought in its feeling, just that it's done.

D

A

Frankly, is by default it's our memory buffer, so it means perfect. It's our student, its own memory. So if you want to talk about this problem, you change the file buffer, so my profile is just all even into the local file, so it doesn't took consume to the memory. So it's the beige, robust folder such unstable destination.

A

Well, if you can use the secondary use setup of the company, you can use a secondary set of Takada. To put it mean, this is useful for defender distinction is done, so the perfect chance is send to that secondary. So you can use that if you can use those cs3 output, so you can see the upper follow chunks to the rs3 is three foods are nikah Bonnie.

D

Five people publicize.

D

C

The suggestion is that if elastic was running this flow, they have a time when you just too much data enough that you have premium system, everybody use it by system approach, because because, if you're pulling B is called getting more Tara, you cannot close to there. You need to set it at somewhere somewhere. So.

A

You choose memory.

C

Consistency for.

D

C

To continue until at some point, so that's why he suggests use a predefined system. So, even if we cannot ship the locals.

D

So I think it already is a parasitical right.

D

C

If you contain it.

B

C

You run an airport and generated a lot well. That thought maybe will be destroyed after a minute two minutes. But what is important that what fluid is running? You need to be the mid laner for that hole. So if you don't get that and they get destroyed.

C