►
From YouTube: Logging for Containers by Eduardo Silva, Treasure Data
Description
Logging for Containers by Eduardo Silva, Treasure Data
"The implementation of Linux Containers provides enough flexibility to isolate applications with restricted access to CPU, memory and networking within others. While this technology is stable and production ready, there are some challenges that still needs to be addressed for the containerized application when deployed at scale: Logging.
While some applications writes their logs to the file system, others use the generic STDOUT and STDERR interfaces; when the application runs on top of a framework or virtual machine (JVM), it may generate some extra information. Since monitoring is a must, handling this data coming from different sources and formats adds an exponential complexity, specially when scaling to thousands of containers.
In this presentation I will describe the Logging challenges for containerized applications and how this is being solved with Fluentd."
About Eduardo Silva
Eduardo Silva is a principal Open Source developer at Treasure Data Inc. It currently leads the efforts to make logging ecosystem more friendly in Embedded, Containers and Cloud services. He also directs the Monkey Project organization which is behind the Open Source projects Monkey HTTP Server and Duda I/O. He is a known speaker having experience in South America and in recent Linux Foundation events in the US, Asia and Europe.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Okay,
a
thanks
for
coming
hey.
My
name
is
Eduardo
Silva,
a
software
engineer
at
treasured
data
and
I,
mostly
work
in
the
open-source
team
trying
to
fix
a
problems
with
login
and
trying
to
help
with
the
logging
tools
that
we
have
built
over
the
years
and
well
we're
quite
happy
here
because,
as
you
may
read
somewhere
we're
from
the
fluency
team
and
yes
today,
we
join
it
to
the
CN
CF.
A
A
Okay,
I
would
not
nag
more
questions,
so,
like
half
of
you
is
using
kubernetes
right,
awesome,
okay,
so
we're
going
to
go
from
the
beginning
about
this
session,
about
what
is
log
in
and
then
introduce
to
the
different
patterns
for
an
aggregation
in
the
cloud.
How
that
integrates,
Kuni,
DS,
docker
and
other
surprises?
Okay,
so,
let's
get
started
log
in
is
something
very
fundamental
for
everybody.
You
know
login
exists
because
of
three
main
principles.
A
You
need
to
monitor
something
or
you
need
to
make
some
troubleshooting
things
happens,
or
sometimes
you
need
to
cover
some
statistics
right.
What's
going
on
with
the
application,
how
its
behaving
so
log
in
I
would
say
that
is
not
longer
optional.
It
used
to
be
in
the
past,
but
when
you
need
to
dig
in
something
really
important
you
need
to
you
need
login.
You
need
to
get
information
from
somewhere.
A
What
happened
it
right
and
not
just
restore
an
application
which
trailer
container
and
let's
think
that
nothing
happens,
and
now,
when
we
have
the
login
the
scenarios,
a
login
happens
at
different
levels.
The
first
one
is
like
the
operating
system.
You
know
we
need
to
know
what's
happening
with
the
kernel.
Sometimes
we
have
some
storage
running
our
service,
but
that
storage
needs
a
driver.
Sometimes
a
driver
fails
and
we
need
to
know
about
it.
This
is
pretty
common
in
enterprise,
a
virtual
servers.
A
Also,
we
have
our
applications
units,
my
WordPress,
my
web,
on
my
back-end
for
any
kind
of
things,
but
also
I
need
to
monitor.
What's
going
on
with
that
unit,
and
also
we
have
system
services
like
the
same
login
services,
we
have
a
mail
services
or
any
kind
of
third
party
component
that
our
application
and
depends
on,
for
example,
a
database
and
now
we're
going
to
talk
about
the
containers
right,
I
assume
that
most
of
you
know
what
the
containers
is,
but
anyways
we
are
going
to
give
a
quick
overview
about
the
container.
A
A
container
exists
just
to
become
to
provide
isolation
to
a
process.
You
know
the
minimum
a
unit
in
an
operating
system
of
something
running.
It's
called
a
process,
but
when
we
don't
when
we
don't
have
the
containers
that
process
is
running,
it
has
a
bunch
of
CPU.
It
has
a
bunch
of
memory,
but
at
some
point
we
would
like
to
insulate
that
application
and
restrict
the
number
of
resources
that
that
application
has
access
to.
So
that
is
how
what
a
container
is
a
container
is
not
an
a
future
and
implementation.
A
A
container
is
a
concept
on
top
of
cgroups
control
groups
and
namespaces
from
a
pre
on
an
operating
system
level.
So
your
server
is
running
here.
You
had
a
host
operating
system
and
you
have
some
user
space
tools
that
allows
you
to
create
this
kind
of
concept
of
container
and
of
course
you
create
a
container,
and
on
top
of
that,
you
are
deploying
your
application.
Your
application
has
its
own
dependencies.
A
It
own
libraries
and,
of
course,
is
fully
isolated,
so
on
one
container,
I
can
be
running
some
application
with
I,
don't
know
open,
SSL
and
maybe
I'm
testing
a
new
in
crypto
library.
I
can
try
in
the
other
side
a
board
SSL
or
something
like
that.
It
doesn't
matter.
The
system
doesn't
know
what's
going
on
inside
the
container.
It
just
know
about
what
time
and
what
kind
of
CPU
is
consuming,
what
kind
of
memory
and
what
kind
of
devices
is
trying
to
access?
A
And,
of
course,
when
you
have
containers,
you
have
applications,
you
have
the
system,
you
have
services
unit
and
all
of
that
needs
logging.
So
a
what
are
the
kind
of
distribution
channels
that
we
have
volume
in
containers,
the
first
one
which
is
a
not
legacy,
but
the
one
that
always
works
in
a
file
system.
Well,
always
when
your
storage
is
working
right,
so
the
file
system
write
some
data
to
the
file
system,
so
you
can
trigger
and
review
that
later.
Also
we
have
the
standard,
input
and
output
interfaces.
A
A
We're
not
going
to
discuss
and
discuss
right
now
when
you
need
to
use
each
one
right,
there's
a
problem
I,
don't
probably
with
that,
but
they
exist
and
they
are
useful
widely,
and
sometimes
you
want
to
make
some
login
over
the
network.
Maybe
have
some
network
and
remote
networking
service
and
you
want
to
send
my
messages
to
that
central
place.
So
as
of
now,
we
have
this
kind
of
a
channels.
A
So,
okay,
but
when
you
have
your
container
okay,
you
have
the
channels,
but
also
you
can
let
your
logs
behave
or
put
them
in
a
different
way.
So,
for
example,
docker
integrate,
which
is
called
the
login
drivers.
That
means
you're
going
to
spawn
your
own
process.
This
application
is
going
to
do
something
and
maybe
write
some
locks
right,
but
you
can
tell
locate,
please
you
want.
You
know
what
I
want
my
logs
to
be
store
in
a
JSON
file.
You
say
what
yeah
there
are
people
using
MongoDB
right,
so
that
is
quite
useful.
A
Some
people
use
syslog,
which
is
pretty
fine,
I'm,
not
saying
that
when
a
solution
is
better
than
the
other
I
think
that
as
much
as
mass
solutions
you
have
not,
options
is
better.
You
have
generally
from
system
D
and
also
we
have
something
which
is
called
flew
in
D,
which
is
a
log
in
solution,
and
some
time
ago
we
wrote
the
driver
for
Tokyo.
So
every
time
that
you
get
docker,
you
had
the
flu
indeed
driver
on
it
and
ever
by
default.
A
We're
going
to
make
a
pause
here
and
talk
about
struck.
Two
blocks
know
every
time
that
you
get
some
load
message,
there's
not
an
instructor
most
of
the
time.
For
example,
if
you
look
at
the
Apache
web
server
logs,
you
will
see
that
you
have
some
timing
associated
there's
a
time.
There's
time
zone
IP
address
and
what
was
about
the
HT
dear
requests
right,
but
there's
a
kind
of
a
structure.
A
A
For
example,
when
you
look
at
the
Apache
logs,
you
know
you
know
that
there's
a
time
the
key
value
will
be,
the
key
will
be
time
and
the
value
of
course,
the
time
the
time
comes
the
time
times
over
and
of
course,
you
have
to
minimum
keys
the
time
and
the
message
time
is
something
very
critical,
because
you
need
to
know
when
something
happens
and
when
you
deal
with
structure
locks,
there
are
two
main
formats
around.
Maybe
there
are
more,
but
these
two
are
quite
famous
the
first
one
is
Jason.
A
I
know
that
everybody
knows
Jason,
but
jason
has
a
really
really
big
problem.
When
you
want
to
parse.
Json
is
quite
expensive
because
it's
a
string
is
not
binary.
You
don't
know
when
a
key
start
when
the
key
ends.
So
you
need
to
read
the
whole
message.
Imagine
that
you
have
a
JSON
map
and
the
message
field.
A
I,
don't
know,
imagine
that
it's
a
thousand
of
characters,
but
you
don't
know,
there's
a
thousand,
so
you
need
to
go
each
by
each
one
iterating
and
to
fight
the
last
one
and
say:
oh,
this
is
1000,
and
that
is
the
very
CPU
intensive
and
also
there
is
something
a
different
option,
which
is
called
message
pack,
which
is
like
a
JSON,
but
in
binary
mode.
It's
a
lesser
realization,
format
use
it
widely.
It's
adjacent
like,
but
it's
quite
very
for
that.
Of
course
you
cannot
read
message
pack.
A
A
So
how
docker
deal
with
this
getting
back
to
containers
when
you
write
at
a
docker
log
or
log
message,
two
standard
outputs
on
the
error
you
will
get
some
like.
This
is
a
test
message,
but
when
you're
using
the
login
drivers
you're
getting
something
more
interesting
like
this,
you
get
a
container
ID,
you
get
a
container
name
the
source
and,
of
course,
the
message.
A
So,
if
you
compare
this
scenario
with
the
other,
I
will
say
that
this
is
a
little
bit
a
little
bit
better,
because
you
have
some
metadata
about
where
this
application
is
running
right.
It
doesn't
matter
if
it's
JSON
or
message
in
fact,
the
things
that
you
have
a
struct
or
you
know
which
kind
of
key.
You
want
to
look
up
and
do
something
with
it.
A
And
now,
if
you
start
thinking
about
microservices,
meaning
how
do
we
deploy
applications?
Okay,
we
are
not
going
to
discuss
about
in
general
about
maker
services,
but
we
want
to
focus
on
login.
Imagine
that
a
monolithic
application,
basically
you
have
users,
the
service
application
and
this
service
application
is
writing
logs
to
somewhere
over
a
network
or
file
system.
But
when
you
have
micro
services
you
have
applicant,
your
application
is
split
in
many
components,
maybe
one
two
or
three
it
doesn't
matter.
A
But,
as
you
know,
each
of
these
components
needs
also
have
some
logging
capabilities,
so
hero
login
could
be
sown
a
quite
easy
right.
You
just
write
to
somewhere,
but
here
you
have
multiple
applications
run
into
the
same
unit
of
logging
and,
of
course,
when
you
want
to
scale
that
it's
a
bit
complex.
So
imagine
this
when
you
creating
is
some
kind
of
application
it
generate
the
logs,
maybe
in
a
different
format
and
a
meaning,
a
different
format.
The
string
is
different
for
each
application.
A
I
know.
If
you,
the
developers,
are
here
at
radians
one
application.
They
wrote
some
kind
of
log
message
right
and
maybe
two
months
later
they
would
say:
oh
I
am
going
to
add
more
load
messages,
and
maybe
those
messages
has
a
different
format.
So
how
do
you
parse
that
there
are
some
people
smiling
right,
so
it
happens
so
and
also
parse.
The
plain
text
file
is
really
really
expensive
and
not
all
contain.
As
per
Apple.
My
name
is
Suresh.
A
Sometimes
you
can
start
a
container,
but
that
does
not
means
that
you
have
access
to
the
file
system
so
where
you
write
the
logs,
so
here
we
introduce
the
concept
of
distributed
logging.
Where
do
we
put
our
logs?
So
when
we
are
talking
about
this
kind
of
architecture,
we
need
to
stop
thinking
about
just
a
file
justified
system,
just
a
fixed
storage.
It's
about
how
I'm
going
to
deal
with
the
different,
a
aggregation
patterns.
How
to
distribute
my
logs.
Imagine
that
you
have,
for
example,
you
make
a
web
application.
Sorry,
a
mobile
application.
A
You
wrote
the
packing
okay,
you
have
a
hundred
of
users
and
your
login
system
in
the
backing
is
working.
Pretty
fine
is
writing
to
the
file
system.
It
works.
But
what
happens
if
your
application
goes
viral?
And
now
you
have
5,000
users,
you
know
or
Pokemon
go.
You
know
that
it
was
used
like
a
rocket.
So
how
do
you
scale
the
login
on
that
a
scenaries?
A
Okay?
So,
for
that
we
have
different
architectures.
We
need
to
you
need
to
find
that
we
have
collector
nodes,
something
that
collect
information
or
collect
information
from
somewhere.
That
information
is
sent
to
which
is
called
an
aggregator.
Then
we
get
a
collect
information
and
its
job
is
to
distribute
that
information
back
to
some
destination.
It
could
be
a
database,
a
cloud
service
or
anything
like
that.
A
So
logging
is
hard
because
there's
many
steps
in
the
middle
and
now
there's
something
some
items
to
consider
when
we
are
implementing
this
kind
of
logging.
The
first
is
what
kind
of
network
traffic
do
we
want
to
implement
on
this?
How
much
CPU
I
can
afford
for
this
and
what
kind
of
a
failover
system
high
availability
I'm
going
to
implement
so
dependent
of
these?
We
have
different
application
patterns.
A
Everyone
has
a
different
load,
so
we're
going
to
discuss
which
one
is
better:
okay,
we're
going
to
discuss
the
source
aggregation
patterns
so
when
the
application
is
running,
for
example,
on
this
case,
without
a
source
aggregation,
we
have
the
collectors
writing
directly
to
one
aggregator
right,
but
with
source
application
we
put
an
aggregator
container
inside
our
collector,
typically
in
a
docker
container,
typically
inside
our
kubernetes
pot
right.
It's
about
how
do
you
want
to
handle
this
thing
and
what
are
one
advantage
of
this,
for
example,
without
source
aggregation?
A
The
configuration
is
quite
simple
because
you
are
just
writing
to
one
place,
but
there
has
a
problems
and
you
cannot
change
the
regular
endpoint
address.
It
will
be
a
problem.
You
will
have
many
network
connections
and
also
you
will
have-
will
face
a
high
load
on
the
aggregator.
So
how
do
you
want
to
deal
with
that
with
with
source
aggregation?
You
have
less
connection.
A
Of
course
your
aggregator
don't
have
too
much
overhead
and
of
course
you
have
less
configuration
in
the
containers,
but
you
need
more
resources
because
you
need
one
aggregator
container
per
cost,
because
you
are
adding
some
one
layer
in
the
middle,
but
there's
nothing
free
here,
you
need
to
add
some
cost
at
some
level.
The
thing
is
worth
now
when
we
talk
about
the
destination
when
we
are
sending
the
data
back
to
somewhere
without
destination
aggregation.
A
We
have
this
pattern,
but
with
destination
aggregation
we
can
make
some
interesting
because,
as
you
know,
network
fails,
storage
fails,
things
happens.
So
at
some
point
you
would
like
to
implement
some
kind
of
never
make
a
lesson
in
in
what
you
can
say:
okay,
I'm,
going
to
send
the
data
to
the
assassination,
but
also
I
want
to
have
this
second
a
secondary
destination.
Just
in
case
something
happens:
okay
without
destination.
You
have
less
note.
The
configuration
is
simple.
A
The
this
one
touches
the
storage
side
here
affects
the
collector
side
and
you
have
the
worst
performance,
of
course,
because
you
have
many
small
writes
each
time
you're
likely.
We
would
like
to
have
some
kind
of
buffering
and
a
big
write
instead
of
many
small
writes
with
the
destination
aggregation.
Of
course,
your
collector
configuration
is
free
from
storage
changes.
So
if
you
change
something
at
this
site,
this
has
this
guy
here
on
the
top
level.
A
So
they
think
is
how
many
nodes
you
have,
they
think
is
how
you
want
to
distribute
the
load
of
the
logs
in
different
places.
I
have
I
was
talking
yesterday
with
a
person
who
said
a
ok
I
have
some
problems
because
I
don't
know
if
I
should
have
used
an
aggregator
here
or
have
multiple
destination
or
small
writes
and
the
question
the
simple
question
was:
okay:
do
you
have
some
network
limitation?
No
I?
Don't
you
have
some
CPU
limitation?
Yeah
I
had
a
diamond
AMISOM?
A
Ok,
so
you
know
how
to
distribute
the
load,
so
you
don't
get
a
more
expensive
bill
and
to
to
implement
all
this
irrigation
patterns,
which
is
kind
of
concept.
You
need
the
right
tool
for
this
kind
of
solution,
so
this
is
where
flu
in
D
joints
in
flu
in
D
is
an
open
source
data
and
lock
collector,
which
was
decided
to
achieve
all
these
kind
of
aggregation
patterns
and
adapt
to
your
own
needs
and
well.
It
was
made
with
high
performance
man,
its
built-in
reliability.
A
Here
you
can
imagine
that
you
are
getting
the
data
in
from
some
kind
of
input,
so
fluency
supports
many
inputs.
Imagine
that
a
log
file
you
can
listen
from
TCP
or
maybe
you're
listening
for
the
darker
log
files
over
the
network.
Once
you
get
the
data
in,
do
you
need
to
do
some
parsing
over
that
data
right,
because
the
data
has
a
different
format?
It's
not
the
same.
A
To
listen
for
a
JSON
message
that
listen
from
a
syslog
message,
so
you
need
to
parse
that
so
here's
a
lot
of
complexity
and
then
you'll
like
to
filter
out
some
messages.
Maybe
you
want
all
of
them
or
maybe
you
want
to
drop
some
messages
once
you
finish
this
internally.
What
fluency
does
is
provide
you,
some
kind
of
buffering.
What
means
buffering
that
every
time
that
I
get
the
data
in
I
want
to
keep
them
in
memory
or
either
in
the
file
system,
because
before
to
send
this
information
back
to
a
destination,
something
good
happens.
A
Okay
destination
goes
down,
the
storage
is
not
working,
so
you
need
this
buffering
capability.
So
if
something
gets
well
gets
wrong,
you
can
reload
this
kind
of
chunks
of
data
and
set
it
again
later,
but
before
to
set
it,
you
have
the
you
need
the
output
plugins
so,
where
I'm
going
to
send
this
information,
so
imagine
that
a
ok
I'm,
sending
information
to
Amazon
s3
right
Amazon
has
their
own
API
their
own
format.
But
sometimes
somebody
tell
you,
know:
I
want
to
use
my
own
elastic
search.
A
An
elastic
cell
has
their
own
API
their
own
format.
So
the
thing
is
that,
from
the
main
point,
where
you
collect
this
information
and
then
send
information
back
to
somewhere,
you
need
to
collect
you
need
to
parts.
You
need
to
fill
that
you
to
make
buffering
and
adapt
this
buffered
data
to
the
destination.
A
So
now
you
can
realize
the
make
login
in
the
right
way
is
quite
complex
and
expensive
by
it
nature
unlikely
you
want
to
format
the
data
out,
so
a
fluent
'I
have
been
doing
this
test,
the
beginning.
It
was
starting
on
2011
and
now,
as
you
know,
it's
quite
loosely
it's
very
usable
and
well.
It
has
a
lot
of
a
opera
opera.
Leave
it
obeyed
ability
with
sorry.
A
A
A
How
this
works
in
the
docker
use
case,
for
example,
when
docker
flush
out
some
blocks.
It
adds
a
lot
of
metadata,
for
example,
container
ID
container
name
a
timestamp
and
the
lock
message.
So,
ideally
you
want
your
containers
started,
flushing
the
whole
logs
to
one
central
flu
in
D
and
from
this
flu
in
D
you
can
send
back
the
information
to
elasticsearch
or
any
kind
of
service
that
you
are
working
here.
A
Some
people
use
Hadoop
elasticsearch,
my
sequel
and
some
people
use
an
SQL
light
and
it
works
now
in
the
case
of
kubernetes,
they
put
flu
in
the
inside
the
node
right,
so
you
have
your
own
applications
in
the
pod
running
their
logs
and
you
have
flu
in
d
consuming
that
data
from
the
file
system
we
need.
It
has
a
philosophy
that
the
whole
logs
need
to
go
to
storage
first
and
then
the
the
low
collectors
can
read
from
that
which
is
predefined,
because
it's
like
a
safe
way
to
work
and
fluently
in
the
real
world.
A
Well,
it's
being
used
by
Microsoft
extensively
at
less
angry
Nintendo
and
a
lot
of
companies.
The
use
case
of
Microsoft
is
they
have
something
which
is
called
OMS
I,
don't
know
if
you
hear
about
it
like
operation
management,
suite,
okay
and
what
they
do
is
that
they
said
okay,
one
year
ago
they
say
we
need
an
agent
on
each
host
that
is
working
with
Microsoft
services
and
we
need
to
collect
information
so
what
they
did.
They
took
Flo
nd
they
package
flow
in
d
and
add
some
really
good
extension
for
that
so
flowing
this.
A
A
You
know
we
are
scaling
that
and
at
the
company
layer
who,
where
we
promote
this
project,
as
we
are
proud
member
of
Linux
Foundation
than
a
CNC
F
and
as
I
say
before,
we
got
some
news
that
the
fluent
D
also
joining
the
CN
CF
today.
So
what
that
means
for
everybody
who's
on
this
audience
is
that
flew
in
D
will
will
be
a
better
citizen
inside
the
kubernetes
ecosystem.
A
It
will
expose
now
different
a
kind
of
interfaces
for
permit
you,
so
they
can
gather
some
matrix,
I
get
more
information
about
it,
and
this
is
really
good
for
us,
because
the
community
is
quite
big.
Now
we
have
thousands
of
companies
using
flow
in
D.
We
have
thousands
of
individual
users
and,
as
you
use
so
we
have
like
more
than
600
plugins
around
and
most
of
them
are
made
by
individuals
and
now
I
wanted
to
do
something
different,
get
him
back
to
logging.
A
You
know
I'm
going
to
talk
at
the
ANU
Prague
that
just
born
like
a
year
ago,
which
is
called
full
and
bit
is
different
that
flew
in
D
and
through
a
bit
initially
born
initially
like
a
year
ago
to
solve
the
data
collection
problem,
but
from
embedded
devices.
So
when
you
have
a
really
small
device,
you
cannot
think
I'll
have
flu
in
the
inside
it
right
because
fluid
is
made
in
Ruby
right.
So
it
needs
some
kind
of
memory,
some
kind
of
CPU
and
for
a
really
really
small
a
device.
A
You
cannot
have
it
so
we
started
creating
a
new
solution
for
that
specific
market
and
open
source,
of
course,
which
is
called
flow
and
bit
which
was
reading
in
see
fully
high
performance,
has
a
an
architecture
with
plugins
and
also
be
capable
to
gather
some
matrix
of
the
system.
Word
it's
running
like
CPU
memory
and
networking
stuff
is
fully
even
driven
and
also
is
fluently
compatible.
What
that
means
that
for
a
min
ba'di
device
on
that
moment,
you
can
gather
some
matrix.
A
A
The
architecture
is
quite
similar
and
I'm
not
going
to
get
into
much
detail
so
name,
but
it
has
like
input
plugins,
you
have
the
main
engine,
and
once
you
get
the
data
in
optionally,
you
can
dust
buffering,
so
you
it
does
buffering
either
in
memory
or
in
the
file
system.
It
has
its
own
POSIX
threads
to
handle
this
information,
and
the
good
thing
about
this
is
that
there's
no
locking
at
a
program
level.
Everything
is
just
the
engines,
one
thread.
We
have
multiple
workers,
POSIX
threads
on
this
side,
but
there's
no
login
inside
them.
A
A
Well,
well,
you
can
do
is
to
modify
your
application
to
write
the
logs
to
any
standard
output
like
a
printf
and
you
let
docker
check
that
out
to
journal
D
or
fluency
or
whatever
that's
a
easy
ways.
This
way
so
getting
back
to
flowing
bit
so
they're
character
square
root,
that
is,
it
does
not
have
too
much
overhead,
there's
no
locking
if
threads,
but
there's
not
locking
or
there's
no
contention
on
that
area
and
at
the
implementation
level.
When
distributing
this
information
to
some
destination,
we
are
using
some
coroutines.
A
You
can
imagine
that
I'm
talking
about
a
bit
about
how
go
lonk
works
and
I
will
set
up
flowing
bit,
implement
almost
the
same
concepts
that
the
goal
and
language,
but
in
c,
so
it's
quite
it's
pretty
straightforward
to
create
a
plug-in
or
some
extension
for
it
and
Fulham
beat
a
support.
Many
things.
For
example,
you
can
send
right
now
it's
not
just
for
embedded
devices,
it's
being
used
as
a
compliment
for
fluency
and
also
collected
logs
from
different
services.
A
Okay,
we
are
not
yet
there
with
a
lot
of
extension
that
we
have,
but
we
are
working
hard
on
it.
So
when
the
destination,
which
is
really
good,
for
example,
elasticsearch,
if
you
use
an
elastic
search,
you
can
just
pull
in
bit
to
show
to
push
the
logs
also
to
elastic
search
and
so
for
in
bit
support
HTTP
output
endpoints.
So
if
you
have
some
HTTP
web
server-
and
you
can
listen
for
events,
you
can
just
flew
in
bid
to
distribute
the
logs
to
that
place.
A
Also,
a
fluent
bid
was
made
with
forward.
You
know
per
future
in
mind
what
that
means.
If
I
have
one
flew,
indeed
to
other
fluency
of
wind
bits,
they
can
wrote
the
messages
between
them.
On
that
way,
you
can
make
some
kind
of
file
over
at
fluent
bit.
It's
quite
fast,
the
last
benchmark.
We
saw
that
we
can
send
one
message
from
one
flowing
bit
to
the
other
around.
We
can
send
500,000
messages
per
second
in
this
computer.
A
So
imagine
what
you
can
do
in
a
biggest
environment
using
just
one
CPU
core,
and
that
is
quite
good
and
getting
back
to
architecture
is
basically
yes,
a
core
engine
with
us
initialization
it
does
buffering
it
handle
the
networking
stuff,
it
also
supports
TLS.
So
if
you
care
about
security
over
the
network,
it
provided,
you
have
a
plugins
manager,
so
you
can
have
your
own
plugins
in
the
April
for
collect
data
or
to
send
the
data
back
through
output
use
message
back
internally.
We
just
talked
about
that.
A
You
support,
TLS
and
use
some
specific
libraries
like
monkey
to
handle
the
whole
event
loop
and
memory
handle
another
good
feature
about
flambéed.
Is
that
if
you
are
writing
in
an
upper
level,
language,
for
example,
Python
or
Ruby
or
whatever
you
can
take
advantage
of
fluent
bit
in
a
library
mode,
that
means
that
flowing
bit
is
not
just
a
service.
It
can
behave
as
a
library,
so
you
can
say:
okay,
my
apply
I'm!
Writing
my
application.
A
A
Ok
in
the
first
line,
we
just
include
a
header:
a
are
you
familiar
with
software
development
here,
most
of
you,
okay,
this
is
C,
but
it's
pretty
straightforward.
Okay,
in
the
first
line,
we
just
include
the
header
right.
We
just
create
a
context
of
the
library.
Flb
is
like
the
prefix
for
flowing
bit.
A
Then
we
define
an
input
type
because
flowing
B
needs
to
collect
data
from
somewhere
right,
we're
going
to
use
the
lip
or
library
mode,
which
means
that
my
own
application
is
to
ingest
data
into
flowing
bit
after
that,
we're
going
to
add
some
metadata
to
this
input.
We're
adding,
which
is
called
attack
attack,
is
like
a
space
special
flag
to
each
record
that
is
coming
from
that
input.
So
once
I
get
that
data
in
in
inside
the
flowing
bit,
engine
I
can
say:
okay,
all
the
records
that
had
this.
A
Can
this
kind
of
tag
are
going
to
elasticsearch
this
kind
of
record
that
has
a
different
tile.
They
are
going
to
Amazon
s3.
That
is
what
tag
is
for.
Then
we
can't
need
to
define
the
outputs.
Always
we
are
using
the
context
from
the
library
and
define
an
output
tab
on
this
case.
We
are
used
in
the
standard
output
and
we
are
adding
a
match.
So
everything
that
match
the
tag
test
here
that
it
was
said
before
is
going
to
the
standard
output.
It's
an
I/o,
but
it's
just
for
debugging
right.
A
It's
not
that
it's
not
a
big
deal
and
just
start
flowing
bit,
and
so
in
bid
we'll
start
in
a
separate
thread
running
in
background
and
from
your
parent
read.
You
can
take
control
of
it
like,
for
example,
ingest
some
data,
and
what
we
add
support
for
here
is
that
you
just
add
some
JSON
message
to
flow
in
bit
and
it
will
interpret
that
internally
and
do
something
with
it
and
then
you
can
stop
at
any
time.
A
Of
course,
here
you
had
your
own
code
doing
your
own
stuff,
if
you
are
doing
some
in
data
collection
from
sensors
matrix
staff.
This
is
very,
very
useful.
Maybe
it's
more
easy
for
you
to
write
to
the
file
system,
but
this
is
mostly
for
developers
this
area
and
a
I
wanted
to
do
something
that
was
done.
A
few
days
ago,
because
I
know
that
most
of
people
don't
know,
don't
like
too
much
C
language,
because
it's
quite
complex
for
some
missionaries,
but
sometimes
they
like
go
so
well
with
this.
A
A
So
a
the
thing
is
that
you
can
write
your
own
plugins
in
go
for
fluent
bit.
It's
pretty
functional
at
the
wheel
up
come
in
the
next
version
in
the
next
SEO
that
10
release,
and
also
we
need
contributors
for
that.
You
know
we
are
doing
a
pre,
we're
pretty
strong
and
collecting
data
handed
that
data
optimising
the
whole
engine,
but
we
need
people
to
help
to
contribute
with
destinations.
So
we
need
there's
a
lot
of
people
asking
for
a
casket
output
plugin.
A
You
know,
but
I'm
not
going
to
write
and
a
casket
output
plug-in
and
see
it.
It
will
take
a
lot,
but
there's
some
packets
ready
on
go
like
so.
If
somebody
can
want
to
take
care
about
that
will
be
great
and
how
a
plugin
looks
like
ok,
a
first
of
all
a
package
for
sorry
apply
a
gold
plugin
for
flowing
bead.
It's
basically
a
shared
library,
so
go
when
you
build
a
go
code,
you
can
say:
I
want
to
have
a
binary
that
you
can
run.
A
I
want
to
have
an
archive
file
or
a
shared
library.
This
works
with
a
shared
library.
So
basically
you
import
the
flu
in
bitgold
package,
for
example.
This
is
an
example
for
the
standard
output,
but
made
in
gold
right
and
you
define
three
callbacks
and
you
need
to
define
the
export,
because
that
is
something
required
by
the
law
by
the
by
the
code,
language.
A
First,
we
do
the
initialization
and
you
register
the
plug-in
right
and
then,
when
flew
in
Pete,
collect
the
data
some
world
and
want
to
send
this
information
through
the
plugin
in
it'll
call
some
callback
that
will
take
care
of
that
data,
and
this
is
a
flush
call
back
right
at
this
point
here.
You
will
like
to
send
this
information
or
do
something
with
it,
and
the
information
that
you
get
here
is
a
message,
a
packed
format,
there's
a
good
hunter
for
that
on
golang.
A
A
A
A
If
you
can
see
here,
it
means
that
the
input
plugins
we
can
get
a
message
from
the
CPU
from
the
memory.
You
can
read
that
your
kernel
log
messages
directly
from
the
corner
log
buffer.
You
can
read
some
files,
you
have
an
input
plug-in
to
check
how
healthy
is
a
TCP
service,
so
you
can
tell
through
a
bit
please
pink
this
service
every
second,
and
let
us
know
when
this
is
visa
light
or
not.
We
have
some.
You
can
listen
for
message
from
the
start,
the
serial
interface,
the
standard
input
TCP.
A
We
support
n
QT
t
forward
protocol
for
flow
in
D
and
and
so
the
outputs
most
interested
one
I
will
say
that
is
elasticsearch
forward
that
we
can
set
the
message
messages
to
a
different,
fluent
e
and
H
to
the
P
output,
endpoint
or
treasure
data.
Our
cloud
service
for
analytics,
so
we
are
going
to
hear
pretty
quickly-
is
to
just
collect
information
from
the
CPU
and
send
it
to
the
standard
output.
Every
5
second
is
going
to
measure
how
much
CPU
many
computers
consuming
and
print
out
some
kind
of
JSON
messages
right.
A
A
A
A
Look
at
this
line.
This
is
a
way
to
wrap
to
build
a
go
plugin
for
Fulham
bit,
so
we
are
going
to
type
make.
The
only
issue
is
that
when
you
build
shared
libraries
with
golang,
it
take
a
few
seconds
like
10
11,
but
the
good
thing
is
that
it
works
there.
You
go.
So
if
you
look
at
here,
we
have
the
shared
library.
A
We're
50
megabytes
to
bring
to
the
standard
output,
but
of
course
this
is
a
way
that
goal
and
works.
You
know
that's
easy
cause
the
size,
but
performance
is
not
too
much
so
okay,
so
we
have
the
plug-in
we
have
flowing
beats.
So
we
want
flowing
bit
to
load
this
plugin,
as
you
saw
before.
If
you
look
at
the
outputs,
there's
no
output
here
in
the
there's,
no
golang
standard
output,
but
we
can
load
it
with
extension
model,
go
which
our
shared
library
exam
through,
indeed
go
out
of.
Sorry
there
we
go.
A
A
So,
as
you
realize,
my
intention
is
to
motivate,
go
like
developers
to
join
this
project
and
contribute
with
something
so
we're
going
to
collect
where
I
am
so.
Indeed,
we
are
going
to
collect
all
we
load,
this
tension,
we
load
data
from
the
CPU
and
we
send
out
the
information
to
the
Google
so
goal
and
standard
output.
A
So
it's
pretty
much
the
same
at
the
previous
example,
but
here
is
going
to
use
the
golden
code
right,
so
it's
pretty
straightforward
how
you
can
extend
a
login
solution.
The
same
is
for
flu
Indy
is
you
know,
Ruby,
you
can
explain
mint
your
own
destinations
in
Ruby
language
or
you
can
do
it
and
go
from
flu
in
people
and
fluid
ulcers
joining
the
CN
CF
a
when
to
indeed
join
the
CN
CF
need
not
join
them
alone.
It
join
it
with
35
repositories
of
SDKs
languages
and
child
projects
like
this
one.
A
A
Oh,
a
lot
of
slicin
next
time
will
be
shorter,
so
the
rule
not
for
sure
that
10
is
2
right,
gulp
plugins,
we
are
finishing
their
built-in
HTTP
monitoring
so
flowing
bit.
When
it's
start,
you
can
query
the
internals
a
add
some
variable
support
because
you
may
like
it
to
add
some
hostname
to
each
record
IP
address
or
anything
that
customizable
and,
of
course
filtering.
Maybe
you
want
to
drop
something.
This
is
the
last
slide.
There's
my
contact
in
the
breaks
address,
they're
all
open
source.
So,
if
you
want
to
join
us,
you
are
welcome.