►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Deep Dive: Fluent Bit - Eduardo Silva & Masahiro Nakagawa, Treasure Data
Fluent Bit (https://fluentbit.io) is a sub-project of Fluentd Ecosystem, considered as a lightweight and high-performance log-processor and forwarder. On this session we will dig into its internals and special features available for Kubernetes and cloud-native environments in general. Everybody interested in Logging is welcome to join us.
To learn more: https://sched.co/GreD
A
Everybody's
happy
no
worries
the
projects
continue
working
on
it,
so
we
have
more
resources
for
to
hire
more
people,
so
we
are
hiring
so
if
you're
interested
just
come
to
us
talk
so
a
at
this
company
I
work
as
a
principal
engineer
at
arm
and
I
maintain
this
break,
which
is
called
fluent
bed,
which
is
a
fluent
deep
project,
ecosystem,
I'm
sure
that
most
of
you
are
familiar
with
fluency.
Oh,
please
raise
your
hands
if
you
are
from
liquid
flowing
the
deaf
person
we're
good.
So
a
fluent
bed
is
like
a
sub
Prague
of
flu.
A
A
So
the
meter
of
this
there's
no
backup
now
so
in
terms
of
logging?
Everybody
knows
that
we
do
login,
not
because
it's
fun
we
do
login,
because
we
need
it
because
we
want
to
achieve
some
data
analysis
right,
hey
I
would
say
that
logging
is
kind
of
boring
yeah.
It's
not
something
that
you
want
to
be
looking
every
day.
A
You
just
take
a
look
when
you
want
to
see
some
metrics
or
why
something
is
failing,
or
why
your
things
are
not
right
right,
but
our
focus
is
always
data
analysis
so,
and
the
challenge
with
data
analysis
is
comes
from
that
the
data
come
from
multiple
source
of
information.
This
came
from
TCP
UDP
log
files.
We
talked
about
log
files,
but
anyways.
There
are
a
stream
of
data,
and
also
this
data
comes
in
different
formats.
A
So,
a
years
ago,
when
fluency
was
decided
at
the
company
and
the
team
behind
it
said:
okay,
if
you're
going
to
deal
with
data
and
that
come
from
one
place
to
another,
there
are
many
faces
in
the
middle
and
many
things
that
we
need
to
solve
step-by-step
and
anyways.
If
we
want
to
implement
login
or
solve
any
problem
in
our
login
layer,
we
need
to
understand
how
it
works
internally.
A
Without
this
knowledge
is
really
hard
because
it's
easy
to
say:
oh,
this
is
not
working,
but
sometime
can
be
your
input,
source
of
data
or
any
kind
of
face
in
the
middle.
So
we
come
up
with
this
design,
which
is
called
the
login
pipeline.
When
you
say
that
at
the
beginning,
you
have
some
the
off
this
code
stuck
give
me
one.
Second,
there
you
go
so
when
you
collect
the
data
from
some
kind
of
input
source
of
data
right,
but
this
data
then
needs
to
be
parceled.
Maybe
it
doesn't
have
a
structure.
A
You
need
to
give
a
structure
to
this
data,
or
maybe
this
data
and
you
know,
have
a
structure,
but
you
need
to
tell
the
block
processor
how
to
operate
this
data
because
don't
forget
that
our
goal
is
data
analysis,
so
we're
in
the
face
before
that
and
after
we
purge
the
data.
Sometimes
you
you're
not
interested
in
the
whole
data
that
you
are
getting
in.
A
Maybe
you
want
to
filter
data
out,
or
maybe
you
want
to
add
some
kind
of
metadata
on
a
rich,
your
logs
with
additional
information
common
cases
I
like
I,
want
to
add
my
host
name
to
the
log,
so
my
IP
address
Harvard
address
or
anything
like
that.
Once
you
collect
the
data,
parsed
data
filter
the
data.
This
data
is
going
to
be
stored
in
a
temporal
place
for
a
fraction
of
seconds
on
maybe
a
couple
of
minutes,
and
for
that
you
need
something
that
is
called
buffering.
A
Well,
any
any
product
running
a
computer
is
running
in
memory,
so
buffering
is
not
something
new,
but
for
reliability.
Sometimes
you
need
to
provide
different
and
make
an
S
and
from
buffering
so
because
what
happens?
Maybe
you're
going
to
wrote
the
data
to
a
different
destination,
because
maybe
your
n
is
to
stored
in
elasticsearch,
maybe
scott-carr,
nuts
or
whatever,
but
maybe
what
would
happen
if
one
of
the
output?
A
Okay,
so
now
talking
about
the
flu
embed,
the
flu
embed
is
a
project
that
born
on
2015
and
I'm,
going
to
tell
you
the
flu
it
burns
for
embedded
Linux.
That
was
your
original
idea
for
fluent
bit,
because
we
have
flu
Andy
and
people
from
the
embedded
side
or
the
IOT
space
with
some
small
boards.
They
say
we
need
a
lock
processor.
We
need
to
collect
the
data,
we
need
to
parse
this,
but
sometimes
flu
indeed,
can
be
a
little
bit
heavy.
A
So
we
come
up
with
starting
writing
this
break
from
zero
and
after
one
year
and
a
half
of
the
project,
we
got
something
functional.
We
got
some
plugins
in
place
and
people
said
you
know,
the
embedded
market
is
quite
fragmented.
Everybody
writes
their
own
solution
is
not
like
in
the
cloud
where
everybody
tried
to
see
all
your
writing
this
component.
A
Maybe
can
you
say,
let's
try
to
match
this
in
a
new
project
is
totally
different,
but
people
from
the
cloud
space
pay
attention
to
flow
and
path,
and
they
say
eh
I
need
something
lightweight
to
process
my
locks
and
kubernetes
I'm
using
fluently.
I'm
pretty
happy
has
aggregation
and
everything,
but
I
need
something
that
consumes
less
memory
and
less
CPU,
because
I
have
a
hundred
of
node,
and
that
is
a
is
a
ballad.
A
A
statement
so
flume
beat
is
written
originally
I'm
still
in
C
language
and
that
will
not
change
I
hope
it's
tied
to
have
a
low
memory
and
low
CPU
footprint.
If
you
start
flow
in
bed-
and
you
do
nothing
with
it,
it
will
consume
around
500
kilobytes.
But
of
course,
if
you
ingest
data
on
it,
you
know
your
buffers.
Your
memory
will
go
up.
A
So
when
everybody
say
oh,
my
tool
is
using
this
amount
of
memory
or
say
that's
a
minimum,
but
if
you're
doing
some
a
lot
of
working
a
lot
with
a
lot
of
load
of
records
and
that
will
change
and
we
have
a
pluggable
architecture.
So
when
I
talk
about
flow
in
bed,
I
always
say
that
flow
in
bed
is
not
something
this
from
scratch,
because
we
took
about
a
okay
what's
wrong
fluently.
What
are
the
best
practices
of
flow
Andy?
A
And
while
we
can
do
better,
so
we
took
the
same
design
of
a
logging
pipeline
and
say
one
of
the
big
things
and
good
things
about
fluently
and
that
this
has
a
huge
ecosystem.
I'm,
not
sure
if
you
know,
but
we
have
like
800
plugins
and
our
company
may
be
maintained,
20
no
more
than
that.
So
all
of
the
780
plugins
are
made
by
different
individuals
and
companies,
and
all
of
that
is
possible
because
the
whole
login
pipeline
and
the
different
face
that
I
just
described,
italic
input,
filter,
parcel
and
so
on.
A
Are
plugins
so
in
any
case,
if
you
have
any
kind
of
a
problem
in
your
Barrowman,
what
you
do
is
to
just
write
a
plug-in
for
something
that
is
missing
to
solve
your
own
problem.
So
we
implemented
also
plugins
in
fluent
bit,
but
of
course
we
don't
have
800,
we
have
50
and
it's
not.
The
goal
to
have
800
I
think
that
the
goal
is
to
solve
the
problems
because
also
maintain
a
hundred
plugins
next
time.
So
we
also
care
about
security.
A
Basically,
we
have
a
one
single
thread,
one
main
process
and
some
people
will
say:
hey
why
you
don't
have
multi
process
we're
going
to
explain
that
later
and
we
have
different
input
plugins.
So
the
main
event
loop
is
in
church
to
coordinate.
You
can
think
about
that.
That
is
the
kind
of
we
have
a
scheduler
that
runs
on.
That
is
special
everlook
that
coordinate
the
whole
work
that
is
required
by
the
plugins.
So,
for
example,
input
plugins
can
say
you
know
what
I'm
listening
messages
over
TCP,
and
this
is
my
TCP
socket.
A
So
please,
let
me
know
when
you
have
a
new
message
so
who
coordinate
that
thing.
Saying:
hey,
there's
a
new
data
coming
for
you
and
pass
a
control
to
the
input.
Plugin
is
a
scheduler
who's
running
on
the
main
Ava
loop
right
after
some
plug-in
get
their
own
data
transferred
the
data.
It
passes
the
data
to
a
filters.
Well
it
actually
the
input
like
this
doesn't
know
about
filters.
A
They
just
ended
the
data
through
an
API,
but
that
API
does
filtering
and
filtering
is
filter,
plug-ins
that
take
care
about
two
parts
of
data
filter,
the
data
or
a
pencil
record.
Once
the
area
is
processed,
it
go
out
through
some
output
plugin
and,
of
course,
you
can
set
the
data
to
one
destination
or
multiple
destinations.
A
So
when
you're
doing
data
processing
influen
bid,
we
care
about
a
lot
about
okay.
We
cannot
assume
that
everybody
is
going
to
use
JSON,
because
that
is
a
lie.
Everybody
use,
JSON
log
format,
route,
text,
messages
and
so
on.
So
we
come
up
with
a
way
to
also
to
consume
all
those
looks
messages
and
provide
a
layer
to
process
all
that
data,
because
at
the
end,
in
your
data
analysis
face,
you
are
not
going
to
parse
all
row
text
messages
actually
you're
going
to
try
to
find
some
key
values
that
are
intercept
for
you.
A
For
example,
if
you're
parsing
a
Apache
web
server
locks
or
nginx
you're,
maybe
you're
interested
into
the
status
code
around
500
everything
that
was
an
internal
server
error,
but
you
don't
want
to
parse
the
whole
role
of
the
message,
because
maybe
that
message
has
I
don't
know
thousand
of
lines.
So
it's
maybe
just
to
focus
on
a
special
fraction
of
that
data
and
continue
person
of
that,
and
for
that
we
provide
backends
in
based
on
rejects,
JSON
LTS
B
and
the
new
one
is
log
format.
A
We
also
can
do
lookup
a
time
timestamp
resolution
based
on
the
log
files.
This
is
all
about
flow
and
bit,
but
this
is
base
also
and
if
fluency
support
is
for
years,
so
how
the
things
works,
an
input
plug-in
when
it
consumed
a
lock
message
or
whatever
it
generate.
What
is
called
a
record
and
a
record
is
like
a
small
array
that
has
two
fields:
why
of
them
is
a
timestamp.
A
That
means
when
this
message
was
generated,
it's
up
to
the
plugins
said:
a1
was
what
was
a
time
not
up
to
the
engine
and
then
the
message
that
the
message
can
be
a
map.
But
when
they
talk
about
message
array
I'm,
not
talking
about
JSON,
you
can
think
about
JSON,
but
we
use
something
that
is
called
message
pack,
which
is
like
a
binary
version
of
Jason.
So
message
pack
is
a
like
a
serialization
format
and
a
specification
that
has
implementation
in
different
languages,
for
CD
go
language,
a
Python,
Rus
and
so
on.
A
A
message
pack
was
created
by
the
flu
in
the
Creator
before
fluently.
A
message:
abacus
is
a
is
very
well
known,
and
they
think
is
that
you
have
a
binary
format
if
you
are
doing
going
to
do
some
parsing
or
you
want
to
iterate
the
fields
of
your
data,
you're,
not
going
byte
by
byte,
because
the
format
said:
okay,
you're.
Looking
for
the
key
number,
three
okay,
please
jump
to
the
offset.
I,
don't
know
100,
so
you
skip
parts
in
all
that
there,
so
you
just
jump
between
values.
A
Okay,
so
when
the
input
plug
in
and
generate
these
records,
maybe
we
have
this
dose.
Maybe
we
have
the
notion
about
tagging
because
you
need
to
add
some
notion
about
or
some
label
from
where
this
data's
coming
from,
because
maybe
we
have
input
plugins
consuming
from
the
file
system.
Maybe
we
had
an
input
plug-in
consuming
from
TCP
or
syslog
in
this
case,
so
we
said,
for
example,
he
put
plugins
generate
three
records
that
has
attacked
Apache
that
because
one
and
two
records
that
come
from,
we
attach
it
something
syslog.
A
So
what
happened?
Next,
these
messages
go
to
the
storage
layer,
a
storage
when
I'm,
saying
storage
means
that
okay
in
this
records
are
been
saved
somewhere
for
a
temporal
space,
maybe
in
memory,
maybe
in
the
file
system,
until
the
last
version
of
flow
embed,
because
today
we're
releasing
a
one
that
all
we
support,
just
a
memory
buffering.
Okay
and
that's
what's
quite
good,
because
it's
not
that
bad,
because
if
you
think
about
a
kubernetes
use
case,
what
is
your
buffer
of
data?
A
Your
buffer
of
data
is
your
log
files.
So
if
something
goes
wrong,
it
doesn't
matter
because
the
data
is
there,
but
what
happens
you're
consuming
data
that
doesn't
come
from
log
files
from
TCP
or
whatever.
You
need
to
be
able
to
continue
consuming
data
and
store
this
data
somewhere.
But
of
course,
your
memory
in
your
system
is
limited.
You
don't
have
like
gigabytes
free
to
do
whatever
you
want,
but
you
can
do
that
with
the
file
system.
A
So
once
the
data
get
into
the
storage,
then
it
jumped
into
the
Rhoden
phase,
meaning
where
is
this
data
going
to
for
roading?
If
you
remember,
we
had
this
notion
of
tagging
so
in
the
output
side
for
plugins,
we,
when
you
configure
this,
you
said
okay
I'm,
going
to
take
out
all
the
locks
or
messages
that
has
a
special
tag.
They
are
going
out
to
a
special
opal
destination,
and
for
that
we
have
this
notion
of
matching
rules.
A
So
right
now,
in
the
newer
version
you
can
do
matching
rules,
rules
with
will
card
or
with
regular
expressions.
If
you
want
so
a
one
of
the
challenges
that
we
had
when
we
were
creating
or
designer
flowing
bed,
we
said
eh.
Okay,
this
will
run
in
a
single
process.
We
get
the
data
in
and
we're
going
to
send
the
data
out,
but
most
of
the
places
where
the
data
is
going
out
is
network
operations.
If
you
have
been
done
some
programming
with
TCP,
that
means
create
a
socket.
A
A
So
the
thing
about
the
cold
box
here
is
like
if
you
have
one
event,
loop,
who's
handling
events.
People
creates
on
this
kind
of
notion
of
a
okay.
If
this
is
been
is
happening,
I'm
going
to
call
this
function,
if
this
another
event
is
happening
and
even
I
was
able
to
connect
to
the
server
I'm
able
to
write
data
to
the
server
those
kind
of
events.
But
when
you
write
a
this
kind
of
model
with
coal
box
is
quite
hard
because
you
are
trusting
that
the
developer
of
that
output.
A
Plugin
knows
what
he's
doing,
and
sometimes
they
don't
because
they
don't
know
what
would
happen
with
a
network
problem.
What
habits?
I'm,
writing
my
data
and
the
socket
gets
closeted
or
get
a
notification
from
the
kernel.
Do
we
can
leak
data?
We
can
leak
memory
or
we
can
get
many
problems,
so
we
come
up
with
a
design
where
we're
trying
to
reduce
in
the
blocking
time
in
the
process
and
try
to
defer
all
these
tasks
that
were
blocking.
So
this
is
like
the
common
thing
that
was
plainly
less.
A
If
you
went
going
to
switch,
send
data
to
a
destination,
we
basically
create
a
TCP
connection,
convert
our
message
back
internal
representation
to
the
drawing
format,
because
elasticsearch,
for
example,
is
not
waiting
for
a
message
back
it
suspecting
for
Jason.
So
I
need
to
convert
my
message
back
to
Jason
and
that
happens
in
the
output
plug-in
context.
Then
I
write
data
over
the
network,
wait
for
a
response.
A
200
was
ok
or
maybe
another
error,
and
after
that
the
road
or
the
engine
said.
Ok,
it
was
ok.
A
I
need
to
retry
this
or
there's
an
error.
So
how
do
we
solve
the
problem
here?
If
you
look
carefully
all
these
tests
about,
create
a
TCP
connection,
write
data
over
the
network
or
wait
for
a
response?
It's
not
something
that
we
can
ask
for
that
and
wait
until
it's
finished,
but
why
are
we
going
to
block
our
program
for
that?
So
that's
why
we
come
up
with
a
implementing
Cortines.
A
Maybe
if
you
are
using
golang
you're
quite
familiar
with
coroutines
or
on
a
way
to
suspend
execution
of
your
function
and
just
wait
that
somebody
wakes
you
up
and
restored
that
specific
entry
point
where
you
were
working
I'm,
going
to
explain
that
with
a
Cottam,
so
implementing
coroutines
in
the
output.
Plugins
was
a
solution
because
we
abstract
how
the
development
of
the
output
plugin
is
handling
the
connections.
How
is
read
in
data
and
so
on?
A
So
all
about
the
mental
operations
error,
the
shadows
and
that
kind
of
things
the
flu
embed
API
take
care
about
that.
So
we
we
reduce
the
risk
of
memory,
leaks
or
any
kind
of
problem,
so
how
this
works.
This
is
just
a
fraction
of
the
implementation
of
elasticsearch
plugin.
Well,
this
is
a
callback,
but
it's
a
callback
for
the
plugin,
for
the
registration
is
not
a
callback
hell.
So
if
you
look
carefully,
for
example,
on
line
8,
what
happens
here
is
we
are
creating
a
connection
to
the
TCP
server.
A
A
A
We
can
I
show
the
last
line
so
well,
anyways
the
after,
that
is
releasing
the
the
buffer
and
returning.
If
we
look
at
visually
at
this
code,
you
will
see
that
this.
Oh,
this
is
a
blocking
code,
but
internally,
every
time
that
you
issue
any
kind
of
network
operation
and
I'm.
Focusing
on
this
place
like
upstream
connection,
get
get
a
connection
to
an
upstream
server.
A
I
mean
create
a
socket
connect
and
be
ready
to
write
data
flow
and
bit
with
the
carotene,
is
going
to
suspend
execution
of
that
function
and
then
we'll
continue
working
processing
data
and
once
the
kernel
notify
a
the
connection
is
ready.
It's
going
to
resume
from
that
specific
entry
point.
So,
with
this
kind
of
model,
we
avoid
many
problems
with
NATO
Rio
or
leaking
data
or
whatever
and
of
course
a
at
the
beginning
is
not
too
straightforward
to
implement
the
design
internally.
A
But
after
that,
we
come
up
with
a
solution
where
you
can
write
your
own
plugins
in
a
safety
way.
Of
course,
you
can
leak
memory
if
you
don't
write
good
correct,
but
that
happens
to
every
person
and
ok,
that's
what
we
say
we
suspended.
While
the
connection
was
done,
we
resumed
later
and
then,
when
we
issue
the
HTTP
request.
That
means
I'm
going
to
send
my
payload
to
the
server
I.
A
Just
include
the
data
and
I
suspend
again,
because
I
will
tell
the
colonel
please
let
me
know
when
you
just
flushed
the
data
I'm
not
going
to
sit
and
wait
right,
I
had
things
to
do
and
then,
when
I'm
ready,
ok,
I'm
going
to
resume
and
the
internal
API
is
going
to
validate
the
status
and
so
on.
So
this
is
the
most
simple
example
of
how
this
works
internally
and
very
reliable
and
at
the
end,
a
plug-in
can
return
three
statuses.
First,
one.
A
Okay,
that
means
I
was
able
to
flush
the
rate
flush
in
the
data.
I
read:
try,
for
example,
if
I
go
to
prolene
and,
for
example,
here
a
return
ayesha
a
retry
write
that
retry
means
for
the
engine.
Oh,
something
happens:
the
plugin
is
asking
to
retry,
but
it's
up
to
the
engine
and
up
to
the
configurations
say:
ok,
you
are
allow
it
to
retry
again
or
you
are
not
allow
it
well.
A
If
it's
not
a
lawyer,
you're
not
going
to
tell
it
right
if
you
have
to
retry
I'm,
going
to
call
that
function
again
later
with
the
same
payload
of
data,
and
sometimes
maybe
you
can
get
an
error
like
you
know
what
I
was
not
able
to
format
the
data
for
elasticsearch
and
there's
nothing
I
can
do
so.
This
is
an
error.
Please
don't
try,
basically,
so
also
internally.
In
fluent
bit
from
a
developer
perspective,
we
have
many
helpers
to
do
different
operations.
A
A
All
of
the
plugins
are
built
in
into
one
binary.
So
if
you
love
Gollum,
because
you
have
one
binary
here,
we
have
one
binary.
Okay,
filtering
a
some
months
ago,
so
I
will
steady
story.
We
were
in
cube
con
Europe.
You
know
this
story
about
the
GDP
are
right
or
Security
blah
blah
blah
blah.
So
they
said
we
have
you
seen
flu
embed
in
production,
but
we
need
to
obfuscate
the
data
Oh
perfect,
why
you
don't
write
a
plugin
yeah,
but
we
need
that
our
plugin
does.
If
else
then
otherwise
do
this
thing.
A
A
They
said,
ok,
why
we
don't
implement
Lua
scripting,
so
we
implement
a
Lua
filter
that
will
call
a
Lua
script
that
you
can
load
in
your
own
configuration
in
your
own
config
map,
so
you
can
replace
values,
make
conditionals
or
whatever
you
want
in
a
very
easy
way
like
this.
This
is
a
simple
function
that
take
the
track
of
the
record,
the
timestamp
and
the
record,
and
basically
just
create
a
new
entry
assign
some
values.
I
just
returned
this
created
a
new
record
into
flow
in
bed
with
a
I.
A
A
A
Okay,
look
processing
in
couple
midis
I'm,
going
to
explain
really
quickly
how
this
works.
If
you
a
low
processing
company,
this
means
that
you
have
pots
that
are
run
in
your
notes
and
every
pot
had
can
have
many
containers,
but
the
logs
are
physically
in
your
note,
because
that
whole
docket
operates
right
by
default.
We
are
using
the
file
system
to
store
the
logs,
so
the
pots
or
the
containers,
the
application
that
runs.
A
Of
course
you
have
to
bomb
onto
that
volume
and
once
it
consumed
the
logs,
you
know
that
in
kubernetes
you
have
the
notion
about
labels
or
annotations,
so,
but
those
labels
and
annotation
are
not
in
the
node
context.
They
are
in
the
API
server
outside.
So
what
does
our
filter
does,
which
is
called
free
to
communities
which
was
a
strict
copy
of
the
flu
in
different
communities?
It
just
go
to
the
API
server
guard
at
the
metadata
and
then
just
put
everything
together
for
you.
So
you
can
do
your
own
data
analysis.
A
Please
show
me
all
the
logs
in
I
don't
know
elastic
all
that
this
label
blue.
You
work
all
together.
It
doesn't
matter
if
you
had
10
replicas
of
that
application
is
due
to
be
read
in
20
nodes,
but
it
will
work.
So
a
simple
message
like
this
was
generated
from
a
docker
perspective,
so
the
application
just
said:
hey
CRO,
but
in
docker
at
the
file
system
level
you
get
the
lock
messages,
the
stream
from
where
this
data
came
from
and
the
time
Stan
when
this
data
was
generated.
A
A
So
a
one
of
the
features
that
we
implemented.
I
think
that
was
this
year
a
well.
What
what
is
the
problem?
The
problem
is
sad.
Ok,
everybody's
deploying
pot
right
application,
every
application
has
a
different
format
and
there
was
no
way
in
the
login
processor
side
to
say.
Oh,
this
data
is
this
format.
This
pod
is
this
format,
so
we
come
up
with
the
solution
that
why
we
don't
allow
to
the
guy
who's
deployed
spot
to
say
it
to
suggest
a
parser
for
their
own
data.
A
So
we
implemented
this
annotation
inside
the
pot
were
fluent
bed
when
stuck
into
the
API
server
is
getting
all
the
data
and
all
the
annotations
for
that
pot.
That's
running
so
and
also
before
to
merge
all
the
data.
It
just
look
for
this
special
annotation
flow
and
bid
array
of
such
person
and
said
Apache
and
for
flow
embed.
Apache
is
a
person
that
is
known
in
the
configuration,
so
it
knows
how
to
parse
Apache.
A
So
you
don't
mess
up
with
a
whole
row
text
messages
in
your
database
and
also
we
got
contributions
in
this
year
from
Don
who's
here
in
the
audience
who
he
expanded
this.
So
one
thing
was
to
suggest
a
person,
but
maybe
your
pod
can
have
many
containers
and
containers
stream
data
from
standard
output,
standard
error
and
trust
me
that
there
are
people
who
put
different
formats
in
the
same
container
for
different
streams,
so
yeah.
A
This
was
implementation,
where
you
can
mix
up
the
options
and
try
to
accommodate
how
from
your
pot,
you
are
going
to
a
suggest
a
apart,
sir,
or
maybe
you
can
say,
please
exclude
the
logs
from
this
pot
because
maybe
our
pots,
they
are
from
testing,
they
are
too
noisy
and
you
don't
want
to
have
that
data
in
your
database
we
implemented.
Also,
this
is
not
new
but
primitive
monitoring,
so
we
have
a
built-in
metrics
in
JSON
and
in
parameters
format.
A
So
you
can
who
you
can
put
your
own
login
layer
and
you
can
put
your
own
primitive
separator
consuming
those
metrics
for
you,
okay,
so
fluent
bit
one
that
or
release
it
today,
a
flume
will
weather,
though
we
can't,
we
counted
the
contributors
from
the
last
suit
at
14
BER
version
up
to
now,
and
that
is
more
than
30
people.
So
thanks
so
much
for
contributing
these
individuals
and
companies
a
we
also
from
an
ecosystem
perspective
and
contributors
are
focusing
on
how
to
improve
testing
and
CI.
A
So
when
a
project
grows,
when
a
thing
start
to
grow
in
terms
of
extension
architectures
and
for
the
next
year
portability,
we
need
to
have
a
way
to
make
sure
that
the
things
that
we
are
doing
and
running
right,
okay,
that
is
all
they
see,
I-
think
a
there
are
integrated
sanitizers
now
for
memory
for
memory
leaks
and
also
as
a
just
some
news
a
since
March
of
this
year.
Until
now
we
have
around
18
million
deployments.
Well,
it
is
seventeenth,
that's
something
almost
18
and
that's
a
lot.
A
So
every
day
we
got
like
a
hundred
thousand
deployments
in
kubernetes,
because
somebody
is
running
a
node
or
spinning
up
new
Bien
four-day
cluster
they're
using
flow
in
bed
or
flu
India
for
flu.
Indeed,
we
have
their
own
matrix
to
its
a
couple
of
million
more
so
here
we're
focusing
on
flu
and
bed
metrics
so
and
I
think
that
after
look
at
the
metrics
and
the
contributors,
it
was
a
really
good
timing
to
release
that
one
that,
oh
because
it's
really
being
used
in
production.
A
So
what
is
coming
up
in
this
one
that
all
version?
One
of
the
things
that
we
have
a
new
storage
and
buffering
mechanism.
So
when
I
said
at
the
beginning,
that
barfing
is
really
important
a
by
default,
we
always
use
memory,
but
now
we
have
a
way
to
store
the
data
in
memory,
but
also
in
the
file
system.
Just
in
case
something
happens,
we
have
a
backup
of
that
data,
the
matrix
in
the
metric
side.
The
filters
didn't
have
a
matrix,
but
now
they
have
it.
A
We
have
a
new
person,
log
format,
back-end
a
we
have
an
improvement
company.
This
filter
we
probably
blew
a
filter.
We
have
a
new
bigquery
plugin,
but
there
are
many
changes
that,
of
course,
we're
not
going
to
list
here
because
there
are
bug
fixes
on
most
of
the
plugins.
We
have
performance
improvements,
improving
documentation.
That
is
really
important.
A
So
talking
about
the
storage
later
pretty
quickly
and
we
started
to
design
in
a
way
how
to
sort
or
have
a
better
notion
of
how
the
data
is
handled
in
certain
internal
influent
bed,
but
how
to
come
up
with
a
representation,
a
diagnostic
representation
in
the
file
system.
So
we
come
up
with
that.
This
will
be
a
root
pad
right
where
you're
going
to
store
different
streams
of
data.
This
is
the
first
layer
is
a
diagnostic
library,
but
it's
being
used
by
flow
embed.
A
A
Okay
and
each
chunk,
has
this
a
decide
pretty
quickly?
We
have
a
here's,
two
bytes
optionally,
it's
disabled
by
default,
but
you
can
enable
crc32
for
checksum
just
in
case
you
care
a
lot
about
data
integrity
and
you
can
come
up
with
a
high
I/o
you
can
enable
and
in
the
content
side
of
the
that
same
file,
we
have
a
space
for
metadata.
We
have
two
bytes
to
specify
the
length
of
the
metadata
and
for
metadata
for
flu.
A
Embed
is
attack,
so
every
file
chunk
in
the
stream
in
the
file
system
contain
the
tag
plus
the
data,
and
then
we
have
the
content.
So
we
need
a
lot
of
a
testing
and
performance
test
over
this
design
and
well.
This
is
the
final
design.
We
did
a
lot
of
iterations.
We
started
with
a
such
a
256
for
checksums,
but
was
really
expensive
in
terms
of
computing,
but
I
think
that
this
solve
the
problem
so
yeah.
That
was
the
presentation.
I
hope
you
are
using
flowing
bead.
A
Okay,
the
question
is,
if
you
I'm
getting
started
what
we
recommend
flew
in
the
airflow
embed,
a
fluent
d800
plugins
and
800
updated
in
the
last
year
through
embed
50
plugins.
So
what
are
plugins
are
different
kind
of
features
or
things
that
maybe
you
depends
on
if
you're
running
kubernetes
and
you
need
special
filters
that
or
input
plugins
or
oper
plugins
that
doesn't
exist
in
fluent
bed.
You
should
go
with
fluent
ii-if
wendy
is
too
heavy.
A
Maybe
you
should
consider
flow
in
bed,
but
also
what
some
people
does
is
put
flow
in
bed
as
an
agent
in
the
node
as
a
team
onset.
But
then
they
use
a
flu
indeed
to
aggregate
the
data,
so
you
make
fluent
bedrock
to
fluently
because
both
can
talk
each
other.
But
there's
not
like
a
straight
rule
for
that,
because
every
use
case
is
different.
So
the
thing
is
that
a
we
thought
some
years
ago:
okay,
somebody's
going
to
make
something
for
cloud
native,
more
lightweight
or
we
do
it
or
somebody
else
will
do
it.
A
A
The
difference
to
write
plug-ins
for
fluency
I'm
fluent
bit
for
flow
nd.
You
just
write
plugins
in
Ruby,
so
I
saw
about
ruby,
gems,
so
I
think
that
is
is
quite
straight
forward,
but
for
fluent
bed
there
is
a
complexity,
because,
as
of
today,
the
plugins
are
in
C
language.
One
and
two
is
static.
A
link
right
now
we
are
going
to
play
main
share
plugins
such
a
library
for
plugins.
A
So
you
can
write
your
own
plugins
outside
of
the
binary
and
just
link
it
on
grunting,
but
in
fluent
bit,
I
didn't
mention,
but
we
support
plug
is
in
go
link,
so
you
can
read
your
plugins
and
go
along,
but
there's
not
an
issue,
but
it's
a
performance
penalty
in
the
integration
in
the
golden
world.
When
you
have
your
C
code
running
and
you're,
going
to
call
this
goal
and
code.
So
the
golden
code
includes
all
the
things
that
goal
and
needs
to
run.
A
So
it
does
a
context,
switch
which
is
really
expensive
in
terms
of
computing,
but
all
the
ng,
the
film
bed
is
shut
it
down
until
the
goal
and
function
returns.
So
if
it's
a
pivotal,
this
guys
are
using
a
syslog
and
they
create
a
syslog
plug-in
in
going
it's
working
for
them.
But
I
think
that
if
you
have
really
intensive
load
while
you're
cheap
in
thousand
of
logs
per
second,
maybe
you
can
face
some
performance
penalties,
but
I
think
that
it
depends
of
the
use
case
and
and
the
load
that
you
have
in
your
system.
A
A
Yeah
they
in
the
in
for
one
influen
bed
and
doesn't
support
TLS,
just
the
open,
yeah
isn't
the
road
mad.
But
honestly
we
didn't
have
any
priority
before,
because
we
didn't
have
the
storage
layer
in
the
file
system,
because
I
think
that
that
was
more
relevant
or
wise,
you're
consuming
data.
Your
memory
will
go
up
and
everybody
will
complain
about
it
right,
but
now
that
we
have
the
file
system
buffering,
we
don't
have
excuse
to
not
implement
it.
So
that
is
something
that
we
have
to
do
this
year.
Yep.
A
A
A
Yeah
the
question
is
a
the
the
output
Kafka
plug-in
that
we
have.
It
doesn't
support
dynamic
topics:
okay,
that
was
a
limitation
that
we
have
in
the
backend
library
that
we
used
leave
our
takaka,
where
you
have
to
did
pretty
find
your
topics.
I
think
that
that
is
solved
in
some
way,
but
I
need
to
do
some
research,
because
I
know
that
we're
getting
a
lot
of
feedback.
A
A
A
A
Yeah
yeah
this
a
truffle
plug
a
filter,
so
you
can
limit
the
window
and
you
can
set
a
window
or
the
amount
of
data
that's
flowing
through,
but
I
think
that
there's
no
way
to
do
it
through
a
namespace
level
and
I'm
sure
that
everybody
needs
that
yeah,
because
most
of
the
multi-tenant
environments,
that
with
people
haven't
talked
it
to
that
the
implement
multi-tenant.
They
say.
A
No,
we
split
the
things
with
namespace,
so
we
need
to
provide
a
context
of
namespace
or
how
to
get
that
information
to
to
apply
all
the
rules
before
we
continue
with
the
latest
question:
a
we're
going
to
have
an
informal
short
meeting
outside
for
everybody
who
wants
to
contribute,
or
has
some
a
production
usage
of
fluent
bet,
because
we
are
going
to
try
to
handle
the
priorities
in
a
better
way
for
because
you're
running
in
production.
So
we
want
to
know
from
your
perspective.
What
do
you
need?
A
A
Do
you
develop
in
:
yeah?
No,
it's
not
in
the
roadmap
to
have
go
along
input
plugins,
but
if
you
want
to
write
the
interface
you're
welcome
yeah
another
option
is
like
a
you
know
what
can
be
done.
It's
like,
for
example,
for
the
output
plugins.
We
we
build
a
goal
and
output
package
so
for
fluent
bed.
A
So
in
the
input
side,
what
makes
sense
may
be
to
write
a
simple
package
that
connects
over
a
UNIX
socket
to
fluent
bed,
so
anybody
can
write
their
own
a
thing
to
collect
data
and
use
a
UNIX
socket
interface
through
your
own
framework
to
ingest
the
data
I
dunno.
This
is
straightforward.
It's
not
too
hard!
If
you
want
to
contribute
with
that,
we
can
give
you
the
right
mentoring
to
to
get
there
yeah
anybody
else.
That's
one.
A
The
the
best
way
to
get
help.
That
is
some
going
conversation
that
we
have.
Actually
we
have
channels,
but
we
need
more
people
to
help.
That's
the
thing,
but
we
have
slack.
We
have
a
slack
channel.
We
have.
The
mailing
list
sometimes
is
something
that
we
need
to
improve
for
the
next
year
to
be
more,
you
know
proactive
in
communication,
but
we
have
the
channels
Twitter
main
list
and
slack
what
you
can
send
us,
an
email
anytime
to
bothered
us.
Oh.
A
B
A
Yeah
we
have
support
in
file
buffer
right
now,
yeah,
that's
the
new
thing
yeah.
Of
course,
it's
just
implemented.
I
think
that
nobody
is
running
in
production,
but
you
can
be
the
first
or
in
testing.
So
I
always
say
that,
because
sometimes
people
announce
new
features
in
the
products
and
sometimes
things
are
not
quite
stable,
so
I'd
say
we
did
our
tests
with
you.
Pooi
did
the
search
late
as
a
separate
project,
so
we
make
sure
that
was
working
has
a
good
performance,
but
when
you're
running
production,
the
things
different.
A
So
we
need
feedback
on
that
is
just
implemented,
but
maybe
we
need
some
iteration
of
clicking
performance
improvements
or
I.
Don't
know,
okay,
I
think
that
we
aren't
on
lunchtime
now
right.
So
maybe,
if
you
want
to
have
this
quad
meeting,
we
can
have
it
here
because
there's
nobody
using
this.
Okay.
Thank
you.