►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Deep Dive Fluent Bit: Logging & Stream Processing - Eduardo Silva, ARM Treasure Data
Fluent Bit is a Fluentd sub-project that aims to solve hard data challenges in the cloud space. On this deep dive session, we will talk about its architecture, how data workflows operate and the ability to perform advanced data transformation. Also, we will demonstrate the new ability to perform Stream Processing on the Edge.
https://sched.co/MPjW
A
Hello,
everyone:
how
are
you
doing
fine
I
assume
that
that
is
good
good
land?
My
name
is
Eduardo
Silva
I'm
a
principal
engineer
at
arm
and,
as
you
know,
we
are
in
the
deep
dive
session
of
flew
indie
project
and
this
deep
dive
session
will
focus
on
the
flu
and
bed
project,
which
is
a
sub
break
of
the
flu
in
the
ecosystem.
A
As
I
said
I'm
engineer
at
arm,
we
used
to
be
treasured
data,
the
company
who
created
flu
Indy
and
I'm
a
maintainer
of
this
project,
which
is
called
flowing
bed.
First
of
all,
do
we
have
any
flu
and
bed
users
here?
Please
raise
your
hand.
Oh
that's
good.
So
I
think
that
I
will
owe
you
some
stickers,
t-shirts
stuff.
A
Well,
a
we
will
cover
something
that
is
really
new
here.
It's
about
called
stream
processing,
but
before
jumping
to
that,
we
are
going
to
get
to
the
basics
of
logging.
When
application
generates
a
message.
That
message
is
composed
as
a
record.
That's
like
the
technical
term,
internal,
either
influenza
or
flu
in
bed,
and
that
record
comes
is
not
just
a
message.
It
also
comes
with
some
metadata
and
that
metadata
means
the
timestamp
of
when
this
message
was
created,
plus
the
same
message
that
now
this
toy
is
going
to
be
sterilized
it.
A
A
So
an
application
generate
a
message
that
message
becomes
a
lot
that
low
close
to
the
file
system,
maybe
over
the
network
and
in
the
other
side
we
have
to
embed
generating
a
record
with
this
information
that
becomes
a
binary
record
then
inside.
In
the
internal
part,
the
records
can
become
our
group
right,
for
example,
if
you
are
consuming
logs
from
an
input
section,
and
this
input
section
means
a
couple
of
log
files.
Maybe
these
log
files
a
has
many
information,
and
you
want
to
group
them
by
tags.
A
So
you're
saying
all
of
these
messages
are
coming
from
Apache
or
all
of
these
other
messages
are
coming
from.
Syslog
and
why
grouping
them
is
really
important
when
you
want
to
do
login
stuff,
because
you
need
to
sort
and
have
some
order
for
these
logs
because
you're
you're
in,
as
we
said
in
the
keynote
yesterday.
Well,
it's
not
logging
log
in
as
the
same
any
time
is
boring.
But
what
you
want
to
do
in
reality
is
data
analysis
for
troubleshooting
learning
or
whatever.
A
Once
the
data
of
the
records
goes
to
the
storage
phase,
this
can
be
either
in
memory
or
the
filesystem.
It
goes
to
the
rotor.
The
rotor
basically
said:
where
are
these
records
going
to
be
delivered
and,
as
you
can
see,
you
can
define
mayn't,
multiple
output,
plugins
or
configurations
output
from
where
you
would
like
to
ship
these
locks
out
and
where
you're
going
to
ship.
A
It
come
your
elasticsearch
database,
your
cloud
service
or
whatever,
and
if
you
look
carefully,
the
rotor
does
a
really
special
job
here,
because
it
matched
the
tag
from
each
record
with
a
match
section,
define
it
in
the
configuration,
so
the
rotors
say:
okay,
all
this
bunch
of
record
had
this
tag,
I'm
going
to
look
for
the
matching
rules,
so
this
data
is
going
to
this
place
or
maybe
that
same
data
is
going
into
multiple
places.
That
can
be
done
too
without
any
problem
in
the
community
space.
A
It's
like
this
I'm
sure
that
most
of
you
using
kubernetes
or
are
familiar
with
docker,
but
in
kubernetes
we
have
the
concept
of
the
master
api
server
and
the
node
in
a
cluster.
We
have
many
nodes
right,
but
if
you
have
applications
deployed
in
a
node,
you
need
to
have
your
login
processor
to
be
able
to
consume
the
login
information
from
the
other
ports
or
containers.
So
what
we
do
is
that
we
deploy
fluent
pip
or
flew.
Indeed
it
doesn't
matter
what?
Where
is
the
case?
A
A
So
with
one
bit
one
that
one
release,
we
focus
on
two
major
things:
one
was
Windows
support
and
you
can
realize
that
many
windows
service
running
and
many
people
has
been
asking
for
Windows
support
windows
support.
Maybe
people
is
not
really
interested
to
do
a
huge
processing
in
Windows
environments,
but
they
want
take
out
the
logs
out
of
the
box
to
a
UNIX
box
for
processing
irrigation
or
whatever.
So
we
have
Windows
support
in
beta.
A
So
if
you
are
a
Windows
user,
you
are
administrating
windows
services,
please
a
built
from
dip
from
that
it's
in
a
beta
beta
phase,
so
any
feedback
you
can
provide
us
is
really
important
and
the
second
part
is
a
stream
processing.
So
what
is
the
stream
processing
who
can
describe
a
stream
processing?
Please
raise
your
hand,
let's
try
to
make
it
more
more
friendly.
It
doesn't
matter
if
you're
wrong
or
you're
right,
maybe
I'm
wrong.
But
what
means
for
your
stream
processing,
leh.
A
A
From
the
data,
as
far
as
I
get
it
into
my
pipelines
perfect,
so
this
is
not
just
streams
of
data,
so
estranged
data
can
be
anything,
but
the
stream
processing
is
ability
to
transfer
the
data,
while
the
data
is
still
in
motion.
Okay.
So
if
you're
going
to
process
that
in
your
database,
your
data
already
was
collected
processes,
aggregated
indexed
set
in
the
database
and
then
you're
doing
queries.
But
here
we
are
talking
about
the
stream
processing
on
the
edge
side.
A
Who
of
you
are
using
elasticsearch.
Please
raise
your
hand,
okay,
so
all
of
you
use
elasticsearch,
not
just
to
use
Kabana
have
fancy.
You
know
works
right,
you're
using
it,
because
you
need
to
index
your
data.
You
need
to
query
your
data,
but
I'm
sure
that
at
some
point
a
most
of
you
are
not
quite
happy
with
the
performance
or
the
results.
So
the
performance
of
your
results
and
not
because
of
elastic
search
because
indexing
expensive
right
need
to
get
the
data
go
to
the
database.
A
They
can
attach
flash
blah
blah
blah
blah
blah,
and
sometimes
you
need
to
get
some
feedback
loop
faster
than
indexing,
so
stream
processing
on
the
edge.
It's
an
ability
to
have
data
selection,
filtering
by
patterns
perform
data
aggregation,
okay,
this
sounds
familiar
because
fluid
the
effluent
bit
already
kind
of
doing
that-
or
maybe
this
is
kind
of
you
know-
create
new
streams
of
data
based
on
query
results.
A
Maybe
this
sounds
familiar
to
people
who
ingest
data
into
character
into
a
half
storage,
then
pull
this
data
into
an
with
analog
processor
and
create
a
new
stream
based
on
results.
But
here
we
are
talking
about
a
Java
stack
right.
If
we
talk
about
lassic
search,
you
means
Java.
If
you
talk
about,
casca
is
Java,
so
sometimes
we
need
something
more
lightweight
and
something
that
can
provide
us
data
as
full
as
soon
as
possible.
As
something
is
happening.
A
A
No,
maybe
in
two
into
two
seconds
after
the
indexing
took
ten
minutes
right,
but
that
it's
not
ideal
and
the
point
of
this
talk
is
not
to
blame
any
database,
because
that's
not
the
point,
but
the
point
that
indexing
takes
time.
Indexing
is
expensive.
We
cannot
get
rid
of
indexing
because
we
need
it,
but
also
we
we
can
improve.
How
that
how
do
we
operate
and
create
queries?
A
So
all
of
this
is
about
performance
performance
in
technical
terms,
in
computing
time,
performance
in
networking,
bandwidth
and
also
performance.
In
how
fast
do
I
get
my
data
back
and
in
login
in
general,
we
have
many
pains.
Okay
accessing
the
file
system
is
expensive.
It
doesn't
matter
what
kind
of
storage
you
have
well
phone
data
parsing
is
even
more
expensive.
You
know
we
are
not
dealing
with
binary
data.
Usually
if
you
have
a
JSON
map
or
a
JSON
log
that
is
really
expensive
to
parse.
A
Okay,
it's
not
magic,
it's
expensive,
because
you
need
your
parser
to
go
by
by
by
checking.
Oh,
this
is
a
map.
This
is
an
array.
Did
it
end,
they
didn't
know,
maybe
have
a
start
to
try
to
check
if
this
is
a
valid
or
not,
and
then
you
have
your
database
indexing,
so
expensive,
expensive
plus
times
expensive,
and
then
you
have
your
data
okay.
A
So
we
work
on
this
area
on
flu
in
the
affluent
bit
we
are
flexible.
We
are
really
fast,
both
can
integrate
each
other
or
we
can
work
that
as
a
standard
on
solutions
and
that's
perfect.
But
what
can
we
do
in
the
other
scenario,
to
help
to
reduce
the
load
on
the
database
for
some
queries
that
we,
maybe
we
can
do
on
this
side
on
the
edge?
A
So
we
are
going
to
do
a
really
quick,
a
basically
I
have
1
million
reports
in
just
one
locked
file.
It's
a
hundred
megabytes
file,
but
anyways
is
expensive
and
every
line
is
in
JSON
format
and
I'm
going
to
process
the
data
and
index
it
in
a
database.
Please
guess
how
much
time
it
will
take
right.
We
can't
speak.
A
A
As
I
said
many
times,
don't
do
this
at
home.
The
delete
command
is
really
dangerous,
plus
with
all
okay
I
just
wiped
the
database.
Okay
I'm
using
Coral
a
common
line
to
query
elasticsearch
I'm
not
going
to
use
Cabana,
because
this
point
doesn't
make
sense.
Okay,
so
where's
my
file,
my
file
is
called
cube.
Conn
espied
that
look.
A
Maybe
we
can
do
something
like
this:
oh
I'm,
going
to
break
it.
So
every
record
has
like
this
a
date.
An
IP
address
a
word
and
a
country
name
a
flag
and
a
number
between
1
and
at
100.
So
it's
nothing
complex.
If
you
can
realize
the
record
is
quite
small.
So
what
I'm
going
to
do
is
to
process
this
log
file
with
flu
embed
and
I'm,
going
to
send
it
to
elasticsearch
so
and
please
try
to
come
up
with
to
realize
how
much
time
that
could
take.
A
A
Okay,
what
we
care
about
is
the
number
of
documents
that
we
have
in
database.
We
have
nothing
now
and
going
to
my
full
and
bit
configuration
I'm
going
to
delete
that
did
some
cleanup
okay.
This
is
a
setup
friendly
for
elasticsearch,
okay
I'm
going
to
do
something
I'm
going
to
be
a
really
bad
logging
tool
that
is
going
to
process
the
data
really
fast
and
send
that
every
fast,
very
fast
to
the
database.
Right
no
limits,
and
here
I'm,
going
to
send
the
data
to
elasticsearch
for
all
records
in
logstash
format.
A
Because
our
file
has
a
million,
so
we
sent
a
4000
and
it's
stuck
so
what
about
here?
Let's
try
to
reach
that
erase
oops
I,
don't
get
a
response
from
indicted,
Alice
I'm
running
local
fuzz
right,
but
I'm
not
doing
anything
like
crazy
Oh.
Here
influent
bit,
I
can
cut
my
connection
rejected.
That's
why
we
get
a
ticket
in
our
github
say:
hey
my
data
slows
I
cannot
flush
the
data.
Ok,
we
need
to
do
some
troubleshooting,
but
here
I'm
not
able
to
correct.
A
It's
question
and
we
sent
how
many
records
for
well
5,000,
so
in
the
cloud
space
tanning
and
buffering.
Is
everything
I'm
not
saying
that
elasticsearch
is
bad
I'm
saying
that
if
you
send
adjust
a
little
bit
of
load,
you
can
crush
things
well,
but
our
golden
this
talk
is
not
to
crush
the
things
right.
So
we're
going
to
properly
stop
elasticsearch
and
we're
going
to
study
the
game
but
I'm
going
to
stop
fooling
bit.
So
it
would
stop
sending
data
and
I'm
going
to
wipe
the
database
again.
A
Okay
database
is
empty.
Now
we
are
going
to
be
a
good
citizen,
we're
going
to
apply
this,
which
is
called
memory
buffer
limit.
So
we've
got
a
problem
that
when
we
send
a
low
process
of
flu
in
the
affluent
bits
and
data
too
fast,
but
that
is
not
a
problem.
The
problem
says
sometimes
database
cannot
process
all
the
data.
A
A
Okay,
we
are
doing
better,
as
you
can
see
in
this
number
here,
then
the
number
of
documents
is
going
up.
What
we
did
is
just
to
put
some
limit
some
kind
of
threshold
in
the
fluent
bit
to
send
the
data
more
slowly,
but
not
slowly,
but
again
we
sent
one
million
and
we
are
having
like
10
seconds,
15
seconds
and
so
on,
and
we
have
not
done
any
query.
Just
indexing.
So
I
think
that
you
got
my
point
okay.
A
A
A
So
how
much
time
did
it
take
1
minute
or
more
so
here
we
have
some
challenges.
Why
I'm
carrying
too
much
about
data
indexing?
The
indexing
allows
you
to
query
your
data
faster,
that's
the
whole
goal,
but
if
you
need
you
want
to
query
your
data
and
indexing
is
becoming
a
blocker,
for
you
we
need
to
think
is
something
different,
and
that
is
flu
embedded
with
stream
processing
capabilities.
We
was
releasing
in
just
the
last
week,
but
what
you
can
see
there
is
the
log
in
pipeline.
A
So
what
we
did
is
like,
after
the
input
site
where
we
collect
the
data
we
parse
the
data
filter.
We
have
the
storage
side
before
to
wrote
the
data
out.
What
we
do
we
put
a
new
engine
from
a
stream
processor
and
the
stream
processor
can
do
a
lot
of
queries,
and
this
is
really
good
because
we
do
it
with
SQL
straight
to
query
range
language,
so
SQL
used
to
be
boring,
but
nowadays
I
think
that
is
quite
fun,
but
you
say,
but
what
do
you
put
any
database
on
fluent
8
or
so?
A
A
So
let's
do
us
a
second
test.
I
had
the
same
file,
but
I
just
want
to
retrieve
the
fields
called
IP
country
unknown,
which
are
the
JSON
keys,
but
with
the
conditions
where
the
field
num
is
greater
than
80
and
the
field
country
equals
to
chill.
So,
if
I
want
to
do
that
with
a
database,
it
would
take
a
minute
or
more,
which
are
very
load
load.
But
how
can
I
accomplish
the
same
thing,
but
with
fluid.
A
We
are
going
to
do
something
here.
Let
me
check
something:
okay,
right
now
we
are
not
going
to
ship
the
data
to
a
database
just
to
the
standard
output,
but
from
the
engine
so
I'm
going
to
modify
my
output
to
know
and
I'm
going
to
increase
my
buffers
for
reading,
because
I
want
to
become
faster.
We
don't
need
me.
A
A
There
you
go
so
you
start
getting
the
data
right
out
of
the
box
in
your
tamela.
Well,
that
is
useful
for
debugging
purposes
right.
You
cannot
do
anything
with
that,
but
you
already
got
what
you
want
it.
Well,
it
got
all
the
records,
so
it
worried
the
1
million
in
just
a
fraction
of
seconds.
That
is
what
we
are
trying
to
achieve
with
the
strain
processing
on
the
edge
reduce
the
number
of
CPU
that
you
have
to
invest
in
your
cloud
side,
but
defer
it
to
the
H.
A
A
These
results
were
sent
to
the
standard
output,
but
now
what
we're
going
to
do
is
create
a
new
stream
of
data
in
fluent
bit
using
the
results
of
the
string
processor.
Are
you
familiar
with
K
sequel
of
Carta,
some
of
you?
Well,
so
we
will
do
create
stream.
M
cube
con
s,
so
the
new
stream
of
data
will
be
called
cube
con
and
all
the
data
on
that
stream
will
comes
from
the
output
of
the
query,
but
we're
going
to
do
something
better
with
do.
You
know
that
we
talked
about
tagging.
A
A
Go
to
my
fluent
bit
configuration
I'm
going
to
send
the
data
where
elastic,
search
and
match
everything
by
we're
going
to
match
only
results,
because
we
care
just
about
the
data
that
we
just
process
it.
We
don't
care
about
the
others
and
yeah
we're
going
to
use
the
non
friendly
set
up.
Maybe
it
works,
maybe
not
I,
don't
know.
So.
Are
we
good
now?
It
should
work.
Okay,.
A
That
is
normal,
and
that
happens
every
day.
So,
okay,
let's
give
it
a
try,
so
it
will
take
the
same
amount
of
time
to
ingest
the
data
in
the
standard
output,
but
to
elasticsearch.
So
here
we're
just
getting
the
final
result
and
it
ended
up
on
60000
because
that's
a
result
of
our
query,
we
don't
get
to
one
second,
as
the
lady
won
it
right,
but
we
didn't
in
three
or
five
seconds.
So
with
this
approach,
we
can
extend
too
many
possibilities.
A
Do
you
use
in
my
sequel,
post-acute
from
the
command
line,
one
of
the
things
that
we
are
well.
Our
thoughts
is
okay,
how
we
can
implement
some
client-server
model
on
this
in
mind,
the
URL
in
your
cluster,
your
notes,
you
don't
know,
what's
happening
with
your
data
on
certain
nodes
for
some
type
port
or
some
type
container.
Imagine
that
you
can
connect
remotely
to
flu
and
paid
directly
today
to
the
string
processor,
create
your
query
and
get
your
data
remotely
you're
skipping
a
database
you're
skipping
everything.
So
the
possibilities
here
are
many.
A
Okay.
So
this
is
just
one
of
the
use
cases,
but
there
are
other
use
cases.
Do
you
know
the
concept
of
data
aggregation
for
simply
have
a
table
of
the
range
of
relational
database,
and
you
want
to
get
some
an
average
value
of
certain
fields
that
has
so
much
in
criteria
and
so
on.
Let
me
show
this
I'm
going
to
split
a
configuration
here,
we're
going
to
prove
this,
for
example.
Just
so
you
can
get
a
notion
of
what's
going
on
here.
A
A
A
Exactly
we're
going
to
select
okay,
we
we
are
going
to
spray
in
a
different
way.
We
are
going
to
create
a
window
of
time.
That
means
every
20
seconds
calculate
I.
Get
me
the
whole
records
that
the
country
is
Chile
and
group
by
country,
but
calculate
the
average
number
of
all
of
them,
where
this
use
case
a
what
is
useful,
IOT
embedded
gateways.
You
have
a
bunch
of
sensor.
Do
you
want
to
send
thousands
of
vehicles
to
the
cloud
to
calculate
something?
What's
the
average
temperature
in
the
at
home
in
the
last
week?
A
No,
maybe
you
can
do
the
calculation
on
the
edge
and
just
send
out
the
results
pretty
much
as
we
did
with
elasticsearch
some
minutes
ago,
so
I'm
going
to
did
I
set
it
now.
Oh
cool
yeah.
We're
going
to
fix
this
because
a
in
the
configuration
flowing
bit
is
very
strict
everybody's,
just
one
line,
but
I
think
that
we,
this
use
case
is
is
painful
but
anyway,
so
our
stream
here
is
called
results.
A
A
I
should
have
put
10
right.
Well,
there
we
go
so
we've
got
the
record
output
irrigate
it
and
they
average
number
between
20
seconds.
We've
got
1
million
records,
so
we
process
it
all
of
them.
We
look
it
up
for
Chile,
we
did
some
aggregation
window
and
the
average
number
is
50
for
all
of
them.
So
these
kind
of
things
are
very
flexible.
What
enough
futures
are
missing
at
the
moment,
for
example,
join
streams.
If
you
have
multiple
streams,
maybe
you
want
to
do
some
query
with
a
join
between
them.
A
B
So
inside
the
square
you
just
showed
us
there's
this
time
window.
Sometimes
you
wanna
backfill,
for
example,
log
files
can
I
access
there,
the
timestamp
value
and
say
okay
group
by
the
timestamp
value,
because
now
the
fluent
bit
was
waiting
20,
real-time
seconds,
but
I
want
to
have
20
seconds
slices,
maybe
off
the
timestamp.
Is
it
possible.
B
A
So
you
want
to
have
Hadoop
in
fluent
it,
and
you
know
what
I'm
going
to
be
honest
and
we
are
taking
the
best
ideas
of
what
is
in
the
market
and
what
are
the
needs,
because
what
you
just
said
is
being
between
a
purchase
part
with
Kafka
right
yeah,
because
that's
because
right
now
everything
is
happening
in
memory,
but
I
think
that
will
not
be
hard
toward
some
kind
of
storage
like
Kafka,
because
it's
like
a
pen
look
wait
time:
son,
where
we
can
do
that.
Actually,.
A
D
A
A
We
render
rejects
the
text.
The
text
yes
happened,
that
extreme
precision
happens
can
be
done
because
we
already
do
the
conversion
from
a
structured
data
to
structure
because
we
parse
the
data,
so
we
have
a
binary
map,
so
we
know
the
fields
that
we
have
on
every
record.
Okay,
so
you
can
do
whatever
you
want
in
terms
of
query.
On
top
of
this
all
is
possible
right
now
we
extend
in
the
language,
not
the
language,
but
the
support
is
SQL,
because
right
now
support
very
basic
stuff
comparison.
A
A
A
E
In
this
new
world
of
micro
services
and
lots
of
containers,
lots
of
pots
and
everything
we're
talking
about
lots
and
lots
of
data,
would
you
recommend
running
fluent
built,
for
example,
as
a
sidecar
container
running
near
the
pod?
That
would
be
one
log
producer,
but
if
I'm
interested,
for
example,
if
I'm
having
stylus
services,
they
do
the
same
job,
they
deliver
the
same
data
I'm
interested
of
summing
the
averages
of
CPU
or
whatever
from
all
these
sources.
E
Would
you
recommend
putting
a
fluent
bit
aggregator
before
of
the
sink,
because
we
are
currently
facing
exactly
this
problem?
We
are
sinking
into
a
lastik
search
and
there's
a
huge
amount
of
data
and
we
are
going
to
move
into
to
put
a
fluent
aggregator
before
of
it.
But
that's
fluent
traditional
flu,
not
one
bit.
Yes,.
A
So
I
would
say
that
just
give
it
a
try,
we
as
a
sidecar
as
a
daemon
set
and
there's
not
like
a
magic
world
for
that,
because,
as
you
know,
data
processors
or
logging
tools,
we
are
really
fast,
but
that
can
mess
up
your
environment
because
database
or
whatever
is
not
so
fast
or
your
network
doesn't
have
enough
bandwidth.
So
I
would
say
that
and
just
give
it
a
try
as
a
sidecar
or
just
give
it
a
try.
Sad
Iman
said,
but
there's
no
like
a
straight
rule
for
that.