►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Fluent Bit: Logging and Data Processing on the Edge - Eduardo Silva, Arm Treasure Data & Masoud Koleini, Arm
Fluent Bit is a Fluentd sub-project that aims to solve hard data challenges in the cloud space. On this deep dive session, we will talk about its architecture, how data workflows operate and the ability to perform advanced data transformation. Also, we will demonstrate the new ability to perform Stream Processing on the Edge.
https://sched.co/Uagy
A
A
So
the
whole
point
of
logging
is
to
deal
with
data,
but
if
you
have
too
much
data,
it's
useless
if
you're
not
able
to
gather
the
value
and
the
problem
is
that
data
is
everywhere.
So
if
you
don't
stop
value
data
as
I
said,
is
useless
and
if
you
have
all
a
bunch
of
data
that
data
is
everywhere,
and
you
want
to
gather
these
insights,
you
need
to
have
the
your
data
centralize
it
in
some
place
like
a
database
or
a
cloud
service.
A
So
then
you
can
extract
the
value
and
I
would
say
that
that
is
like
the
main
reason
behind
login
and
all
this
kind
of
to
Linux
flew
in
DM,
flew
in
baby,
be
able
to
concentrate
your
data,
so
you
can
take
advantage
of
that
and
extract
the
problem
value.
But
there
are
many
kind
of
challenges
here.
Log
in
in
general
is
not
fun,
but
also
it's
not
easy,
because
data
comes
from
different
places:
TCP
UDP
the
file
system
nowaday,
we
have
system
D
with
Journal
D.
A
We
have
data
coming
from
sensors
or
remote
Hardware
like
firewalls.
So
how
do
we
be
able
to
collect
all
this
information
and
centralize
all
this
information?
That
is
not
just
that
comes
from
different
places,
also
different
formats.
In
general
we
say
that
we
take
this
hardware
or
software
data
or
as
events
we
centralize
them
in
a
database,
and
then
we
can
do
analysis,
but
we
have
multiple
of
them
and
saying
you
know
that
every
year
things
are
scaling
up.
A
So,
as
I
said,
we
got
this
kind
of
challenges
and
if
you
are
familiar
with
a
unique
services
like
Apache
my
sequel,
you
will
realize
that
every
data
has
a
different
format:
okay,
I'm
in
for
a
computer.
This
is
just
text
right.
It's
just
an
array
of
light,
there's
no
difference,
so
it
really
when
we
consume
this
data.
In
addition
to
deal
with
different
source
of
data
with
the
format,
we
need
to
have
a
way
to
convert
this
data
from
an
instructor
way
to
a
struktur
way.
A
You
can
see
that
JSON
is
kind
of
a
structure
because
we
have
a
map.
We
have
keys,
values,
arrays
and
different
data
types,
but
that
is
not
it's
not
true
for
all
the
kind
of
locks
that
we
have
available,
and
this
is
where
flu
embed
comes
in.
If
you
embed,
is
a
project
that
borns
as
a
child
break
of
flu
Andy
in
order
to
solve
different
challenges.
Four
years
ago,
forecasting
that
we
are
going
to
need
something
more
lightweight
and
something
that
needs
to
play
well
either
in
containers,
but
also
with
embedded
Linux.
A
It
is
started
in
2015
and
quick,
evolved
it
for
the
cloud
space
and
with
the
cloud
space
I
mean
that
people
who
was
running
containers
or
any
kind
of
cloud
services
say:
hey
we're.
Recent
flu
Indy
is
really
good,
but
also
we
need
something
lightweight
for
some
special
scenarios
like
flu
embed.
So
sorry
like
H
note
or
something
like
that,
so
we
started
this
project
with
that
in
mind,
and
we
said:
okay,
let's
develop
both
projects
and,
of
course,
both
projects
are
open
source
under
the
CN
CF
and
under
the
Apache
License
fluent
bed.
A
Today
talking
to
people,
we
realize
that
there
was
quite
a
fun
that
they
were
surprised
that
is
reading,
see
language
yeah.
It's
not
goal
is
not
rust.
It's
not
Java.
It's
C
language
people
get
that
question,
but
when
they
see
the
final
results
in
performance
in
low
CPU
and
low
memory,
you
say:
okay,
this
project
is
really
good
and
it's
a
good
fit
for
different
markets,
not
just
a
cloud
where
you
have
big
servers,
but
also
for
embedded
Linux
for
any
kind
of
data,
processing
or
H
computing
side.
A
A
When
flu
indeed
was
decided,
it
was
a
sign
in
a
way
that
it
was
considered
like
a
data
pipeline.
So
say
we
have
an
input
from
where
we
collect
data.
We
have
a
parser
section
or
compute
component.
Where
we
take
some
data
and
we
convert
from
one
structure
to
structure.
Then
we
have
a
filter
baby
because
you
learn
to
filter
out
information,
or
maybe
you
want
to
obtain
some
metadata
to
your
records.
A
We
are
going
to
see
that
right
now,
then
you
buffer
your
data,
because
you
don't
want
to
lose
it
and
then
you're
able
to
roll
your
data
to
your
favorite
in
database
and
from
a
fluent
bed
perspective.
We
took
the
same
design,
the
same
principles
if
you
are
familiar
with
kubernetes,
a
or
containers
in
logging,
just
a
simple
message
that
stacks
hey,
Keep
Calm:
it's
not
just
that
message,
because
also
it
has
a
timestamp
and
also
an
origin
from
where
this
message
was
generated.
A
So
a
simple
message
get
more
context
and
in
cover
knit
is
that
this
is
not
enough,
because
if
you
are
familiar
with
kubernetes,
how
do
you
deploy
applications?
You
can
say
I
take
this
pot,
which
it
will
be
my
application
and
make
10
replicas.
Ok,
if
you
do
10
replicas
likely,
the
replicas
will
be
spawned
on
different
nodes.
So
at
some
point,
how
do
you
identify
when
you
get
some
issue
on
which
node
the
problem
is
happening
right
on
which
of
the
replicas
already
getting
problems?
And
this
is
where
we
append
some
context.
A
So
for
us
an
application
generates
something
that
is,
we
call
a
record.
A
record
is
select
technical
term
and
that
record,
as
I
said,
has
a
timestamp
and
a
message.
But
internally
we
don't
handle
that
Asakura,
an
array
of
bytes
at
normal
text.
We
use
like
a
binary
representation
of
that
message
in
a
format
which
is
called
message
back
so
message
back,
considers
lab
binary
JSON.
A
So,
and
this
is
how
flu
embed
and
fluency
in
general
operate.
So
they
take
data
and
compare
any
kind
of
information
in
a
binary
representation
for
internal
purposes
when
the
data
a
comes
in.
This
is
a
quite
of
learn:
global
architecture,
for
example,
in
the
left
we
group
by
all
the
records
by
their
origin.
So
we
can
apply
a
tag
because
we
say
all
this
data
that
is
coming
from
one
place
will
have
a
tag
and
that
tag
is
going
to
be
rolled
back
to
any
kind
of
destinations.
A
So
we
support
like
tagging
with,
will
curve
matching
and
we
can
send
one
record
to
multiple
places,
which
is
pretty
common.
People
said:
hey
I
want
to
have
some
kind
of
Records
on
elasticsearch,
but
also
I
want
to
put
some
of
the
into
Amazon
for
persistence
in
communities.
We
deploy.
How
do
we
solve
this
login?
He
said
we
deploy
flume
D
and
flu
embed.
As
a
daemon
said,
a
demon
said
is
support
that
runs
on
every
node
and
this
was
a
common
question
so
how
it
operates.
A
You
can
see
that
in
the
node
you
just
deploy
1
lakh
processor,
but
this
lot
processor
has
access
to
the
whole
container
locks
and
once
it
get
access
to
the
container
locks
then
is
able
to
talk
to
the
API
server
because,
as
I
said
before,
we
want
to
get
all
the
context
for
the
locks
like
labels
and
notations
and
any
kind
of
information
that
is
not
local
in
the
node,
but
the
API
server
is
a
word.
So
if
you
look
at
this,
you
say:
ok,
you
cannot
handle
this
manually
right.
A
You
need
the
right
trolling
for
this
and
that's
why
I
flew
in
DM
flowing
bed
has
the
kubernetes
filter
that
handle
all
of
these
details
behind
the
scenes.
So
when
trying
to
say
you
just
need
to
deploy
one
bit
I'm
doing
this
a
demon
set,
and
they
will
do
the
work
for
you
and
after
that
they
are
able
to
push
all
the
logs
or
information
to
your
decided
database
or
cloud
service
can
be
Kafka
influx
in
a
place,
data
dog
or
so
on.
A
So
now
a
couple
of
project
updates
and
the
project
has
been
running
for
about
40
years,
and
they
are
really
happy
that
we
have
more
than
100
contributors.
If
you
consider
this
that
we're
a
small
company
start
a
project,
and
then
you
see
many
companies
contributing
back
because
of
his
adoption,
because
this
is
working.
I
think
that
we
feel
really
happy
about
this,
because
also
there's
a
one
moment
that
the
project
is
bigger
than
us
right.
A
It's
a
community
dream
project
where
different
companies
are
also
invest
in
their
own
time,
their
own
money
to
make
the
Priya
better
and
the
player
will
not
be
in
the
status
that
is
today
is
not
where,
for
these
people
and
one
of
the
happy
stats
for
this
year
is
that,
as
of
today,
we
had
more
that
50
million
deployments
just
on
this
day.
Just
today.
So
every
day
we
are
getting
like
300,000
deployments
in
kubernetes.
That
means,
and
some
cluster
is
spinning,
Amazon
beams
or
bare
metal
machine
and
is
deploying
full
embed
on
it.
A
A
In
terms
of
who's,
contributing
back
to
the
project
and
I
think
that
we
got
a
good
investment
from
AWS
AWS
said
we
are
going
to
use
full
and
bit
and
going
to
normalize
a
fluent
bid
for
all
most
of
our
login
solutions,
and
we
are
going
to
start
implemented
their
own
plugins
for
full
embed.
So
AWS
is
contributing
plugins
to
connect
through
embed
with
their
own
services.
File
ends,
Amazon,
s3,
redshift
and
so
on.
Google
cloud
platform
is
using
flue
embed
internally
to
and
data
to
contribute
impacted
on
the
adopt
plugin.
A
One
of
the
ordinance
that
we
have
is
that
a
used
to
be
the
sole
maintainer
of
this
project,
but
this
break
is
growing.
So
I
would
like
to
say
that
we
have
two
new
maintainer.
One
of
them
is
Massoud
who's.
Here,
Moscow
has
contributed
a
lot
on
the
stream
processor
core
part.
Anything
as
has
played
a
key
role
on
division
on
the
how
this
operates
or
to
get
applause
for
Massoud.
A
He
has
to
be
very
patience
with
me
and
we
have
a
new
also
meant
a
contributor
who's,
not
here
something
he
could
not
float
from
Japan,
but
Fujimoto
is
managing
all
the
windows
platform
port
which,
as
you
know,
windows
is
really
painful
environment
for
developers
personally,
I
want
to
stay
away
from
it.
So
hopefully
we
have
somebody
with
the
right
expertise.
We
have
fluent
big
packages
for
windows
and
we
have
many
people
this
week
working
in
to
have
a
fluent
bit
windows
containers
for
kubernetes.
A
It's
a
still
working
process,
but
we
also
we
got
people
a.
This
is
not
a
working
wolf.
We
need
more
time
for
it.
Today
we
just
release
fluent
bait
13.3
and
just
explain
a
little
bit.
We
have
the
major
version,
which
is
one
they
sorry,
the
stable
version,
one
major
version
three
and
the
last
number
becomes
some
inner
fixes.
A
So,
every
time
that
we
release
a
new
version
of
will
embed,
we
are
aware
that
if
we
mess
up
the
things
we
are
going
to
mess
up
thousand
of
nodes,
so
you
can
always
rely
on
the
stable
version
like
this
and
every
time
that
the
last
number
change
you
want.
You
can
make
sure
that
it's
just
a
bug
fix
something
that
is
not
working.
Now
it's
working
and
we
tested
heavily.
A
Now,
a
bit
of
the
road
map,
a
where
we're
going
for
the
next
year
on,
at
the
end
of
January
thing
we
are
planted
in
major
release.
1.4
a
one
of
the
futures
is
that
config
Maps.
This
is
not
about
kubernetes
config
maps,
but
confirm
acts
is
about
a
way
to
avoid
that
you
make
typos
in
your
configuration
or
you
mess
up
things,
and
we
don't
tell
you
why.
So
if
you
have
a
configuration,
property
called
parcel
and
you
type
a
parser
a
it
will
tell
you
a
this
is
wrong.
A
A
They
have
latin
characters
and
all
kind
of
encodings
where
our
old
and
we
are
providing
support
for
them
and
a
we're
continuing,
invest
in
stream
processor,
as
in
Massoud
Co
explained
in
that
part,
birth,
better,
implement
machine
learning,
algorithms,
because
now
fluid
is
not
just
for
the
cloud
it
can
run
on
the
edge
and
the
edge
can
be
anything
an
old,
an
embedded
device
or
an
embedded
gateway
and,
of
course,
continue
improving
on
performance
and
connectors.
A
little
introduction
about
the
stream
processing
before
to
give
the
award
to
Massoud.
A
If
we
concede
that
when
we
do
a
stream
processing,
we
say
that
its
ability
to
work
with
data
modification
or
perform
data
calculation.
We
have
two
scenarios,
we
have
the
edge
and
we
have
the
cloud-
and
this
is
fine
that
is
currently
working.
But
most
of
the
stream
processor
now
needs
to
have
a
JVM
running
a
couple
of
clusters
in
the
cloud
and
when
you
want
to
do
data
analysis
and
get
faster
response
from
your
queries,
it
takes
time
or
minutes.
It
depends
of
the
month
your
data
you
have.
A
So
if
you
send
all
your
data
to
the
cloud,
you
have
to
wait
for
indexing
or
stream
processor.
Just
complete
these
jobs
to
run
your
own
queries.
So
how
can
we
make
it
better?
So
we
said
eh,
let's
build
log
in
on
asteroids
right,
let's
put
a
stream
processor
on
the
log
in
layer
and
why
well
the
wisest
plane,
and
how
is
this
so
instead
the
same
login
layer,
we
are
providing
a
stream
processing
capabilities
so
for
certain
egg
queries
or
analytics
functions
that
you
want
to
run.
You
don't
need
to
go
to
the
database.
A
B
One
thousand
nodes
is
instead
of
waiting
for
the
node,
so
send
the
data
to
the
server,
do
the
computations
and
possibly
get
back
to
the
node.
The
other
thing
is
that
you
only
send
required
data
to
the
cloud,
so
you
don't
need
to
send
everything
so
we'll
get
back
to
the
to
that
later
in
general.
If
you
want
to
write
a
computation
on
fluent
bit
or
and
everything,
so
you
prefer
not
to
write
a
complicated
code
yourself.
B
You
need
some
sort
of
simple
declarative
language
that
you
can
describe
your
computation
in
so
in
fluid
stream
processing.
We
are
providing
a
declarative
SQL
like
language
that
you
can
define
most
of
the
common
computations
in
that
we
have
provided
lots
of
functions
and
we
are
also
adding
more
function
to
that,
and
one
of
the
main
important
thing
is
that
it's
integrated
in
fluid
co-regency,
it's
pretty
fast
and
it's
very
resource
efficient.
B
Ok,
so
many
of
you
may
be
familiar
with
fluid
configuration,
so
we
can
write.
We
can
have
a
syntax
for
the
streams
like
we
can
create
a
stream
using
the
syntax
that
they
will
describe
later
and
I
will
show
that
to
you
later
that
you
can
just
simply
add
your
stream
tasks
to
a
file
and
just
think
sort
of
include
that
file
in
your
fluid
country
file
and
that's
it.
It
is
very
simple.
B
So
this
is
a
kind
of
a
simple
form
of
how
you
define
a
stream
task
in
fluent
bit
and
you
can
go
to
documentation
and
see.
How
exactly
can
you
do
that?
I
will
show
that
in
your
to
you
later
in
the
demo,
but
you
definitely
say
create
a
stream
with
a
specific
stream
name
as
and
like
a
school
language,
you
will
have
a
selector
statement.
You
can
say
select
this
field.
This
world
is
filled
from
this
stream,
which
is
a
stream,
and
the
stream
name
can
be
your
input.
B
Plugins
assume
that
you
are
reading
from
memory
plug-in
and
you
want
to
do
some
sort
of
averaging
of
the
other.
The
memory
usage
you
can
say,
select
mem,
that
used
from
Stream
memory
and
then
in
general,
for
stream
processors.
We
have
to
do
the
computations
over
a
window,
a
window
of
some
specific
size,
so
we're
defining
window
of.
We
have
two
types
of
windows
here
that
we
collect
all
the
data
in
the
window,
and
then
we
do.
The
calculations,
like
average,
mean
mics
time
service
functional
it
is,
and
this
windows
are.
B
We
have
tumbling
window
which,
for
example,
is
a
window
of
size.
Five
seconds
which
collects
on
the
data
insight
use
does
a
computation
sense.
The
computation
throughout
to
the
output
plug
in
throws
away
everything
waits
for
the
window
to
be
filled,
or
we
have
a
hopping
window,
which
many
of
you
may
know
that
as
a
sliding
window,
which
we
can
have
a
window
of
size,
five
seconds,
we
wait
for
the
window
to
be
filled.
B
We
move
the
window
one
second
by
one
second,
and
do
the
calculations
over
that
and
I
can
say
that
it
is
very
popular
more
than
tumbling
window.
Now
we
can
do
the
filtering
over
the
records
that
we
get
by
the
word
condition
like
SQL.
So
we
can
just
do
this
calculations
and
filter
a
filter
already
caused
by
the
word
condition,
and
we
can
also
do
the
group
by
over
the
fields
that
you
want.
B
For
example,
you
can
have
temperature
from
different
temperature
sensors
in
different
rooms,
and
you
want
to
get
the
average
based
on
the
room.
So
we
can
use
the
group
by,
for
example,
you
say
group
by
room
number
then
in
the
output
records,
then
you
will.
You
will
have
one
record
per
room
for
the
average
temperature
okay,
but
where
does
this
stream
processor
resides
in
in
the
work
flow
that
we
had
for
fluent
bit?
So
the
film
bit
has
input
plugins,
parser,
filter
buffer
and
in
general.
B
What
we
do
is
that
so
we
can
we
sort
of
pipe
the
output
of
the
buffer
to
the
stream
processor,
based
on
the
name,
for
example.
If
the
input
is
memory,
and
then
we
define
a
signal
that
says
cell,
like
these
things
from
stream
memory,
then
we
are
sort
of
passing
the
output
of
this
buffer
toward
a
stream
processor,
and
then
it
is
designed
in
a
way
that
it
exactly
looks
like
another
input
plugin.
B
B
This
is
a
subset
of
the
functions
that
we
provides.
So
we
have
more
than
that
and
we
are
going
to
add
more
than
this
as
well,
so
we
can
have
average
over
cheese
count
minimum
maximum
and
some
you
can
have
time
functions
that
you
can
be
added
to
your
output
record
like
and
now
UNIX
times
Sam,
one
of
the
important
things
that
we
are
working
on
and
we
have
added
currently,
is
that
we
are
started
adding
time
series
functions.
B
For
example,
we
have
a
forecast
function
here
you
have
a
stream
of
memory
or
CPU
as
soon
as
you
have
an
a
stream
of
memory
and
you
want
to
forecast
when
so
what
will
it
be?
What
will
be
the
memory
usage
in
the
next
100
seconds
and
then,
based
on
this
forecast,
you
may
some
send
some
alerts
to
some
services
on
the
cloud.
B
This
is
actually
a
simple
linear
regression
here,
but
it's
just
possible
if
someone
is
interested,
can
add
more
time
service
function
to
that
and
we
are
working
on
adding
more
what
about
important
things.
So
I
don't
have
that
in
the
demo,
but
assume
that
you
have
a
nested
Jason.
So
can
we
do
anything
on
the
nested
Jason?
So
yes,
so
we
are
also
providing
sub
keys.
So
you
can
do
all
these
kind
of
operations
on
key
sub
key
one
sub
key
too.
B
Example,
so
I
will
I
will
demo
this
now
so
like
out
of
memory
forecast.
So
as
soon
as
we
have
an
input
memory
in
an
input
plug
in
memory
that
we
are
reading
the
memory
usage
on
a
specific
note,
and
then
we
want
to
forecast
the
memory
usage
in
the
next
100
seconds.
So
this
is.
This
is
a
simple
simple
example
of
that
you
can
say:
select
transfers
forecast.
The
first
parameter
is
your
records
time.
B
B
What
will
be
the
forecast
for
memory
usage
on
this
note
in
the
next
100
seconds,
from
a
stream
memory,
input
memory
plug
in
and
we
have
a
hopping
window
or
sliding
window
of
size
30
seconds
that
does
this
forecasting
over
the
30
seconds
and
in
general
we
read
one
one
sample
per
second,
so
it's
like
over
the
30
samples
and
move
it
one.
Second,
by
one
second
in
general,
so
if
you
have
a
line,
your
windows,
you
have
a
better
forecasting
by
the
way,
so
you
will
have
less
noise
in
your
forecasting.
B
B
So
it
is
a
bit
different
from
what
I
showed
you.
That
was
a
simple
thing.
You
will
define
your
tasks
like
this.
You
will
define
your
stream
tasks
here
and
then
exact
and
define
here
as
Krita
stream.
I
just
go
through
the
end
now
to
say
that
we
do
from
a
stream
memory
and
a
hopping
window
15
seconds
a
small
hopping
window
for
the
sake
of
the
demo,
it's
better
to
have
a
longer
one
to
have
better
and
less
noise
and
better
accuracy
and
I'm,
applying
an
average
effort.
B
The
memory
usage
memory,
I'd
used
and
I'm
trying
to
also
forecast
the
memory
usage
in
the
next
100
seconds
and
send
it
to
the
output
record
with
the
name
of
forecast.
Oh
yeah,
sure
it's
good,
ok,
so
in
the
second
test
offer
for
the
sake
of
having
less
noise,
I,
just
read
from
the
above,
a
stream
forecast
and
apply
an
average
over
the
forecast
have
a
list.
Less
noisy
forecast
for
the
sake
of
them
are
by
the
way
and
now
I
have
another
stream
here
called
alert.
B
So
I
read
from
this
forecast
average
the
average
over
the
forecast,
which
is
smoother
than
the
forecast
by
the
way
and
create
a
stream
called
alert
to
whenever
our
forecasts.
It's
over
1.5
gigahertz
of
memory,
that's
it
now!
We
try
to
run
it.
Ok,
so
build
from
be
it,
and
the
configuration
file
is
demo
one
bit
for
chess
okay.
So
if
user,
it's
a
CD
out,
so
this
shows
the
output
of
the
memory
plug
memory
plug
in.
B
So
this
is
what
memory
plug
in
sends
to
our
forecast
stream
after
30
seconds
here
we
are
starting
to
get
forecasts
as
well.
So
this
is
the
output
of
the
forecast,
so
I
stop
here
sort
of
so
this
is
a
forecast
that
has
average
of
memory
used
and
the
forecast
in
the
next
100
seconds,
and
this
is
an
average
over
the
forecast,
and
this
is
again
under
you
know
earlier
output
from
the
memory
field.
So
if
I
go
here,
I
can
see
yes,
this.
These
are
the
green
line.
B
Is
the
forecast
and
the
the
green
line
is
the
average
of
memory
usage
and
one
is
the
forecast
which
is
almost
the
same,
because
we
are
not
changing
the
memory
usage
on
there
on
the
machine.
But
now,
if
I
start
running
some
service
that
starts
consuming
memory,
then
you
can
see
that
the
forecasts
start
to
increase.
That
means
that
the
yellow
line
says
shows
the
forecast
in
the
next
100
seconds.
That
says
all
in
the
National
100
seconds.
We
have
this
amount
of
memory
used
again,
because
the
window
is
not
that
long.
B
B
Yes,
possible
is
coming,
oh
here,
I
think
it
notification,
but
we
are
I
was
waiting
for
the
notification,
but
we
are
getting
alerts
on
the
slack
channel
here.
So
that
means
that
so
it
takes.
You
know
it
takes
a
memory
usage
and
sending
others
for
us
on
this
next
channel.
So
I
was
hoping
that
I
get
notification
on
here
under
notification
bar,
but
I
didn't
so.
B
So
this
a
snapshot
is
a
new
feature
that
is
pretty
well
working
with
that,
like
we
create
a
snapshot
over
as
well
our
record
a
snapshot
over
on
a
stream
and
we
just
flush
flush,
the
snapshot
when
something
happens,
for
example
an
Apache
server.
So
they
don't
want
to
send
all
the
info
info
likes
to
the
server.
But
whenever
in
the
500-hour
happens
they
want
to
send
love
to
the
server.
But
the
prop.
B
So
we
can
have
the
snapshot
of
size
or
the
number
of
records,
or
both
of
them
or
one
of
them
here,
I
added,
both
of
them
and
here
in
the
seconds
now
in
the
second
rule,
we
can
say
flush
a
snapshot
from
this
stream
Apache.
When
code
is
500,
then
whenever
a
500
error
happens,
so
it
flushes
the
snapshot
to
the
output
plug
in
this.
B
B
The
live
snapshot.
We
exactly
have
these
two
things
that
you
saw
on
the
previous
slide,
so
we
are
reading
a
from
an
Apache,
a
lot
from
up
from
without
reading
opportu
likes
and
whenever
an
error
500
happens,
be
flushed
and
and
I've
used
some
sort
of
lagging
like
these
artificial
eyes
that
I'm,
using
as
a
kind
of
parse
as
Jason
in
a
file
that
I'm
just
passing
those
likes
to
fluent
bit
to
see
how
it
how
it
works.
B
B
B
B
We
are
just
sending
a
snapshot
of
the
lock
to
the
server
whenever
an
event
happens
or
air
or
happens,
which
is
very
important
for
edge
nodes,
especially
because
they
have
problems
with
the
resource
limits,
their
problem
with
bandwidth
and
everything.
So
we
don't
want
to
send
everything
we
don't
want
to
consume
bandwidth.
So
that
is
also
a
feature
that
we
are
going
to
release
in
in
the
next
in
1.4.
I.
Think.
B
So
we
define
the
size
of
the
snapshots
in
two
ways,
so
we
can
define
it
in
seconds
like
the
size
of
the
window
of
the
snapshot
or
we
can
have
another
another
measurement
for
that
which
is
limit.
So
we
can
have
the
snapshots
that
is
of
size,
10
records
or
5
seconds,
which
were
comes
first
or
it's
not
mandatory
to
use
both
of
them.
So
you
can
use
just
limiting.
So
we
can.
We
keep
10
records
or
just
size.
We
give
a
snapshot
of
size,
5
seconds.
A
A
Well,
in
that
question
is
forcing
us
to
release
our
secret
to
take
over
the
world,
but
a
yes
dynamic
configuration
is
something
that
we
have
been
asking
many
times,
but
of
course
we
got
other
priorities
and
I
think
that
what
you
see
with
conflict
maps,
which
is
a
way
to
help
the
users
with
configuration,
it's
also
a
way
for
plugins
to
say
eh.
This
is
a
Manman
configuration,
and
this
is
that
data
types
with
that
in
place.
A
We
can
implement
dynamic
configuration
because
I
don't
want
to
restart
a
plug-in
if
they
configure
a
new
configuration
is
wrong
and
I
need
a
way
to
validate
that
before
hint,
so
config
Maps
is
for
that
also.
Second,
question:
hey.
This
is
really
cool,
really
and
I'm
very
excited
via
NASA,
too,
but
also.
What
we're
most
excited
is
that
UB
will
be
able
to
run
sequel
queries
remotely
so
from
your
own
application.
A
Okay,
that's
the
push
backs
off
today.
Multi
line
looks
when
a
java
application
or
you
get
a
stack
trace
and
your
application
generate
like
twenty
lines,
but
the
containers
handled
that,
like
twenty
different
records.
Okay,
in
fluent
bit,
we
support
multi
line
out
of
a
Dockers.
We,
you
can
configure
a
multi-line
mode,
and
you
said
the
first
line
of
a
multi-line
stack
trace,
looks
like
this,
so
everything
after
that
is
a
continuation.
A
C
B
B
B
So
if
you
sit
under
discussion
by
the
way,
so
we
are
trying
to
see
if
we
can
sums,
we
can
add
some
sort
of
tensorflow
capabilities
to
that,
but
it
it
is
not
an
easy
thing
to
to
work
on
because
it
might
be
heavy
for
edge
devices.
So
that
is
under
discussion.
So
so
at
the
moment,
I
am
Not
sure
that
if
we
can
release
any
kind
of
tensor
tensor
flow
based
inference
or
something
for
version
1.4
for
at
least
so
we
can
add
more
time.
Series
functionalities.