►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Scaling Log Collection with Fluent Bit - Wesley Pettit & Ugur Kira, Amazon
Ugur and Wesley will present their experience migrating customers first from Fluentd to Fluent Bit, and then to Fluent Bit with multi-workers. Benchmarking data will be shared along with recommendations on ideal settings for different use cases. Finally, they will end with a discussion of remaining scale limitations and possible solutions to solve them.
A
Hello,
everyone,
my
name
is
urkira.
I
am
senior
specialist
stamp
for
containers
in
here
in
aws
today,
with
my
colleague
wesley,
we
will
talk
about
scaling
aspect
of
land
bit
and
provide
you
more
insight
around
it.
Based
on
our
experience
here
is
our
agenda.
We'll
consider
scaling
consideration,
scaling
best
practices
and
we
will
mention.
We
will
discuss
the
worker
concept
with
a
bit
and
we
will
present
some
benchmarking
statistics.
A
This
destination
can
be
guinnesses,
they
just
stream
club
watch
logs
or
any
other
destination,
but
you
need
to
understand
the
throttling
and
registration
rate
of
your
destination.
This
has
the
direct
impact
on
you,
throughput
and
performance
of
your
log
delivery
pipeline,
and
then
you
need
to
consider
the
performance
and
the
performance
and
actual
throughput
of
your
pipe,
which
is
different
demon
in
this
case.
So
you
need
to
understand
flan
bit
fully.
You
need
to
optimize
the
this
configuration
and
you
need
to
actively
monitor
your
pipeline
and
another
aspect.
A
However,
this
the
this
stuff
to
enrich
your
logs
and
creating
api
endpoint
to
get
this
metadata
about
your
logs
and
where
it
attempt
to
provide
more
put
more
context
to
your
logs,
it
just
it
add
additional
work
on
your
api
endpoint,
which
may
impact
your
performance
and
scale
of
your
log.
Dev
log
delivery
pipeline,
and
here
this
is
the
default
behavior
and
one
thing
I
want
to
call
out.
There
is
a
me
proposal.
A
They
tend
to
rely
on
the
cubelet
to
obtain
those
information
which
reduced
to
load
on
the
api
service
and
is
really
interesting
to
watch.
If
you
didn't
watch
it
yet,
and
another
thing
you
need
to
consider
is
you:
whenever
possible,
you
need
to
offload
your
workload
to
the
source
and
you
need
to
apply
filtering
how
you
can
do
the
filtering.
A
You
can
use
friendly
filters,
so
in
a
successful
log
delivery
pipeline
you
need
to
only
ship
the
logs
or
information
data.
That
really
adds
value
for
your
business
and
there
is
no
point
to
stream
a
log
that
doesn't
have
any
business
value,
so
you
can
use
finance
filters
to
filter
the
your
locks
at
source
level
flat
bit
has
a
data
stream
feature
which
can
be
also
used
to
filter
and
process
your
data
and
log
as
source
and
when
we
come
to
the
scaling
best
practices.
A
First
thing,
as
mentioned,
you
need
to
understand
your
the
the
behavior
of
your
the
behavior
of
your
filter
and
plugins
that
you
are
using,
and
here
we
mentioned,
the
kubernetes
filter,
generate
an
extra
a
lot
on
your
api
service.
You
need
to
know
this
and,
moreover,
by
default
it
happened,
put
labels
and
annotation
to
do
your
log
recourse
to
add
as
a
context-
and
here
in
case
the
most
of
time,
this
labels
and
annotation
doesn't
have
any
business
value.
A
A
Along
with
memory
buffer
limit,
you
can
also
enable
file
system
buffering,
and
this
will
also
have
little
level
or
large
scale
and
will
ensure
the
data
and
log
integrity
in
case
your
log
destination
is
not
able
to
cope
with
the
load
or
there's
any
technical
problem
there
on
destination
site
and
for
cloud
watch
locks.
We
recommend
to
use
the
hyper
one
cloud
watch,
log
plugin
called
cloudwatch
underscore
logs
and
based
on
this
information
we
mentioned,
we
have
a
optimized
full
and
configuration
in
our
container
inside
page.
A
So
I
recommend
you
to
review
this
configuration
and
try
to
make
your
friend
with
configuration
as
close
as
possible
to
the
configuration
we
share
under
the
containery
site,
documentation
and
last
best.
Practices
that
I
want
to
recommend
is
a
monitoring
hell
of
your
log
delivery
mechanism.
This
is
really
important
about
an
ignored
aspect.
A
Front
bit
has
a
built-in
http
server
com
comes
with
it,
and
you
can
enable
this
in
your
for
the
end.
Co
config
film
bit
configuration
as
you
see
here
and
then
you
can
use
a
really
nice
graph
on
a
dashboard
where
it
will
tell
you
the
how
to
provide
you
more
now.
I
am
handing
over
to
my
colleague
wesley
who
will
talk
about
worker
concept
and
how
worker
concept
helps
with
scaling
aspect
of
food.
B
Okay,
hello,
everyone,
I'm
wesley,
I
am
one
of
the
co-maintainers
of
fluid
bit.
I've
contributed
a
lot
of
code
to
it
for
aws
plugins,
so
I'm
gonna
explain
how
concurrency
works
in
fluent
bit,
which
is
important
for
understanding
scaling
with
fluent
bit
and
the
new
worker
feature
that
was
introduced
so
prior
to
workers.
This
is
how
concurrency
worked
in
flint
bit.
Flintbit
uses
a
model
called
co-routines
co-routine,
I
believe,
is
short
for
cooperative
routine.
B
B
So
then
what
it
will
do
is
it
will
invoke
a
co-routine
for
the
output
that
runs
the
output
code
pass
execution
to
it,
which
means
the
engine
thread.
Basically
moves
execution
to
the
co
routine.
So
now
only
the
co
routine
is
is
active.
Co
routine,
performs
some
work
when
it
hits
a
network
I
o
call,
rather
than
blocking
on
that
call
and-
and
you
know
blocking
into
all
execution.
B
Instead,
it
will
make
the
network
call,
but
then
it
will
pass
execution
and
yield
back
to
the
engine
which
will
then
take
control,
and
it
can
then
run
other
co-routines
to
complete
other
tasks
and
when
the
network,
when
the
kernel
notifies
the
engine
that
the
network
call
has
completed,
then
the
co-routine
can
be
unpaused
it
can
it.
Basically
it
will
be
sleeping
and
it
can
be
activated
again.
B
So
the
way
it
essentially
worked
is
is
works
is
that
there
is
concurrency
and
in
some
sense,
there's
multiple
threads,
but
only
one
of
them
is
active
at
a
time
and
they're,
cooperatively
and
intelligently,
basically
turning
themselves
on
or
off
so
that
network.
I
o
calls
basically
don't
ever
slow
down
the
execution
of
the
problem
of
the
program.
Essentially
it's
just
to
do
asynchronous
network
I
o,
but
then
now
in
fluent
bit
as
of
1.7,
we
have
the
1.7
series.
B
We
have
multi-workers
so
with
workers,
you
can
increase
you,
you
can
have
multiple
threads
active
at
a
time
and
basically
the
simple
story
is
that
workers
are
just
dedicated
threads
for
outputs.
So
here
we
have
an
example
here,
with
an
output,
you
put
workers,
one
that
enables
a
single
dedicated
thread
just
for
that
output
workers
still
actually
have
can
have
co
routines
inside
of
them,
which
means
you're
still
doing
a
like
non-blocking
network.
I
o
to
make
things
efficient
and
fast.
It's
just
that
now.
B
The
co
routines
for
the
output
are
all
happening
in
the
context
of
a
a
single
or
an
extra
dedicated
thread
just
for
that
output
and,
of
course,
you
can
enable
more
than
one
worker
if
you
want,
but
I
think
in
most
situations,
probably
folks
throughput
needs
can
be
satisfied
with
a
single
dedicated
thread
now,
so
how
how?
How
is
how
are
concurrency
and
workers
implemented
in
the
actual
code
and
the
plugins?
B
So,
basically,
all
plugins
have
a
flush
callback
function
that
they
must
implement,
which
is
called
when
they
are
need
to
be
given
logs
that
then
they
can
send
and
basically,
in
order
to
have
currency,
the
calls
to
this
function
must
be
fully
independent.
Basically,
it
can't
be
storing
state.
There
is
a
context,
object
that
basically
this
function
can
have
that
allows
it
to
store
state
in
between
between
different
calls.
B
But
if
you
are
storing
state
in
your
output
plug-in,
then
that
prevents
you
from
being
able
to
have
concurrency
because
it
you
know
it
means
that
basically,
the
engine
can't
independently
have
you
send
multiple
chunks.
At
the
same
time,
also
another
key
thing
in
this
kind
of
follows
from
from
that.
One
is
that
you
can't
have
your
api
can't
have
any
sort
of
guaran
require
ordering?
Where
calls
must
be
made.
B
You
know
in
some
sort
of
like
order
in
a
series,
because
there's
no
guarantee
that
that
will
happen
when
you're
when
you're
running
currently,
because
one
co
routine
might
be
you
know,
performing
a
network
call
and
we'll
be
sleeping
and
then
another
another
co
routine
might
start
sending
before
the
network
call
for
the
first
sleeping
co.
Routine
might
finish.
B
B
Concurrency
support,
interestingly,
is
actually
not
needed
in
order
to
support
a
single
worker.
The
ability
to
have
users
configure
a
single
worker.
So
the
reason
for
that
is
that
basically,
workers
are
a
dedicated
thread.
Even
if
your
output
is
like
synchronous
and
it's
not
doing,
and
it's
basically
doing
blocking
network
I
o
instead
of
non-blocking
co-routine
network,
I
o
you
could
still
have
a
dedicated
thread
and
then
just
do
all
of
those
synchronous
calls
in
the
dedicated
thread.
So
basically
workers
can
still
help.
B
Now,
if
you
want
to
let
your
users
enable
multiple
workers
in
the
output,
then
you
also
need
to
support
concurrency,
because
if
you
have
multiple
workers,
then
that's
you
can
think
of
that,
as
kind
of
equivalent
to
at
least
having
co
routines,
except
for
now
it's
processed
on
multiple
threads.
Interestingly,
as
a
side
note,
if
folks
are
familiar
with
goaling
and
like
go
routines,
multiple
workers,
which
means
multiple
threads
and
then
the
co-routines
operating
inside
them,
I
I'm
pretty
sure,
like
fundamentally
in
terms
of
the
programming
model.
B
It's
a
lot
like
it's
kind
of
a
lot
like
the
go
scheduler
running
go
routines
onto
actual
real
threads
anyway,
so
this
is
just
looking
some
more
at
the
code.
How
do
you
enable
a
concurrency
in
workers?
So
if,
if
you
had
an
output
that
you
maintained
for
flint
bit,
so
concurrency
is
enabled
by
default
for
all
outputs,
you
can
turn
it
off.
With
this
a
little
line
of
code
there
I
wrote
a
whole
developer
guide.
That
explains
this
actually
a
little
bit
more.
B
B
B
So
here's
just
some
example-
output
definitions
to
show
what
that
what
what
this
all
looks
like,
I'm
emitting
some
of
the
required
fields
that
you
have
to
configure
for
these
outputs.
You
can
see
in
kinesis
streams
or
kinesis
firehose.
You
can
enable
any
number
of
workers
that
you
want
in
s3
with
the
default
multi-part
upload
mode,
which
can
only
be
enabled,
if
you,
which
only
happens
if
you,
if
you
have
a
file
size,
that's
above,
I
think
12
megabytes,
but
anyway
you
can
enable
a
single
worker.
B
This
will
be
synchronous,
calls
but
happening
in
a
dedicated
thread.
The
reason
why
s3
multi-part
is
non-concurrent
is
because
the
multi-part
uploads
are
a
series
of
calls
and
they
require
storing
state.
Basically
so
they're
the
plug-in
is
not
stateless.
This
is
something
that
I've
thought
about.
Probably
there
there
probably
is
a
way
to
allow
it
to
to
work
with
multiple
threads,
but
I
need
to
rewrite
the
code
and
think
a
little
bit
more
about
that
anyway.
So
then
s3
was
put
object
mode,
so
put
object.
Mode
is
an
api.
B
Unfortunately
it
doesn't
support
concurrency.
The
reason
is
because
there's
this
thing
concept
of
a
sequence
token
in
the
put
log
events
api,
which
basically
means
it
kind
of
basically
means
that
calls
to
the
api
have
to
be
ordered
where
something
obtained
from
the
output
of
a
previous
call
must
be
sent
in
your
next
call.
So
this
means
you
have
no
concurrency.
You
can
only
enable
one
worker
as
a
dedicated
thread
for
the
output
and
which
it
will
run
synchronously
in
in
a
moment,
uger's
going
to
show
some
benchmarking
results.
B
Surprisingly,
despite
the
fact
that
there's
no
concurrency,
it
actually
still
performs
pretty
well
anyway,
so
wrapping
up
concurrency
and
worker
support.
So
these
are
like
the
unsolved
remaining
problems
for
the
fluent
bit
community
and
fluid
core
to
anyway.
Now
I
will
turn
it
back
over
to
uger
who
will
go
over
some
benchmarking
results
for
fluid
bit
with
workers.
A
Now
I
will
provide
some
more
insight
how
different
destination
and
log
delivery
pipelines
performs
with
and
without
workers
enabled,
and
particularly
in
this
session,
we
will
consider
amazon
cloud
watch
logs
and
I
was
like
this
data
stream
as
a
destination,
and
here
you
see
the
details
of
our
benchmarking
setup
and
we
will
start
with
a
workload
on
cloudwatch
logs.
So
imagine
you
have
a
production
environment.
A
This
was
the
capacitor
pipe
and
parallel
to
increase
in
the
number
of
locks.
There's
pro
number
of
locks
that
have
processed
in
previous
slide
and
after
enabling
the
workload
the
delivery.
Throughput
internal
networking
also
increased,
as
we
see
here
so
one
thing
additional
video
is
like
we
wanted
to
see
the
impact
of
the
work
enabling
vulcan
or
our
resource
consumption
and
and
how
the
how
it?
What
are
the
extra
load
it
brings
internal
cpu
and
memory
to
do
our
cluster.
A
A
We
see
that
our
process
incoming
process
rate
increased
dramatically
and
when
we
look
to
increase
in
terms
of
networking
throughput-
and
we
will
experience
the
same
where
parallel
to
processing
increase
in
processing
rate
of
the
locks,
our
throughput
also
increased,
and
one
thing
here.
These
numbers
are
not
the
mag
max
throughput
limits
of
the
setup
is
just
how
a
worker
can
impact
your
block
delivery
performance
by
just
enabling
them,
rather
than
the
maximum
capacity
of
the
limits
and
with
kinesis
data
stream.