►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
High Throughput with Low Resource Usage: A Logging Journey - Eduardo Silva, Calyptia
In Logging, there is a common fact: more applications means more data to handle. Running services at scale in a distributed environment brings exciting challenges for data management, but with the volume of data increasing there is a necessity to ship this data faster, but a few ones realize the side effect: high resource consumption. On implementing a logging pipeline, pre-processing of the data is mandatory, a simple example of this is Kubernetes metadata enrichment for every log record, but more data means more computing time, the same cost applies when delivering to the final storage or cloud service. In this session, we will do a deep dive into our journey of performance challenges that we faced in the Fluent Bit project around Network I/O + TLS, filesystem buffers, routing, and multiplexing for high throughput. We will share how did we go from 5k/sec to more than 30k/sec using a single-core CPU using purely design improvements and taking the most of Linux OS interfaces.
A
A
So
in
general,
everybody
wants
performance
right.
If
you
want
to
see
high
throughput,
then
you
think
about
performance,
but
everything
has
a
little
cost
right
so
and
when
people
say
performance
is
sometimes
think
about.
What's
my
the
number
of
records
or
events
that
I
can
process
by
a
unit
of
time
right
and
after
that,
you
start
realizing
that
maybe
the
setup
that
you
have
or
the
strategy
that
your
tool
is
using
might
not
be
conformant,
or
maybe
it's
not
following
the
best
practices
from
a
configuration
perspective.
A
A
A
So
I'm
saying
this
because
I'm
maybe
I
would
say
that
eighty
percent
of
you
may
be
familiar
with
fluently,
which
is
a
graduated
cncf
project,
but
also
here
we're
going
to
explain
and
talk
about
the
journey
of
the
sub
project
of
it,
which
is
part
of
the
ecosystem,
and
when
talking
about
login
there's
a
couple
of
things
that
we
need
to
clarify
right.
Logging
by
itself
is
not
cheap
and
we
need
to
understand
that
logging
in
the
past
used
to
be
pretty
simple
right,
an
application
just
ship,
a
message.
A
A
But
if
you
think
about
what
is
the
end
of
this
message
is
that
it
allows
you
to
perform
some
data
analysis
and
to
do
that,
you
need
to
take
all
these
messages
and
centralize
them
in
a
database.
So
then
you
can
create
some
kind
of
schema
or
run
some
queries
on
top
of
it.
It
doesn't
matter
if
you
store
in
a
sql
or
non-sql
database
or
document
oriented.
A
Actually
the
pattern
is
pretty
much
the
same.
You
want
to
have
some
structure,
which
is
on
top
of
the
text
message
that
was
generated
at
the
beginning,
and
one
of
the
challenges,
if
you
think
about
throw
booth
performance,
is
that
every
time
we
have
more
data
because
we
have
more
applications
right,
everybody
is
now
deploying
microservices,
decoupling
everything
and
it's
really
hard
to
keep
control
of.
What's
the
logging
rate
of
if
application
or
what
kind
of
messages
this
is
spending
right.
A
As
an
example,
let's
take
this
simple
message
that
it
comes
from
a
nginx,
nginx
log
access
law.
Actually,
what
it's
doing
is
just
writing
a
message
with
specified
the
id
address,
a
timestamp
on
when
this
a
request
was
generated,
plus
the
information
from
the
protocol,
the
method,
the
uri,
the
protocol
version,
the
response
site
from
the
server
blah
blah
blah.
But
this
log
message
also
is
not
unique
right.
It
could
be
different.
Sometimes
applications
generate
messages
in
different
formats,
even
coming
from
the
same
application.
Sometimes
you
have
multi-line
application
or
think
about
stack
traces
right.
A
So
you
can
think
that
this
thing
is
a
bit
complex
and
when
you
have
this
application
generating
this
message,
I
would
say
that
eighty
percent
of
the
time
is
just
a
rough
text
message.
It
doesn't
have
any
structure,
it
doesn't
have
any
schema.
Nowadays
we
are
seeing
people
and
companies
trying
to
accomplish
let's
try
to
log
with
json
or
trying
to
unify
a
specific
format,
but
in
reality
that
that
effort
has
been
around
for
years
and
it
is
really
hard
to
say
that
the
industry
is
going
to
align
to
one
specific
format.
It's
really
hard.
A
So
we
need
to
have
this
kind
of
engine
that
is
able
to
listen
for
these
messages
or
realize
that
they
are
rare
and
then
be
able
to
take
them
and
send
it
out
to
any
destination
like
amazon,
3,
elastic,
stackdriver,
splunk
or
any
kind
of
service
right.
Nowadays
we
have
multiple
options
in
the
market,
and
this
is
just
an
example.
One
of
them
there's
no
preference
from
from
my
perspective.
A
A
If
you
are
receiving
some
messages
over
tcp
likely,
you
would
like
to
write
the
ip
address
from
the
where
this
message
is
coming
from
now,
if
you
are
wearing
kubecon
right.
So
if
you
think
about
messages
coming
from
your
port,
you
would
like
to
have
your
labels
inside
the
same
record.
So
then
you
can
group
them
and
perform
your
query
without
any
problems
without
losing
context,
but
also
there
are
other
cases
where
you
want
to
perform.
A
Some
data
reduction
right
if
you're
trapping
a
lot
of
messages,
and
these
messages
have
like
a
as
I
said,
the
early
information
developed
messages.
They
maybe
don't
want
them
right.
So
you
want
to
make
sure
that
every
everything
that
comes
as
a
debug
messages
in
your
pipeline
just
drop
it
right.
If
you
think
about
a
storage
that
charge
you
the
build
right
because
of
the
amount
of
data
that
you
ingest
right,
you're,
maybe
interested
in
to
perform
data
reduction
and
not
send
the
whole
information.
A
Also,
as
part
of
the
role
of
the
log
processor
is
to
perform
buffering
and
buffering.
Is
the
capability
to
take
this
data
and
optionally
maybe
store
it
perceived
in
a
persistent
way
on
this?
Because
you
need
to
restart
the
service
or
maybe
the
subject
just
crashed,
or
you
have
to
get
a
power
failure
or
nettle
audit.
A
So
you
want
to
make
sure
that
you
don't
lose
any
data
and
finally
be
able
to
send
this
information
to
a
different
destination,
and
these
destinations
are
important
and
some
of
them
are
just
to
handle
the
information
as
a
binary
globs
right,
because
I
just
want
to
store
the
information
that
I'm
getting
but,
for
example,
in
the
case
of
a
slash
you
just
care
about.
How
can
I
query
my
documents?
A
Ideally,
I
want
to
have
all
my
documents
in
elastic
with
a
good
structure
in
the
json
map,
so
I
can
query
or
have
kind
of
index
through
them
right
same
thing
happen
with
stackdriver
and
same
thing
happens
with
a
spline
class,
others
in
general.
You
can
see
that
we
have
an
input
with
different
sources.
We
have
this
engine
in
the
middle
and
we
have
the
output.
So
explaining
these
concepts
are
is
really
important
to
understand
performance
right
because
we
need
to
realize
what
how
this
works
behind
the
scenes
right
so
in
general.
A
Simplifying
everything
is
an
input
and
an
output,
but
we
need
to
explain
what
is
happening
inside
the
input.
For
example,
inside
the
input
you
have
a
lot
of
io
io
means.
Maybe
you
are
opening
a
file
from
from
the
disk
right
or
maybe
you
are
listening
from
a
new
network
connection,
or
maybe
you
are
collecting
metrics
locally
from
the
prog
file
system
or
just
receiving
them
from
a
third-party
service
in
the
engine
has
a
lot
of
work
as
I've
explained
it
earlier.
It's
like
you
have
to
parse
information.
A
It's
like
I
need
to
deal
with
network
setup
with
payload
formatting,
because
every
output
destination
expects
a
different
payload
format.
Most
of
them
are
working
with
json
nowadays,
but,
for
example,
difference
is
kaka
right.
Kite
has
in
its
own
format
for
the
network,
but
all
of
the
others,
elastic
and
azure
log
analytics
and
even
splunk
use,
json
right
and
the
output
site
needs
to
deal
with
delivering
content
in
different
formats.
I'll
also
be
able
to
understand.
A
If
I'm
sending
this
information,
I
need
to
expect
for
a
specific
return
code,
because
if
something
failure
and
the
server
maybe
says
you
know
what
I
could
not
use
process
with
information,
we
need
to
reply.
The
engine
needs
to
be
able
to
retry
through
a
scheduler,
well,
something
that
started
as
an
input.
Output
has
a
lot
of
tasks
that
works
internally
and
now
I'm
going
to
introduce
fluid.
A
This,
but
also
included
in
ruby
and
being
written
in
ruby,
has
some
downsides
from
a
scripting
language
right.
It's
really
easy
to
extend.
It's
really
easy
to
skill,
but
if
you
want
to
optimize
some
resources,
it's
really
hard
and
I
would
say
that
it's
not
just
ruby
but
any
kind
of
language
that
start
implementing
a
system
level
application
that
start
coupling
a
bunch
of
dependencies
right.
Actually,
you
start
focusing
mostly
on
usability
or
how
to
extend
it
that
optimizing.
A
Every
single
component
for
performance,
a
fluid,
is
written
in
c
language
and
we
try
to
reduce
as
much
as
possible
the
dependencies
and
try
to
build
our
own
architecture
and
deal
with
most
of
the
staff
on
our
own,
and
I
think
that
from
a
community
perspective,
we
have
get
a
really
good
feedback
and
really
good
traction
from
this.
Nowadays,
the
users
use
fluency
majority
of
them,
while
others
are
using
fluent
bed
and
we
have
a
third
option
which
both
are
using,
but
they
are
using.
A
This
is
pretty
fine.
This
is
not
a
big
problem.
I
would
say
that
the
most
critical
part
is:
how
do
you
manage
the
memory
while
consuming
data
while
processing
the
data
and
while
sending
this
information
out
a
part
of
the
engine,
we
can
call
it
like
a
processor
here,
we're
split
splitting
the
exclamation
in
two
parts.
A
Actually,
the
engine
needs
to
be
able
to
do
parsing
right,
because
we
mentioned
that
we
want
to
have
this
constructor
information
to
a
structure
format
right,
so
we
take
it
and
could
take
a
json,
parser
json
or
apply
a
regular
expression
on
top
of
this
side
data,
because
my
data
is
quite
custom,
but
I
know
how
to
group
them
right
as
the
nginx
example.
We
know
that
the
first
field
is
an
ip
address.
The
second
one
is
the
timestamp
and
so
on
all
lock
formats
for
kind
of
goal
and
messages,
csv,
etc.
A
Also,
as
part
of
filtering,
you
might
want
to
do
some
kind
of
data
enrichment
in
kubernetes.
Of
course,
you
want
to
obtain
all
your
labels,
or
sometimes
your
annotations,
through
every
single
log
record,
but
also
you
want
to
do
some
kind
of
the
exclusion
based
on
some
patterns
and
internally,
I
would
say
in
our
case
we
serialize
all
our
information,
every
single
event
doesn't
matter
if
it's
a
metric
or
a
log
record,
we
use
message
pack,
which
is
a
kind
of
a
similar
to
a
json
binary,
but
quite
more
performant
and
for
baffling
mechanism.
A
We
offer
memory
and
this
which
we
are
going
to
explain
in
a
few
minutes
also
as
part
of
this
processing
and
work.
We
need
to
be
able
to
route
this
data
right.
We
have
a
logic
to
roll
this
data,
meaning
like
for
every
data
that
comes
from
a
different
from
a
specific
source.
You
want
to
send
it
one
example
elastic
search,
but
also
you
want
to
archive
all
the
information
to
amazon
f3
yeah.
A
As
part
of
the
output
side
of
this
pipeline,
we
mentioned
that
everything
is
about
also
network
setup,
so
we
need
to
perform
dns
lookup
connect
to
an
endpoint,
and
you
want
to
implement,
of
course,
nowadays
tls.
So
actually,
as
part
of
that
process,
you
want
to
handle
the
tls
hunter.
And
ideally
you
want
to
maintain
this.
A
connection
like
with
keep
alive
right,
if
you
are
closing
a
connection
and
open
a
new
connection,
every
single
time
that
will
be
really
expensive,
because
the
tls
handshake
process
is
really
expensive
because
of
the
round
trips.
A
We
mentioned
that
also
internally.
We
have
all
this
data
in
binary
format
right
and
this
binary
format
for
us
is
message
back,
but
when
you
are
talking
to
elasticsearch,
elasticsearch
does
not
understand
message
back.
It
understands
json
maps
or
well
any
kind
of
json
object,
so
what
you
have
to
do,
or
what
we
do
in
our
elastic
search.
Connector
is
take
this
message
back
binary
message
and
then
send
convert
it
back
to
json
and
then
send
it
over
the
network
right.
So
we
need
to
do
the
payload
delivery.
A
A
So
now
about
optimizing
all
this
data
process
and
io
in
general,
it's
really
interesting
to
understand
how
things
works.
You
know
at
a
deep
level
so,
as
I
said
for
to
manage
messages
internally,
we
use
a
binary
representation
of
the
data
using
messenger
pack
and
also
when
dealing
with
how
to
handle
this
information.
One
is
sterilized
internally
at
runtime.
A
We
group
all
the
records
in
chance,
and
this
is
something
I'm
going
to
explain
in
the
next
slide
about
data
serialization
and
message
back
here
in
the
screen.
We
have
a
table
which
compares
what
how
many
bytes
compares
using
json
versus
message
back
right,
for
example,
for
a
new
message
to
know
in
json
is
means
four
bytes
in
message.
A
This
is
just
a
comparison
of
how
one
format
works
against
the
other
right.
Of
course,
message
pack
is
quite
small
than
a
json
payload
right,
but
I
would
say
that
this
is
one
of
the
advantages,
but
the
other
advantage
is
that
in
message
pack,
you
don't
need
to
parse
every
single
byte
per
byte
right
in
json.
You
have
to
do
it.
You
have
to
create
some
kind
of
index
across
all
your
map,
and
then
you
know
you
can
know
where
the
data
start
and
where
the
data
begins
and
ends
in
message
pack.
A
You
can
know
that
you
have
an
array
beforehand
with
x
number
of
items,
so
you
can
jump
between
them.
So
when
you
are
dealing
with
data
processing,
where
you
are
removing
some
message,
sorry
so
key
or
adding
more
key
or
doing
some
kind
of
any
kind
of
modification
is
quite
more
performant
and
easy
to
do
it
internally.
When
we
get
this
data,
we
talk
about
data
serialization.
Now.
Imagine
that
we
got
a
text
message
as
a
json
map.
What
we
do
is
convert
this
json
to
message
back
and
then
we
have.
A
This
notion
of
stack
attack
is
just
like
a
label
so
for
every
information
that
comes
from
a
specific
source
that
has
the
same
time
this
the
same
tag
they
are
all
grouped
together
under
the
same
chunk
right
and
a
challenge
is
just
an
amount
of
bytes
that
stored
many
records
right.
Usually,
we
handle
chunks
around
two
megabytes,
which
is
a
has
been
really
efficient
for
most
of
the
use
cases
and
so
consider
the
chunk,
which
is
a
more
critical
unit
of
data,
granular
data
that
we
have
in
the
fluid
pipeline.
A
Now
how
this
operates,
for
example,
in
memory
when
the
engine
starts
getting
data.
What
we
do
is
to
create
just
one
chunky
memory
right
and
what
we
do
is
start
appending.
The
events
right
on
that
tank
until
that
chunk
gets
to
two
megabytes
and
that
we
simply
create
if
we
get
more
deformation,
we
just
get
a
new
chart
and
we
do
the
same
procedure.
It
doesn't
matter
if
we
have
the
same
tag
or
not.
A
We
we
just
create
blocks
of
times
a
in
memory
until
one
point
we
have
a
bunch
of
chunks
linked
in
memory,
which
is
pretty
efficient,
is
pretty
fast.
Actually,
memory
buffering
is
the
fastest
implementation,
but
there's
one
downside
right:
it's
not
persistent,
it's
not
persistent.
So
what
would
happen
if
our
service
is
restarted?
What
will
happen
is
start
with
crashes,
yeah
you're
going
to
lose
that
data
and
also
there's
one
more
critical
path.
A
A
If
this
even
is
not
storing
the
things
in
memory
means
that
we're
going
to
start
start
throwing
the
things
in
the
file
system
right
so,
but
this
is,
of
course,
is
most
intensive
on
io,
but
here
we
have
a
couple
of
optimizations
on.
How
do
we
find
the
chunks
on?
How
do
we
store
them
right?
So,
every
time
that
we
get
more
information,
we
are
getting
a
new
chunk
and
just
appending
this
information
to
the
and
to
what
point
that
we
have
many
chunks
in
the
file
system.
But
there's
a
curious
thing
here
here.
A
This
is
a
couple
of
a
simple
example
where
we
have
five
files
right:
five
chunks,
but
if
you
think
about
that,
we
have
2000
30
000,
because
maybe
you
were
not
able
to
send
the
data
because
the
destination
is
down
right,
you
will
say
hey,
but
I'm
going
to
have
all
these
channels
open,
I'm
going
to
use
a
single
file
descriptor
for
each
one,
and
I
would
say
that
now
we
are
going
to
talk
most
about
a
databases,
concept
right.
We
don't
use
any
external
database
right.
A
We
implemented
our
own
data
to
for
this
specific
use
case.
Actually,
we
think
that
databases
are
really
good
and
butter
proof,
but
they
are
for
generic
purposes
right
here
we
are
dealing
with
this
kind
of
a
log
information
that
we
need
to
group
them
in
a
different
way
and
we
need
to
deal
with
file
system
memory,
but
based
on
our
pipeline
logic
and
not
a
generic
database
logic,
I'm
not
paying
the
databases
are
bad.
I'm
saying
that
for
our
purposes
and
performance
gains
our
solution.
A
This
solution
that
we
implemented
works
quite
good
and
now
we're
going
to
jump
into
a
more
elaborate
complex
and-
and
this
is
what
people
really
use
in
production.
Actually,
it's
not
just
a
memory
of
file
system.
What
we
have
is
like
an
hybrid
mechanism
where
all
the
data
that
we
are
creating,
of
course
goes
to
memory,
but
in
the
pipeline
which
says
for
this
input
source
of
data,
for
example,
you
are
taking
log
files
or
listening
for
messages.
A
You
can
say
just
a
key
app
in
memory.
We
have
the
concept
of
up
and
down.
You
can
see
in
the
screen
that
we
have
in
different
colors
in
gray,
for
down
and
a
light.
Green
for
apps
means
that
everything
that
is
up
in
memory
is
ready
to
be
used,
or
maybe
it's
a
change
that
you
can
append
more
data
right,
but
we
protect
the
memory
saying
that
hey,
you
can
have
a
memory
up
to
for
an
example,
a
hundred
of
chunks
right
and
if
each
chunk
is
in
two
megabytes.
A
Well,
you
can
make
you
can
make
the
map
your
own
right,
but
after
that
we
start
storing
all
the
chunks
in
the
file
system.
But
how
do
we
correlate
this
between
all
the
chunks
that
are
in
memory
plus
file
system?
Actually,
what
we
do
is
to
use
a
memory
mapped
files,
which
is
a
really
common
implementation
for
databases
where
we
map
a
file
that
exists
in
the
file
system
as
if
it
were
in
memory,
and
actually
that
is
really
really
performant
right.
A
We
are
not
using
common
system
calls
like
read
and
write,
because
we
try
to
avoid
the
the
context
copy
of
the
data
between
the
kernel
and
user
space.
So
using
memory
mapped
files
is
quite
fast
and
we
can
have
the
control
of
how
the
data
flows
and
if
we
get
ingested,
for
example,
imagine
that
the
output
side
is
down
and
we're
getting
more
data
in
you
are
going
to
face
this
concept
of
back
pressure
right.
A
If
you
don't
have
a
half
ice
system,
you're
going
to
store
all
your
data
memory
and
at
some
point
your
process
is
going
to
crash.
But
what
we
do
is
study
memory
as
much
as
we
can,
based
on
our
own
configuration
limits
and,
after
that,
continue
writing
to
file
this.
Now,
if
you
look
at
the
image,
you
will
see
that
in
the
file
system
part
or
on
the
disk,
you
will
see
that
some
blocks
are
gray
and
others
are
lightweight.
A
On
this
case
on
create.
We
are
saying
that
these
chunks
are
not
up
in
memory.
They
are
just
in
the
file
system
and
when
the
memory
part
gets
empty,
it
cannot
get
empty
but
start
delivering
the
chunks.
We
get
more
room
more
space
to
work
on
the
pipeline.
We
just
take
the
chunks
and
load
them
up
in
memory,
and
this
has
been
implemented
for
almost
two
years
and
being
a
very
scalable
designs
where
we
can
have.
A
There
are
many
variables
in
your
environment
and
then
one
thing
about
this:
all
this
data
processing,
for
example,
with
binary
representation.
We
can
get
the
cpu
and
the
cpu
consumption
low
right.
Also,
all
these
I
o
to
these
that
all
these
hundred
memory
is
quite
more
performant
and
it's
a
really
scalable
design
right
now.
This
is
butter.
Proof-
and
this
is
fluent-
is
quite
useful.
Heavily
on
most
of
my
cloud
providers
like
google,
aws,
microsoft,
azure
and
as
a
team,
we
have
this
kind
of
companies
that
work
together.
A
Actually,
we
have
weekly
meetings
with
most
of
them,
where
they
contribute
also
to
the
project.
They
are
part
of
fluent
bed
and
they're
trying
to
improve
the
whole
fluent
ecosystem
nowadays,
and
this
is
scalable
design,
I
would
say
that
is
most
a
scalable
and
cheap
implementation
in
the
market.
Right
now,
you
might
find
another
solutions,
but
you
will
find
that
one
thing
is
data
rate
until
this
data
consumption
right.
So
if
you
want
to
send
the
data
fast
enough,
yeah
just
support
the
file
disk
right.
A
If
you
want
to,
if
you
want
to
always
have
a
low
cpu,
a
low
memory
consumption,
maybe
you
can
slow
down
ingestion.
Do
some
dropping.
Also
a
one
good
thing
to
mention
for
everybody
is
like
don't
trust
what
I'm
saying.
I
would
say
that
in
general,
when
running
these
kind
of
things,
you
should
run
your
own
benchmark
and
don't
trust
in
benchmarks
that
are
published
on
the
website
right,
neither
trust
in
our
own
ventures.
A
A
Now,
as
the
next
part,
we
are
going
to
do
a
quick
demo
on
how
fluent
bit
operates
and
with
the
time
we
are
going
to
just
explain
a
little
bit
of
the
configuration
a.
How
do
we
achieve
to
sending
just
something
conservative
numbers
like
100
000
records
per
second,
which
is
most
suitable
for
all
cases?
Actually,
that
configuration
is
using
file
system
buffering
right
together
with
this
hybrid
mechanism,
so
I'm
going
to
switch
back
to
the
terminal.
A
A
The
remote
http
endpoint
is
here
on
the
right
and
it's
just
a
basic,
a
tls
with
http
server
that
is
able
to
understand
the
json
records
and
also
expose
a
prometheus
matrix,
okay
e
and
under
this
third
one,
I'm
going
to
run
the
load
generator,
which
is
a
script
that
will
generate
around
a
thousand
lines
per
second
right.
Each
one
will
have
one
kilobyte,
okay,
so
to
get
started,
I'm
going
to
run
flue,
embed
and
now
the
configuration
file.
A
A
A
A
Okay,
in
the
normal
scenario
like
in
a
kubernetes
cluster,
a
delivery
data
rate
will
be
different.
Actually,
you
might
expect
to
have
a.
I
don't
know
a
couple
of
nodes
sending
data
to
your
own
database
right.
It
could
be
elastic
spline,
but
there's
no
like
generic
setup
for
each
one
to
achieve
a
perfect
rate.
Actually,
it's
quite
complex
to
determine
what
would
be
the
ideal
requirements
from
a
hardware
perspective,
but
we
can
see
that
with
this
way
we
can
a
process
data
without
any
major
problem.