►
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Fluent Bit: Past, Present, and Future - Eduardo Silva, Calyptia
Fluent Bit is a project that gained high popularity in the ecosystem, with high growing adoption of our users between individuals and companies. The project vision, status, and future roadmap are such important as our efforts as a project must be aligned to solve market needs. In this presentation, Eduardo Silva (project creator) will talk about the past, present, and future of the project.
A
A
Now
we
started
talking
now
a
more
interesting
topic.
Right,
metrics
yeah,
you
say
fluent-
is
born
for
logs,
it's
one
bit
born
for
logs,
but
we
always
were
a
project
which
is
kind
of
diagnostic
right.
We
didn't
do
any
kind
of
special
things.
Besides
serialized
data
optimized
memory,
buffers
do
retries,
but
if
you
abstract
from
that
it
could
be
anything
so
this
year,
the
fluent
project
and
fluentd,
we
decided
to
jump
into
the
metric
space
pretty
much
because
we
were
already
there
at
the
beginning.
A
As
I
said,
this
started
from
very
linux
and
we
collected
metrics
before
we
had
plugins
to
collect
cpu
samples,
this
io
network
io
and
all
of
that,
but
as
a
structure
logs,
not
as
metrics
right
and
there's
a
big
difference.
When
you
handle
data
and
structure
logs,
you
can
have
a
structure,
but
maybe
you
don't
have
a
fixed
schema
right,
which
is
totally
different.
A
Now
how
this
correlate
is
like
logs
are
instructors
through
the
message
in
metrics.
You
have
a
fixed
data
model
in
logs.
You
need
filters
to
reach
data.
Do
data
reduction
or
aggregation
which
is
optional,
metrics
just
aggregation.
Now
it
logs
you
don't
have
a
predictable
size
for
the
data
right
as
much
data
you
have.
You
have
to
optimize
memory
performance,
everything
in
metrics,
pretty
much
you
know.
A
What's
the
maximum
size
for
each
matrix
yeah,
you
cannot
know
how
many
mixture
you're
going
to
get
or
create
right,
but
if
you
have
more
control
than
logs
and
it
locks
you,
I
have
map
booleans
integer
floats
any
kind
of
data
types
and
networks
kind
of
counter
gauges
histograms.
So
you
can
minimize
the
use
case.
A
So
we
started
to
think
what
kind
of
value
we
can
bring
to
the
metric
space
right
and
because
every
user,
even
coming
from
this
conference,
keep
going
from
years
ago.
Everybody
says
I
don't
want
to
manage
multiple
agents.
For
me,
one
for
matrix
one
for
logs
can
we
have
a
more
unified
experience
so
and
we
started
to
think
okay.
What
is
the
current
standard
in
the
industry
right
now?
It's
pretty
much
prometheus
and
open
matrix
right.
A
That's
what
the
industry
is
using
today,
I'm
pretty
much
to
say:
okay,
let's
align
on
that,
let's
use,
let's
align
with
the
industry
using,
but
since
we
are
also
vendor
agnostic,
meaning
we
don't
get
married
to
just
a
specific
backend.
We
also
can
be
flexible
enough
to
say,
okay,
we're
going
to
what
the
industry
is
using
today
and
adapt
to
what
the
industry
is
using
tomorrow,
and
we
started
this
small
project
because,
if
you're
going
to
jump
into
something,
you
wanted
to
know
how
to
take
advantage
of
the
ecosystem.
A
We
created
a
lightweight
project
called
symmetrics,
because
we
had
all
the
engine
for
logs
roading
buffers,
but
metric
was
something
new.
So
how
do
you
approach
the
problem?
We
created
a
project
called
symmetrics,
a
very
lightweight
library,
written
in
c
language
that
managed
all
of
these
counter
gauges.
Untyped
metrics,
histograms
labels,
atomic
operations
and
it
was
quite
highly
inspired-
quote
copy
paste
of
prometus
go
client
right.
We
took
the
same
approach
there's
years
of
experience
there.
We
don't
need
to
reinvent
the
wheel.
A
We
just
don't
need
to
implement
it
for
our
own
purposes,
and
one
of
the
benefits
of
symmetrics
project
is
that
it
allows
us
to
create
net
context
of
metrics
serialize.
The
data
to
send
this
information
over
the
network
also
convert
a
context
of
metrics
to
prometheus
format,
to
influx
db,
to
promote
this
remote
right
so
be
agnostic
on.
How
do
we
handle
the
data,
and
this
is
different.
One
things
handle
content
and
the
other
thing
is:
how
do
we
handle
transport?
A
If
you
separate
the
problems,
you
can
come
up
with
a
better
solution,
so
symmetrics
allow
us
that
pretty
much.
This
is
how
you
create.
This
is
a
c
code
on
symmetrics,
pretty
much
you
just
create
a
context
for
symmetrics,
you
get
a
timestamp,
you
create
a
counter
and
you
can
have
labels,
and
then
you
just
can
increment
it
retrieve
the
value
and
print
f,
the
value
to
the
standard
output,
or
do
something
with
it
right.
This
is
kind
of
the
same
thing
is
what
you
do
in
prometheus
go
client.
A
I
don't
think
that
the
api
is
so
different,
pretty
much
the
same
number
of
lines,
but
also
you
can
say
when
you
have
this
metric.
As
I
say
before,
hey
I'm
going
to
send
this
data,
maybe
to
influx
db.
Okay,
just
use
this
api
and
convert
it
to
influx,
and
you
will
get
the
output
that
you
have
down
right.
So
we
have.
We
reduce
the
complexity
for
the
users
on
how
to
handle
the
metrics
payload
same
for
promitex
exporter,
yeah
use
a
different
api
function,
name
and
just
export
this
to
prometus
yeah.
A
A
We
say:
okay,
the
major
pain
from
our
users
in
the
metric
space
is
that
they
want
to
have
a
unified
engine,
but
they
are
using
novades
prometheus,
not
exporter,
which
is
a
tool
that
collects
metrics
from
the
host
and
they
told
us
hey
why
you
don't
create
a
bit
input
plugin
that
gathered
the
metric
in
the
same
way
that
node
exporter
does
great.
So
we
kind
of
implement
a
subset.
A
So,
what's
next
for
fluid
right,
we
talked
about
metrics.
I
would
say
that
the
first
months
of
this
year
this
was
the
primary
work,
but
we
still
have
a
lot
of
things
to
do,
and
what's
next
I
would
say
for
the
next
quarter
and
part
of
the
the
first
most
of
the
next
year
is
to
implement
the
same,
not
exporter,
but
for
windows
yeah,
you
might
be
surprised,
but
from
a
caliph
perspective,
all
of
our
customers
needs
to
have
the
same
metrics
collection
for
windows
they
asking
for
the
same.
A
There
are
many
applications
that
ship
locks
as
metrics
as
a
json
payload,
but
we
don't
have
a
way
to
say
hey.
This
is
a
metric.
Please
create
a
counter
based
on
this.
That
is
coming
up
shortly.
Also,
we
are
going
to
start
implementing
all
the
connectivities
for
a
splunk
metrics
data,
doc,
metrics
and
cloud
watch.
A
metrics.
A
The
last
paint
that
people
is
asking
even
we
got
a
ticket
today-
hey
please,
can
we
write
output
plugins
in
rust.
You
know
fluent
bits
written
in
c
language
and
why
in
russ,
because
they
say
it's
cool
and
that's
fine
right,
but
we
get
requirements
to
write
plugins
and
go
in
rus
in
different
languages,
even
in
javascript,
so
we're
going
to
implement
some
web
assembly
layer
on
top
of
flambe.
A
We
tried
to
do
this
this
year,
but
it
was
too
much
work,
so
we
got
deferred
to
the
next
year
and
so
we're
going
to
provide
the
option
to
use
it
to
create
their
own
filters,
their
own
plugins
in
their
preferred
language.
If
we
use
webassembly,
they
can
use
java,
javascript
c
go
rust
or
whatever
they
want.
A
A
A
Yeah
that
I
think
that
would
be
great
actually
when
we
started
this
year.
We
said
okay,
what
are
the
two
things
that
we
can
do
that
will
bring
more
value
to
the
users
and
community
and
companies,
and
we
got
two
right,
webassembly
and
evpf,
but
also
we
were
working
in
metrics,
so
we
had
to
say:
okay,
there's
one
step
at
a
time
we
started
with
metrics
evpf
actually
is
in
the
roadmap.
I
didn't
want
to
mention,
because
I
don't
want
we
don't
want.
We
don't
have
yet
clarify
what
are
the
first
steps
right?
A
A
A
The
only
ga
part
at
the
moment,
as
I'm
aware
of,
is
just
a
tracing
right
for
metrics
has
not
hit
ga
so
once
it
hit
j
through
a
bit
project
will
create
its
own
connectors
to
receive
metrics
from
application
that
use
that
kind
of
telemetry
system
for
logs
is
hard
to
say.
I
think
that
in
the
login
area,
they're
a
bit
delayed
on
this,
I
hear
from
some
club
providers
might
take
one
year
or
two
years.
A
A
A
That
is
the
transfer
protocol.
If
I'm
not
wrong
right
yeah,
that's
the
way
that
we
work.
Actually,
we
just
try
to
be
protocol
and
diagnostic,
so
everything
that
hits
ga
we
will
start
implementing
right
away
because
we
want
to
be
this
kind
of
middle
layer
that
can
help
in
this
architecture,
but
without
replacing
a
being
a
drop
in
replacement
vendor
looking
solution,
so
we're
going
to
implement
all
these
protocols,
as
we
have
done
this
for
years,
even
from
syslog
to
be
forward
logs
touch
protocols.
A
A
A
for
fluentd,
I'm
not
sure
if
there's
a
rubygen,
that
does
that
probably
there's
something
on
work
you
have
to
consider
that
fluent
deploy
is
quite
big.
We
have
thousands
of
plugins
available,
so
yeah.
We
don't
track
each
one
of
them,
so
there
might
be
something
in
fluent
bet,
which
is
a
more
focused
set
right
now,
yeah,
we
don't
have
it
at
implementation.