►
From YouTube: Opening + Welcome: Fluent Bit in 2022: Logs, Prometheus and OpenTelemetry - Eduardo Silva, Calyptia
Description
Opening + Welcome: Fluent Bit in 2022: Logs, Prometheus and OpenTelemetry - Eduardo Silva, CEO & Founder, Calyptia
Observability continues to be an interesting challenge, data collection for logs, metrics, and traces is an expensive operation and different approaches exist. Fluentd and Fluent Bit have been the long-term defacto standard for logging, but recently the projects expanded their scope to support and assist with Metrics and Traces. In 2022, Fluent Bit now supports data collection, aggregation, and delivery for the world of OpenMetrics (Prometheus) and OpenTelemetry. In this presentation, you will learn how to collect and aggregate Logs, Metrics, and Traces all together without losing control of your data by connecting different protocols at scale. Fluentd and Fluent Bit embrace OpenMetrics and OpenTelemetry vision, come to learn how to optimize your observability pipelines, and implement the best practices for your production environments.
A
Well,
I
hope
that
you're
doing
well
and
you
had
a
safe
trip
to
come
here
to
capcom,
so
welcome
to
valencia.
My
name
is
eduardo
silva
and
one
of
the
maintainers
of
the
fluent
ecosystem.
I
created
the
project
fluent
bed,
now
maintaining
board
more
companies
more
people
around
and
well
thanks
for
joining
this
fluence.
A
This
is
a
really
small
collocated
session
that
we
found
that
has
to
get
a
lot
of
value
along
the
years.
If
I'm
not
wrong.
This
is
our
third
time
doing
this
event,
and
we
are
really
anxious
to
know
more
also
about
your
journey.
It's
not
just
about
a
maintainers
and
developers
sharing
knowledge,
but
also
it's
important
to
us
to
have
this
interaction
with
you
after
each
session
to
get
received
some
feedback
and
understand
what
are
the
challenges
that
you
have
in
observability
right.
A
A
We
always
intended
to
be
something
that
you
can
pluck
in
your
architecture
and
solve
your
problems
when
it's
about
to
move
data
from
one
point
to
the
other,
and
now
when
we
think
that
that
mentality,
we
think
that
the
fluent
is
more
than
a
tool.
It's
more
than
fluency,
it's
more
than
fluent
bet
is
a
concept
that
allows
you
to
integrate
with
different
systems,
different
architectures,
but
also
other
projects
around
pretty
much
a
part
of
the
history
is
like
from
the
beginning.
A
We
aim
to
even
connect
with
competitors
projects
now
talking
more
about
from
ecosystem.
We
try
to
connect
with
everything
that
is
around,
because
the
problem
is
not
just
a
moving
logs
from
one
place
to
the
other.
There's
traces
there
is
metrics,
but
also
you
don't
want
to
remove
your
your
infrastructure
agents
right,
moving
infrastructure
agents,
for
example,
anyone
that
you
can
have
might
take
you
like
a
year.
A
We
have
a
users
and
customers
who
deploy
this
in
at
scale.
100
000
servers
right
so
when
it's
about
to
switch
technology,
it's
really
expensive
because
it
takes
a
lot
of
time.
A
lot
of
planning,
maintenance
and
so
on
so
for
us
fluent,
is
a
full
ecosystem
that
allows
you
to
get
into
this
journey
on
how
to
connect
the
thoughts
between
different,
a
source
of
information,
different
destinations
and
no
matter.
What
is
the
data
we
try
to
be
diagnostic
today
is
loss
metrics
and
traces.
A
One
of
the
values
is
like:
we
think
that
everything
in
observability
must
be
open
source
right
and
not
just
because
I
say
this
because
we
are
an
open
source
conference.
I
say
this
because
of
the
value
of
contribution
across
different
vendors
companies
and
individuals
of
this,
but
also
it's
really
important
that
you
don't
get
into
this
common
vendor
login,
because
when
you
go
into
observability,
even
if
you
think
from
the
management
perspective,
you
think
that
what
you
need
to
get
is
what
you
need
to
accomplish
is
data
analysis.
A
A
A
A
For
example,
when
fluenty
was
created,
the
problem
at
that
time
was
how
do
I
move
my
login
information
to
a
cloud
service
right
and,
of
course,
the
technology
that
was
created
to
solve
that
problem
was
fluenty
and
actually
part
of
the
story
is
like
a
you
know.
Many
people
complain
that
fluent
is
written
in
ruby
language
with
some
components
in
c,
and
one
of
the
fun
parts
is
that
the
poc
was
mean
to
build
in
ruby,
but
the
real
solution
was
aim
to
be
built
in
c
plus
plus.
A
When
we
got
fluent
the
introduction
already
in
thousands
of
servers,
we
decided
well,
we
needed
to
have
a
kind
of
solution
that
was
more
a
lightweight,
more
performant
with
low
memory
footprint.
At
that
time
it
was
embedded
linux
right.
That
was
a
target
and
then,
at
the
time
that
fluency
was
around
kubernetes
was
around
right,
but
also
start
to
explode.
All
this
a
technology
of
containers
right,
I'm
talking
about
docker
at
that
time
and
flew
in
bed
for
somehow
got
like
a
better
fit
for
this
new
area
of
microservices
and
containers.
A
Why?
Because
it
was
written
in
c
it
was
optimized
to
consume
low
cpu
low
memory,
but
have
a
very
high
throughput
when
it's
about
to
move
data
from
one
place
to
the
other,
so
naturally
users
starting
migrating
from
one
project
to
the
other
in
the
initial
intent
it
was.
The
idea
was
to
have
fluent
bed
like
something
natural
for
fluency
users.
What
fluent
bit
couldn't
work
like?
A
A
A
Fluently
might
be
around
for
another
10
years
fluent
bed,
maybe
more,
but
all
the
innovation
and
the
trend
is
happening
more
in
the
fluent
bed
site,
and
one
of
the
things
about
fluent
bid
is
that
production
grade
system
is
very
high
performance
and
consume
very
low
resources.
Of
course,
it's
not
the
same
process,
a
thousand
messages
per
second,
that
10
or
20
000..
Of
course,
we're
going
to
need
more
competing
resources
for
that
right,
but
our
challenge
as
maintainers
and
developers
that
every
year
we
need
to
optimize
more
more.
A
We
need
to
find
okay
how
to
reduce
memory
allocation.
What
kind
of
trading
strategy
we
can
use,
how
we
can
optimize
in
data
cellularization
before
moving
the
data,
even
how
we
optimize,
when
it's
about
to
encode
the
data
into
json,
which
is
a
really
expensive
task
and,
as
you
might
guess,
every
destination
waits
for
a
json
payload.
So
no
matters.
If
in
the
agent
we
have
a
binary
representation
of
the
data
right,
we
need
to
convert
that
back
to
json.
A
And
now
a
which
is
interesting
same
as
the
pattern
that
happened
with
linux
same
as
the
pattern
that
happens
with
kubernetes,
we
started
to
get
distributions
of
fluid.
The
third
distribution.
Well,
of
course,
is
the
upstream
distribution,
where
we
publish
packages,
container
images
and
then
what
follow
after
that
was
aws
when
they
said
amazon
told
to
talk
to
us
and
well,
aws
are
maintainers
of
the
project
too,
and
they
said
hey.
We
have
our
own
specific
backend
services
and
we
want
to
provide
you
know
a
first-class
connectors
for
our
users
by
using
fluid
right.
A
So
we
started
this
synergy
or
working
together
and
they
created
their
own
plugins
initially
in
go
now
in
c,
and
they
say
we
need
to
package
this
for
our
customers,
so
they
are
using
aws
for
fluent
bet.
Aws
for
fluid
bit
is
just
a
flow
and
bit
distribution
that
contains
extra
amazon
connectors.
Now
you
might
ask
why
aws
for
using
the
forward
and
not
aws
from
bit
is
just
a
foundation
policy
until
we
got
some
conformance
committee,
but
we
aim
to
change
that
shortly.
A
Caliptia,
the
company
that
are
representing
also
we
are
launching
our
own.
A
fluid
version,
but
this
is
not
is
upstream
version
but
kind
of
lts.
We
got
many
customers.
That
said
a
okay.
I
want
to
run
fluenbet,
but
fluentbit
appstream
is
released
every
two
weeks
right
and
sometimes
they're
breaking
changes.
We
try
to
improve
in
testing
and
stuff,
but
sometimes
things
happens.
A
So
if
you're
running
google
cloud
likely,
a
google
ops
agent
will
be
available
for
you.
If
you
deploy,
for
example,
a
kubernetes
cluster
in
google
cloud,
you
will
get
flow
embed
as
a
div
as
a
default
pod
running
so
sorry
as
a
default
demon
center
to
collect
all
the
logs
and
ship
all
this
to
stackdriver.
A
So
cloud
providers
in
in
this
aspect
from
project
and
community
base
is
really
important
because
you
see
that
technology
is
has
been
growing
and
everybody
now
is
contributing
back
to
have
a
more
general
value
and
fluent
aims
to
connect
the
thoughts
between
all
the
sources
available,
all
the
destinations.
So
this
is
not
new.
A
Actually,
this
is
a
the
first
button
that
we
envisioned
when
fluentd
was
created
as
part
of
the
community,
we
are
seeing
a
huge
traction
on
how
many
docker
hub
pools
we
are
getting
every
year
if
you
compare
from
2019
when
we
started
publishing.
This
is
just
containers
public
containers.
We
started
with
just
a
few
hundred
right
now.
A
Oh
this
is
the
order
of
millions.
Sorry,
but
I
just
forget
the
unit
right.
So
in
2021
last
year
we
hit
600
million
deployments
and
this
year
and
if
you
think
we're
just
in
may
we're
just
approaching
that
value,
actually
cncf
just
reported
that
flumit
was
deployed
more
than
a
billion
times
right,
and
that
is
insane
it's
insane,
because
it's
a
huge
ad
option.
Right
and
now,
with
more
adoption,
you
get
more
back
reports,
more
enhancement
requests,
and
this
is
like
a
snowball
that
continues
growing
and
it's
really
interesting
because
there
are
more
challenges.
A
You
might
think
that
an
agent
that
solves
a
a
data
problem
will
be
done,
but
actually
every
year
at
at
least
they
come
in
the
this
cloud
environment
is
more
heavily
every
years
more
challenges
more
stuff
that
we
need
to
do
or
innovate
into
the
space
as
part
of
investments
from
maintainers
and
other
companies.
We
have
been
working
on
how
to
make
the
fluid
project
a
stronger
right,
how
to
avoid
regressions,
how
to
in
increase
our
coverage
of
issues.
So
we
incremented
all
our
overseas
cicd
pipeline
on
github.
A
So
every
request
everything
that
has
been
merged
go
to
a
very
strict
sanitization
process
and
making
sure
that
there's
no
problem
and
also
it's
no
problem
running
in
different
architectures
right.
We
aim
to
support
the
four
basic
architectures
x86
in
two
flavors
same
as
arm
and
also
making
sure
that
there
are
no
regressions
as
a
part
of
security.
It's
like
okay,
fluid
beat
is
written
in
c
and
there's
always
this.
This
fear.
Okay,
what
about
memory
safety?
What
about
crashes,
check
folds
so
from
a
language
perspective?
A
What
this
means
fasting
is
a
security
mechanism
where
you
take
any
component
of
the
project
that
receives
an
input
right
and
it
tries
to
put
a
similar
input
or
a
random
input
or
a
modified
input
that
aims
to
make
the
project
to
crash.
It
just
send
trash
data,
and
what
is
the
interesting
thing
in
the
first
day
that
we
run
this?
We
found
many
bugs
many,
many
many
many
and
in
the
last
12
months,
all
of
that
has
been
fixed.
A
They
are
not
relevant
backs
that
you
might
face
in
a
daily
in
a
daily
basis,
but
if
your
service
is
being
attacked,
you
might
face
this
kind
of
problems,
but
google
os
faster.
The
good
thing
is
that
it
runs
24
by
seven,
so
every
what
we
are
doing
is
that
increasing
the
coverage
of
the
fasteners
for
every
interface
in
fluid
and
the
result
is
you
have
a
more
stable
agent
as
a
more
stable
product
and
also
as
a
project
perspective.
A
It's
like
for
the
login.
I
think
that
we
don't
need
to
elaborate
too
much.
We
support
instructor
structure
messages.
We
do
all
of
this
in
a
schema-less
fashion
and
we
do
all
the
processing
filtering
enrichment
of
data,
and
one
of
the
good
things
is
that
we
allow
influenbit
to
bring
your
business
logic
into
the
data
pipeline.
A
Now
the
matrix
journey
is
really
interesting
because
most
of
I
don't
know
which
comfort,
I
think
was
cubecon
europe
some
years
ago
and
they
told
us
hey
I'm
using
what
was
an
example
at
that
moment,
I'm
using
a
prometheus
note
exporter,
I'm
using
fluent
bit
not
exporter
for
matrix
fluid
bit
for
logs.
Can
you
replicate
the
same
functionality
inside
fluid
and
it
was
not
just
one
person
many
many
people
were
asking
about
that.
A
Now,
from
a
plug-in
perspective,
we
did
was
a
you
know.
Our
pipeline
has
inputs
we
or
sources.
There
are
plugins
that
can
read
data
from
some
place
or
receive
it,
and
we
have
a
outputs
right.
How
do
we
connect
to
a
destination
in
the
input
side
we
implemented
now
a
prometheus
scraper
so
fluent
bit
can
scrape
your
prometus
endpoints.
A
If
your
application
is
running
exposing
some
metrics,
we
just
can
connect
fluent
bait,
scrape
those
metrics
and
get
it
get
them
into
the
pipeline.
Also,
we
have
another
plugin,
which
is
the
mimic
of
node
exporter
from
primitives
that
runs
locally
and
is
create
the
same
metrics
that
node
exporter.
Does
we
support
a
coverage
around
60
of
what
the
real
news
exporter
does,
but
I
think
that
that
covers
80,
85
percent
of
majority
of
use
cases.
A
That's
why
we
get
the
data
into
the
pipeline.
Now
in
the
output
site,
you
can
expose
this
data
with
prometheus
exported
plugin
that
we
have
or
remote
write
most
of
a
services
that
are
supporting
prometheus
remote
right
like
well
new,
relic
google.
You
can
connect
to
them
by
using
these
connectors,
so
you
can
put
fluid
in
the
middle
of
something,
no
matter
what
you
have
as
logs
metrics.
You
can
use
it
to
get
this
data
and
ship
this
data
to
your
most
famous
destination
and
the
same
is
happening
with
open
telemetry.
A
The
way
that
the
flow
employee
sees
open
telemetry
we
see
it
like
is
where
the
industry
wants
to
get
a
standard,
for
example,
from
if
you
go
to
any
environment,
the
stat
the
industry
standard
for
metrics,
everybody
will
say
prometheus
right.
Everybody
use
primitives.
That
is
perfect,
open
telemetry.
A
If
you
ask,
for
example,
what
is
the
industry
standard
for
traces?
Everybody
will
say
now:
it's
open
telemetry,
but,
as
you
can
see,
the
standards
always
are
switching
or
evolving
right.
But
at
some
point
you
are
overlapping
features.
You
are
overlapping
protocols
and
you
need
something
that
runs
in
the
middle.
A
But
we
don't
aim
to
replicate,
for
example,
the
open
telemetry
collector.
We
don't
aim
to
implement,
for
example,
traces
processing.
I
think
that
is
open,
telemetry,
job,
right
or
bendo's
job.
Our
job
is
to
be
able
to
move
the
data,
but
in
the
metrics
space
we
are
also
we're
going.
We
are
going
to
allow
to
do
some
kind
of
matrix,
processing
and
yeah.
This
will
be
launched
in
q4
2022.
A
Another
stuff
that
is
really
important
is
investment
in
performance.
As
I
said
at
the
beginning
of
the
presentation,
you
know
you
and
your
environments
are
always
getting
more
data,
more
data,
more
data,
and
we
need
to
be
able
to
handle
that
load.
So
we
have
another
presentation
from
amazon
where
they
contributed
a
new
mechanism
for
a
the
even
loop
that
we
have
now.
You
can
have
priority
queues
and
all
the
events
run
smoothly.
There
there's
no
event
loss.
A
A
Okay,
those
are
the
users
now
developers
is.
How
do
we
extend
this
agent?
How
do
I
put
my
business
logic?
How
to
implement
in,
for
example,
I'm
a
company,
and
I
want
to
provide
a
new
way
that
and
take
advantage
of
fluent
bid
users,
so
they
can
connect
to
my
services.
So
we
from
some
time
ago
we
support
golang
in
the
opposite
side
of
fluid,
but
one
of
the
missing
pieces
was
the
input
side,
and
now
we
are
launching
shortly
this
year,
golan
input
plugins.