►
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Connecting with Prometheus World: Logs and Metrics - Eduardo Silva, Calyptia
From a data collection standpoint, Logs and Metrics were always handled separately: different sources were handled by different "agents". Systems administrators for years have asked for a unified experience where both, Logs and Metrics, were collected, pre-processed, and shipped from a single agent. In the Logging space, Fluentd and Fluent Bit projects are one of the preferred choices, and surprisedly our users have asked that we integrate native Metrics handling, and today this has been implemented. As the Fluent project, we evaluated not only how to collect metrics, but also how to provide smooth integration with the defacto industry standards. We decided to launch our first Metrics support by fully integrating with OpenMetrics and Prometheus. In this presentation, you will learn about the integration, best practices for Metrics collection with Fluent, and how to leverage our Node Exporter, Prometheus Exporter and Remote Write implementations with your current Prometheus services and tooling without any disruptive change. In the Fluent project, we believe in lowering the friction for our users and integrating with their current standard services. These are interesting times where Prometheus and Fluentd are finally in sync.
A
Hello,
my
name
is
eduardo
silva.
Today
we
have
this
quick
presentation
called
connecting
with
prometheus
world
locks
and
metrics.
I
know
most
of
you
might
be
lucky
enough
that
you
are
not
messing
it
up
with
locks
right,
but
we
I'm
coming
from
there
from
the
login
space,
and
I
think
we
can.
We
are
kind
of
the
small
causing
on
this
ecosystem
and
as
far
as
the
community,
we
have
seen
a
lot
of
requirements
on
how
people
whole
users.
A
Companies
can
have
a
more
unified
experience
right
between
logs
and
metrics,
but
not
just
from
the
data
analysis
perspective
when
it
hits
the
back
end.
But
before
that
in
the
right
side
of
the
left
side
from
the
pipeline,
where
in
the
data
collection
happens,
my
name
is
edwardo
silva.
I'm
the
creator
of
this
project
called
fluenbet
one
of
the
maintainers.
A
A
Okay,
before
talking
about
all
this
a
unification
process
and
this
story,
we
need
to
talk
about
a
about
how
logging
works
right,
maybe
you're
familiar
with
it,
but
it's
really
good
always
to
make
some
analogy
in
how
things
interact.
Basically,
in
login,
you
have
one
application
that
ships
one
message
that
message
can
be
sent
to
the
standard
output.
Some
kind
of
stream
can
hit
the
file
system
and,
most
of
the
time,
it's
just
a
route
text
message
right.
A
A
But
if
you
want
to
do
data
analysis,
you
have
to
collect
the
data
process
it
and
send
it
out
in
all
this
process,
which
can
be
summarized
as
like
an
input
and
output.
There's
a
couple
of
things
happening
right.
So
if
we
are
reading
files,
we
have
to
deal
with
log
rotation
files
copy
truncate
we
have
to
deal
with
the
files
were
rotated.
Keep
it
monitoring
for
a
couple
of
seconds,
then
make
sure
that
you
close
all
the
file
descriptors
and
everything
works
smoothly.
A
A
Most
of
the
extraction
traction
started
to
grow
when
aws,
microsoft
and
google
started
using
it.
In
their
own
infrastructure,
nowadays,
just
from
public
repos
fluent
the
influent
bit
are
deployed
more
than
two
million
times
a
day
just
from
the
public
repos.
We
don't
have
any
stats
from
the
private,
but
it
should
be
10x
more.
A
Okay,
so
now
looks
a
metrics.
Let's
talk
about
this
unified
experience
and
we
have
seen
that
in
the
market,
there
are
different
kind
of
tools.
Try
to
accomplish
this
right,
there's
some
kind
of
fatigue
in
terms
of
people
having
multiple
agents
for
different
things,
and
here
we're
not
trying
to
propose
something
to
replace
everything.
A
From
the
fluent
perspective
as
a
project,
we
always
think
and
have
this
mindset
that
we
should
listen
messages,
data
or
anything
from
every
source
and
be
able
to
send
that
data
to
any
destination,
even
if
that
is
a
competitor
right,
avoid
the
vendor
login,
so
that
unified
experience
is
what
users
want,
and
for
years
they
they
told
us.
Hey.
A
Is
your
agent
able
to
handle
metrics
too
and
yeah?
We
kind
of?
Do
it
right
on
our
own
way
is
the
proper
way,
maybe
not,
but
also
from
maintenance
perspective
yeah.
We
have
some
experience
with
matrix,
so
why
we
cannot
try
experimenting
and
creating
a
solution
that
works
with
both
scenarios
logs
and
metrics.
A
So
now
it's
a
quick
intro
about
our
journey
of
these
years
into
the
metrics
world
and
the
kind
of
value
that
we're
trying
to
bring
a
connected
with
connecting
with
the
prometheus
ecosystem.
As
I
said
before,
a
influential
project
specifically
since
the
beginning,
I'm
talking
about
five.
Six
years
ago,
we
always
had
this
metrics
collector
fluent
was
created
originally
for
embedded
linux.
So
we
created
cpu
collector
memory
collector,
but
all
those
metrics
were
handled
as
locks
right.
What
that
means?
A
No
fixed
schema
just
a
simple
structure
in
our
world.
We
use,
we
don't
use
json
when
we
consume
the
data
with
message
pack,
which
is
like
a
kind
of
binary
version,
but
there's
no
enforcement
of
a
physical
structure.
So
if
you
have
a
couple
of
key
value
pairs,
that's
fine,
and,
for
example,
if
I
want
to
gather
cpu
metrics
right,
I
just
gathered
them,
but
they
get
them
as
json
logs,
not
a
symmetric
payload
so
and
that
will
support
the
network.
Thermal
docker
metrics
and
fluid
metrics,
of
course.
A
Now,
if
you
will
compare
both,
there
are
some
in
terms
of
two
spaces
locks
and
metrics
from
one
side
logs.
We
have
as
unstructured
and
structured
messages
right
a
matrix.
You
always
have
a
fixed
data
model,
which
is
give
us
a
lot
of
happiness,
at
least
for
the
logging
world
right.
We
have
a
lot
of
pain
because
when
we
don't
have
that
structure
from
the
other
side
in
logs,
a
user
cares
about
filtering
data
reduction,
data,
enrichment,
kubernetes,
metadata
or
any
kind
of
things,
and
be
able
also
to
do
aggregation.
A
A
In
logs
we
don't
have
a
predictable
size
for
the
data
right.
We
can
get
as
much
as
data
for
one
if
one
of
the
developers
got
a
lucky
day
and
just
enable
debugging
in
their
application.
They
just
went
home
on
monday
we're
going
to
figure
out
that
we're
going
to
have
millions
of
extra
messages,
but
that
we
didn't
need
it
and
we
started
paying
for
them
on
our
splunk
database
or
another
one.
So
it's
hard
to
predict
right.
It's
hard
to
control
in
matrix
yeah,
it's
a
fixed
data
model.
A
You
can
predict
at
a
certain
point
and
well
in
logs,
you
have
map
booleans
integer
floats
pretty
much
like
a
json
and
in
metrics,
of
course,
you
have
more
defined
times
counter
gauges,
instagrams,
and
I
think
that,
for
our
use
case
is
quite
a
quite
good
now,
when
we're
thinking
about
how
to
how
to
bring
value
from
a
fluent
perspective
to
the
metrics
space,
we
went
wanted
to
think
so
what
we
need
to,
what
do
we
want
to
improve
in
the
market?
Do
we
want
to
write
everything
from
scratch
right?
A
Do
we
want
to
implement
our
own
data
model
on
top
of
what
the
market
is
using
and
from
our
mindset?
The
answer
is
no.
We
did
our
research
and
I
think
that
personally,
if
you
want
to
start
adding
value
to
the
users
to
the
companies,
you
have
to
integrate
with
the
standards
that
you
have
in
the
market
and
not
reinvent
the
wheel,
and
if
you're
going
to
reinvent
the
wheel
at
least
make
it
compatible
with
what
you
have
in
the
market
right.
So
we
decided
to
stick
with
prometheus
and
open
metrics.
A
We
say
the
whole
industry
using
prometheus.
We
are
going
to
be
a
good
citizen
with
prometheus.
We
have
to
use
the
same
protocols
the
same
style
of
metrics
there's
the
years
of
experience.
So
why
not
take
it?
Why
not
to
keep
on
that
and
for
us,
prometheus
ecosystem
is
quite
interesting
because
you
have
a
metrospec.
You
have
this
condition
of
collector
exporters
and
also
you
can
ship
the
metrics
over
the
network
with.
Usually
that
is
a
problem
right.
A
If
you
don't
have
the
right
data,
cellularization
format,
shipping
data
over
the
network
might
generate
some
issues,
and
this
is
how,
from
a
fluid
perspective,
we
say:
okay,
we're
going
to
integrate
with
prometheus,
but
that
what
means
right,
because
I'm
not
planning
to
ship
locks
from
fluent
bit
to
prometheus.
But
I
came
more
about
what
kind
of
enhancement
we
can
do
in
the
ecosystem
so
before
to
write
any
integration
on
everything
we
decided
hey.
We
have
all
our
stack
to
manage
logs,
but
also
we're
missing
this
piece
to
manage
metrics
in
fluent
bit.
A
Fluid
beta
is
really
in
c
language,
so
it's
not
like.
We
just
can
go
ahead
and
consume
all
your
your
golang
sdks
right.
It
will
be
a
tremendous
overhead
for
the
project
for
our
perspective,
so
we
started
this
project
called
symmetrics
with
apache
license
pretty
much
the
same
thing
and
we
started
like
reading
the
c.
We
started
to
support
basic
staff,
counter
gauges
and
type
it
yeah
we
needed
and
histograms
are
currently
under
development.
A
We
had
to
make
it
label
somewhere,
it
must
support
atomic
operation
and
actually,
I
can
say,
yeah
it's
pretty
much.
A
copy
of
prometheus
go
client
yeah.
We
took
a
look
at
it
yeah
we
need
atomic
operations,
we
need
labels
and,
if
you're
going
to
start
doing
metrics
stuff,
we
should
take
the
best
practices
from
prometheus.
A
A
Okay,
we
are
going
to
be
a
good
citizen
with
prometheus
right,
but
what
about
some
user
says?
Hey
I'm
using
influx
db,
or
are
you
I'm
using
an
endpoint
I'm
using
chronosphere
when
m3,
I
need
remote
right.
So
how
do
we
play
with
all
these
different
kind
of
payloads
connectivities
and
security
things?
So
we
say:
okay,
symmetrics
will
be
all
about.
How
do
I
create
matrix?
A
How
do
I
manage
its
value
and
how
I
can
take
this
metrics
context
and
convert
it
back
to
any
kind
of
payload
that
we
want,
so
we
extract
the
whole
problem
and
also
we
care
about
okay,
let's
support
the
same
thing
that
prometheus
names
and
subsystem
name,
descriptions
for
help
and
labels
yeah.
Maybe
some
things
will
go
away
with
open,
matrix
changes,
but
that's
fine.
A
What
we're
doing
here
is
just
creating
a
simple
context
of
symmetrics
guiding
what
is
the
current
time
step
create
a
matrix
with
two
labels
right
increment,
the
counter
and
optionally
just
getting
the
value
right,
I'm
not
doing
anything
fancy
here.
This
is
just
a
definition:
incrementing,
a
content,
retrieving
the
value
and
printing
that
value
to
the
standard
output.
A
Now
another
thing
that
we
can
do
here
is
like,
for
example,
with.
If
we
are
using
flux
db,
we
can
use
this
simple
api
to
encode
my
context
to
an
influx
db,
payload
right
and
we're
going
to
that
specific
output.
So
this
is
the
way
that
we
work
in
fluent
bits.
So
no,
we
try
to
be
have
a
agnostic
chord
that
we
take
data.
We
we
unified
one
format
and
compared
to
the
other
symmetrics
does
the
same
thing.
A
We
did
the
same
thing
for
prometheus
spotter
right.
I
have
a
symmetry's
context
and
I
want
to
expose
this
information,
but
this
this
is
the
difference.
Do
you
remember
that
I
said
we
don't
care
about
the
transport
we
just
care
about
the
content
here
we
are
not
using
any
kind
of
http
protocol
of
anything.
It's
just
payload,
just
content.
A
Also,
we
can
do
remote
write.
This
was
a
quite
more
extended
work,
but
we
could
accomplish
using
the
same
prometheus
protocol
buffer
files.
So
with
the
same
specs,
we
are
not
able
to
generate
this
kind
of
a
prometheus
remote
write,
payloads,
okay,
and
now
we
get
back
to
fluent
bit.
So
we
now
we
have
a
line,
a
small
library,
that's
handled,
metrics
that
convert
payloads
and
now,
okay,
now
we
jump
into
the
matrix
part.
A
So
they
ask
it
that
fluent
bed,
since
it
has
a
plugable
architecture,
if
we
can
implement
the
same
metrics
collection
inside
fluid
to
avoid
having
an
extra
agent
right.
This
is
not
just
about
just
complete
one
project
with
the
other
is
about
just
to
add
a
more
flexibility
to
the
users
who
wants
to
have
this
kind
of
unified
experience.
A
Of
course
not
exported.
Has
a
lot
of
hundreds
of
collectors
or
more,
we
try
to
focus
just
on
a
subset
for
this
year,
like
cpu
cpu
frequency
just
for
linux
nets,
stat
eu
name,
bm,
stat,
blah
blah
blah,
and
actually
we
got
a
really
good
reception,
but
that
is
the
collection
site.
So
we
got
flue
embed
with
taximetrics.
We
create
an
input.
Plugin,
that's
gathered
all
this
information
from
the
proc
file
system
and
now
it's
time
to
do
something
with
it.
A
We'd
get
the
data
on
the
input.
Now
we
have
to
expose
that
information,
somehow
so
using
the
same
engine
that
we
showed
some
few
slides
a
ago
now
we
can
use
our
own
built-in
http
server
and
expose
the
payload
of
these
symmetrics
context
in
our
http
server.
But
here's
a
difference
when
you
have
a
an
exporter
most
of
the
time,
the
data
when
you
get
the
request,
it's
just
a
guided
real
time.
On
that
moment,
you
get
the
data
right
here
in
fluid
bit.
We
use
a
different
approach.
A
It's
like
the
input
plugins
runs
at
its
own
interval
of
time,
scrapping
metrics,
scrapping
information,
pushing
that
to
the
output
side
and
in
the
output,
for
example,
the
http
output
plugin.
We
have
this
kind
of
cache
that
is
refreshed
one
every
two
seconds
so
every
time
that
we
get
a
client,
that's
going
to
scrape
this
matrix
like
prometheus.
It's
just
going
to
get
the
data
not
on
demand,
but
the
last
data
that
was
generated
by
the
input
plugin.
A
And
from
a
fluid
configuration
perspective,
this
is
just
a
few
lines.
You
just
write.
You
can
have
multiple
inputs
filters
and
outputs
we're
just
saying
in
the
input.
Just
use,
not
export
the
metrics
plugin
right
associate
attack
to
it.
Script
interval
to
two
seconds
and
in
the
output
side,
use
the
prometus
exporter,
output,
plugin
right
and
you
can
just
get
the
information
without
any
problem.
A
Sadly,
I
will
have
to
skip
the
demo,
because
you
know
we
got
this
linux
problem
today.
So
I
cannot.
I
don't
have
the
environment
to
show
you
the
demo,
but
hopefully
maybe
we
can
start
we
can
or
we'll
try
to
post
something
online,
so
you
can
watch
it
how
it
works
now,
as
a
summary
of
a
in
our
metrics
journey.
What
is
supported
today
inside
fluent
bet,
it's
not
export
and
metrics
for
linux.
A
Our
flow
bit
internal
metrics
now
are
supported
using
basimetrix
and
in
the
output
side
we
can
send
metrics
to
influx
cb,
prometrics
exporter,
prometus
remote
right
so
as
a
kind
of
supported
output
and
forward
forward
is
a
fluency
and
fluent
bit
protocol
to
send
data
over
the
network.
So
if
you
have
another
remote,
a
fluent
bit
instance
that
speaks
forward,
it's
able
also
to
receive
those
metrics
and
reassemble.
A
A
Now,
what
is
the
current
ongoing
work,
and
I
never
expect
this
when
we
started
this
metrics
project
is
that
people
started
asking
for
not
exporter
metrics
for
windows,
and
this
is
crazy
right.
We
always
think
that
everybody's
running
linux,
but
when
you
go
to
the
market
everybody's
using
windows,
2
windows,
server,
of
course,
so
we
are
riding
windows,
exporter,
metrics.
A
A
Another
work
that
is
ongoing
right
now
is
the
ability
to
convert
locks
to
metrics.
Now
you
may
think.
Okay,
when
is
this
needed
right?
Well,
pretty
much.
For
example,
if
you're
reading
a
I
don't
know
nginx
logs,
and
you
want
to
count
the
number
of
200
responses
that
you're
getting
or
trap
the
500
ones,
you
will
be
able
to
create
this
kind
of
metrics
based
the
parsing
of
the
log
and
generating
these
updated
values.
A
In
fluent
bit,
we
also
is
not.
We
started
as
just
afraid
to
take
data
from
one
side
send
it
to
the
other,
but
we
added
many
things
in
the
middle
like
filters,
kubernetes
support,
but
also
we
added
a
stream
processor.
In
it
we
wrote
our
own
sql
parser
and
you
can
run
your
own
sql
queries
on
top
of
your
data
that
is
flowing
through
memory.
There's
no
database,
no
indexing
kind
of
a
string
processing
processing
is
very
similar
to
k
sql.
A
Now
that
allows
us
to
do
fancy
things
like
for
all
the
data
that
matches
a
pattern.
Take
it
create
a
new
stream
and
send
it
out
to
a
different
destination.
Now
the
same
thing
we're
going
to
do
with
metrics
right.
We
want
to
do
create
this
kind
of
metric
processor
that
allow
us
to
take
the
metrics
and
do
some
fancy
processing
aggregate
them
or
do
whatever
we
want
and
maybe
ship
this
data
to
a
different
processing,
different
endpoint,
and
this
will
open
the
opportunity
also
to
create
alerting
or
any
kind
of
other
implementations.
A
Now
the
common
question
is
the
future
open,
metrics,
open,
telemetry
and,
as
I
said,
the
beginning
as
a
project,
we
are
trying
to
be
vendor
agnostic
and
spec
agnostic,
but
we
try
to
implement
right
away
with
what
the
industry
is
using
now
is
prometheus
and
open
metrics
once
open,
telemetry
gets
ga
in
metrics
and
which
I
think
will
be
soon
and
well
looks,
is
a
few
years
away.
We
are
going
to
integrate
also
with
open
telemetry
for
environments
that
the
application
achieve
in
native
applications
metrics
in
open
telemetry
format.
A
C
Hey
eduardo
thanks
for
the
talk
super
interesting
and
I
was
just
wondering
if
you
had
any
plans
for
basically
creating
some
way
to
extract
metrics
from
the
actual
logs,
like
user
logs
flowing
through
fluentd
itself,
as
it's
collecting
application
logs
from
from
end
user
systems
and
and
kind
of
your
thoughts
there.
And
what's
going
to
go
on
in
that
world.
C
Yeah,
I
was
super
interested
to
see,
so
you
know
having
support
for
plugins.
Like
note
exporter
and
the
fluent
b
internals,
and
also
like
other
collections
like
you
mentioned,
statsd
and
collect
the,
and
I
was
just
wondering
if
there
was
any
likelihood
that
there
would
be
development
around
saying.
I
have
logs
that
tell
me
every
time
like
I
have
an
error
so
like
the
error,
log
rate
and-
and
you
know
exporting
that
as
a
counter
or
something
like
that
as
a
metric.
A
Yeah,
actually
that's
one
of
the
ongoing
work
yeah.
It's
a
similar
example
that
I
did
with
a
nginx
locks
right
when
you
want
to
count.
I
don't
know
the
200
errors,
error
rate
and
all
of
that,
and
that's
when
the
collection
standpoint,
but
also
you
want
to
do
this
kind
of
matrix
processor
that
can
help
us
to
take
some
action
based
on
results.