►
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Integrating Prometheus and InfluxDB - Paul Dix, InfluxData (Intermediate Skill Level)
Many Prometheus users would like a long term store of record for their data. InfluxDB can fill this role with recent additions to add support for Prometheus standards and protocols. This talk will show how InfluxDB supports the pull and Prometheus metrics formats. We'll look at Prometheus' remote/read and write interfaces and how those can be used to integrate with InfluxDB. Finally, we'll dig into InfluxDB's new open source query engine, which will allow Prometheus users to perform federated queries against any number of Prometheus and InfluxDB servers. This talk will also show how other monitoring projects can adopt the Prometheus standards to play in the larger ecosystem to give developers more tools for their Kubernetes and application monitoring needs.
"About Paul
Paul is the creator of InfluxDB. He has helped build software for startups, large companies and organizations like Microsoft, Google, McAfee, Thomson Reuters, and Air Force Space Command. He is the series editor for Addison Wesley’s Data & Analytics book and video series. In 2010 Paul wrote the book Service Oriented Design with Ruby and Rails for Addison Wesley’s. In 2009 he started the NYC Machine Learning Meetup, which now has over 7,000 members. Paul holds a degree in computer science from Columbia University."
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
So
I'm
sure
everybody
already
knows
Prometheus
just
in
case
you
don't
know
in
flux,
it
is
an
open
source
time
series
database,
it
is
MIT,
licensed
it's
written
and
go.
It
has
a
query
language
right
now
that
looks
kind
of
like
SQL
I'll
show
an
example
of
that.
In
a
second
a
few
years
ago,
we
decided
to
write
our
own
storage
engine
from
scratch,
which
is
a
difficult
undertaking.
A
We
call
it
the
time
series
merge
tree,
it's
basically
like
an
LSM
tree,
but
specifically
designed
for
our
use
case,
and
actually
the
database
is
kind
of
like
two
databases
in
one.
One
part
of
the
database
is
an
inverted
index
that
matches
metadata
that
described
time
series
with
the
underlying
time
series
itself
and
then
the
other
is
the
actual
compressed
time
series
storage
which,
on
disk
it
looks
very
much
like
a
columnar
store
and
we
have.
A
We
use
different
types
of
compression,
depending
on
the
shape
of
the
data
and
then
finally,
we
have
a
commercial
offering,
which
is
our
high
availability
or
scale-out
clustering
stuff.
So
the
data
model
of
influx
looks
like
this.
You
have
a
measurement
name,
which
is
a
string.
You
have
tags
which
are
key
value
pairs
were
the
values
or
strings,
and
this
is
all
indexed
data,
and
then
you
have
a
set
of
fields.
So
with
Prometheus
measurement
is
basically
like
metric
and
tags
are
basically
the
same.
A
The
one
thing
that
we
have
different
that
Prometheus
doesn't
have
is
you
can
have
multiple
fields
with
an
individual
data
point
and
then
the
other
key
difference
is
our
time.
Stamps
are
in
nanoseconds,
whereas
Prometheus
they're
in
milliseconds
I
was
actually
pretty
surprised.
The
first
time
I
heard
of
people
using
nanosecond
stuff,
but
we
do
have
people
who
use
the
nanosecond
precision
time
stamps
so,
basically
like
people
and
like
high-frequency
trading,
where
they
have
atomic
clocks
that
guarantee
there
are
less
than
300
nanoseconds
worth
of
clock
drift
globally.
A
They
actually
track
the
performance
of
their
network
hardware
at
that
level.
So
one
thing
that's
a
little
bit
different
about
influx
from
other
time.
Series
databases
is,
we
actually
support
multiple
data
types.
It's
not
just
floats
so
float
64
in
64,
you
and
64
mullions
and
strings
so
strings
open
up.
Some
interesting
use
cases
around
basically
keeping
log
data
inside
in
flux,
which
is
indexed
using
the
same
metadata
as
your
time
series
data.
A
A
So
the
the
first
thing
we
did
to
do.
Integration
with
prometheus
was
basically
adding
support
for
from
these
Prometheus's
remote
read
and
write
API,
so
from
easiest
has
a
thing
where
it
can
actually
replicate
its
data
that
it's
scraping
into
remote
targets,
and
it
can
also
query
that
data
out
of
remote
targets
as
well.
So
one
question
I
got
asked
is
like:
why
would
we
do
this?
It
seems
like
Prometheus
and
influx
are
competing
projects,
so
why
would
we
try
to
integrate
them?
I
think
there's
there.
A
A
So
if
you
want
a
guaranteed
long-term
data
store,
you
can
replicate
all
your
Prometheus
data
out.
Another
nice
thing
is:
if
you're
replicating
it
out,
you
can
have
basically
queries
that
span
data
that
is
collected
from
multiple
Prometheus
servers
right,
so
you
can
basically
effectively
get
federated
queries
but
of
a
single
source,
and
the
other
nice
thing
is
that
obviously
we
support
push
as
well.
A
So
you
you
can
do
push
as
well
as
pull
based
data
and
for
some
things
like
push,
is
actually
really
nice
right
for
things
that
are
more
more
like
events,
as
opposed
to
samples
that
are
collected
at
fixed
intervals
of
time.
So
the
other
nice
thing
is,
you
can
have
this
concept
of
like
ephemeral,
Prometheus
right
as
anybody
who
is
running
kubernetes
and
production
is
painfully
aware
state
in
kubernetes
sucks
it's
much
much
better
if
you
don't
have
to
deal
with
state.
A
So
if
you
can
just
deploy
your
Prometheus
servers,
have
like
each
application
team
run
their
own
Prometheus
server,
but
have
a
central
place
where
they
can
all
write
their
data
up
into
right.
So
then
you
can
run
those
as
basically
stateless
applications,
and
you
can
just
have
the
operator
make
sure
that
one's
always
running.
A
So,
let's
get
into
the
remote
read/write
API,
so
basically
to
turn
that
on
first
you
have
to
go
into
your
Prometheus
config
and
tell
it
where
to
go.
So
this
is
the
endpoint
here
I'm
just
pointing
at
my
local
host,
because
I
was
doing
development
there,
so
that-
and
this
is
an
endpoint
inside
in
flux,
DB
and
that
tells
Prometheus
to
write
the
data
there
and
also
when
you
do
a
query,
it
will
pull
data
from
there
as
well
and
combine
it
with
the
Prometheus,
the
data
that
Prometheus
actually
has
so
for
us.
A
The
writes
are
kind
of
easy
to
think
about
right.
How
do
we
translate
the
the
Prometheus
data
model
into
the
influx
data
model?
Well,
we
could
just
say
the
measurement
name
is
the
metric,
and
that
field
is
just
something
we'll
call
value
right.
So
we
won't.
Everything
will
just
always
have
one
field,
but
the
tricky
part
is
there.
There
are
things
that
you
can
do
in
the
Prometheus
queries
right.
A
So
here's
an
example:
Prometheus
query
where
we
have
a
couple
of
different
natures
and
we
have
a
regex
matcher
and
then
we
have
a
not
equals
matcher.
So,
for
for
influx,
their
current
query
language
for
influx
QL,
the
not
equals
matches,
are
actually
problematic.
We
don't
actually
support
that
against
the
measurement
name.
So
here's
what
that
looks
like
in
the
premies
is
code.
They
basically
have
like
a
protobuf
object
that
specifies
what
a
query
is.
You
have
the
start.
A
Timestamp
you
have
the
intent
timestamp
and
you
have
a
set
of
matches
and
the
matches
are
always
combined.
Basically
using
a
logical
and
our
problem
was
we
had
to
figure
out
how
to
support
these
two
use
cases
which
we
couldn't
do
with
the
influx
query
language
out
of
the
box,
but
we
can
do
it
against
tags.
So,
essentially,
when
I
originally
wrote
this
feature,
this
was
back
in
September.
A
I
just
did
this
kind
of
like
ghetto
hack
right,
so
I
basically
made
the
measurement
name
and
the
field
name
kind
of
no-ops
I,
just
shoehorned
all
the
data
into
tags
right,
which
is
actually
when
you
look
at
what
Prometheus
does
under
the
hood.
That's
also
what
they
do
like
a
metric
name
is
actually
just
a
special
tag,
called
underscore
underscore
name
underscore
underscore
right
under
the
covers
they.
They
work
with
it
like
it's
a
tag
so
essentially
like
that's
how
I
wrote
the
date.
A
But
the
problem
is
like
it's
great,
if
you're
just
using
Prometheus
to
query
the
data
out
of
that
influx
server.
But
if
you
want
to
query
the
influx
server
directly
having
that
schema
kind
of
sucks,
because
you
end
up
not
you
end
up
not
using
the
measurement
name
at
all,
not
using
the
field
and
that's
good
like
descriptive
stuff
that
you
want
to
use
to
describe
your
data.
So
let
me
talk
really
quickly
about
the
work
that
we've
been
doing
on
2.0.
A
That
has
enabled
us
to
make
something
slightly
better
than
what
I
done
in
September.
So
the
first
big
thing
that
we're
doing
is
we're
creating
a
new
query
language,
which
is
great
another
query.
Language
I
have
a
whole
talk
about
this
one,
that's
a
separate
thing,
but
I
that,
when
working
with
time
series
data
sequels
an
abstraction
that
is
kind
of
clunky
right
time,
series
isn't
really
sets
so
I.
Think
the
the
abstraction
that
works
really
really
well
is
a
functional
style.
Very
much
like
graphite
I.
Think
that's
one
thing
that
graphite
did
really
well.
A
That
is
because
we
want
to
be
able
to
iterate
on
the
language
quickly
without
putting
the
data
at
risk
right
because
there's
it's
it's
a
very
high
cost
to
update
your
data
plane,
it's
less
cost
to
update
an
application
tier,
that's
basically,
stateless
right,
so
we
can
iterate
and
deploy
on
these
query
processors
and
have
a
much
like
higher
like
much
more
frequent
releases
there
that
don't
put
the
storage
tier
at
risk
and
then
the
other
nice
thing
is:
you
can
scale
those
independently.
So
this
is
very
similar
to
how
Amazon
Athena
works.
A
So
Athena
is
basically
the
Presto
sequel
engine
on
top
of
s3
and
you
pay
for
storage,
which
is
a
3
and
you
pay
for
compute,
which
is
just
the
query
processing
the
other
nice
property
you
get
out
of
this.
Is
you?
Can
trivially
create
workload,
isolation
right?
If
it's?
If
the
query
processor
is
a
shared,
nothing
system,
you
can
spin
it
up
in
a
new
container.
You
can
set
resource
limits
on
it
right.
A
It's
it's
nice,
because,
if
you're
trying
to
design
a
multi-tenant
system,
one
you
get
that
workload
isolation,
but
even
if
you
don't
have
a
multi-tenant
system,
what
I
frequently
see
is
data
scientists
will
have
a
separate
production
cluster
than
the
actual
prod
and
the
reason
they
do.
That
is
because
their
their
query,
workloads
are
kind
of
crazy
and
they
want
to
isolate
that,
but
it
sucks
to
have
to
replicate
an
entire
product
production
infrastructure
just
to
isolate
the
data.
A
Science
query
workloads,
so
our
goal
here
is
that
we
don't
have
to
duplicate
the
storage
piece,
but
we
can
still
spin
up
new
stuff
on
the
fly
for
the
data
scientists
to
work
and
not
you
know,
hit
prod
and
bring
it
down
or
anything
like
that.
So
as
part
of
the
work
for
AF
QL
and
this
new
execution
engine,
one
thing
we
had
to
do
was
we
had
to
basically
create
a
new
like
lower
level
like
query
tier
that
works
with
the
database.
A
So
essentially,
we
have
a
new
like
protocol
that
we
have
it's
basically
just
TCP
and
protobufs
right
now,
we're
actually
going
to
be
switching
it
to
use
Apache
Aero
is
the
data
interchange
format,
but
it's
much
more
primitive
than
in
flux,
QL.
Basically,
it's
it
looks
more
like
those
Prometheus
mattress
right,
you
have
a
start
time.
You
have
an
end
time.
It
can
pull
back
the
data
and
then
it
can
send
it
to
the
query
processor.
We
also
have
stuff
where
we're
going
to
be
pushing
down
processing
as
well.
A
So
essentially
we
can
do
you
know
if
you're
doing
you
know
if
you're
doing
like
min
or
max
in
certain
buckets
of
time,
you
can
compute
those
locally
on
the
node
and
then
have
the
query.
Processor
node
basically
just
combine
the
results
so
with
that
work
that
went
in
that
work
is
in
the
1.5
release
of
influx
DB,
which
we
cut
about
a
month
ago,
but
with
that
work,
I
was
able
to
update
the
Prometheus
stuff
so
that,
instead
of
using
the
influx
QL
query,
engine
I
could
just
bypass
it
and
I
this.
A
This
new
storage
query
tier,
like
works
in
a
way
where
I
can
actually
now
just
have
the
measurement
name,
be
the
metric
name,
so
that
I
can
go
to
my
database
and
I
can
say
this
is
the
info
influx
command-line
interface
and
I
can
do
Oh
show
my
measurements
I
can
see.
Obviously
this
is
a
bunch
of
stuff
from
from
Prometheus,
so
this
work
isn't
yet
in
a
build
I.
Have
it
on
a
branch
right
now
on
the
on
the
influx
DB
repo,
it's
there.
A
A
So
one
of
the
other
big
things
that
we're
doing
with
with
our
efforts
around
200
Vince
is
we're
also
trying
to
decouple
the
idea
of
the
query
language
from
the
compute
like
the
computer
from
the
processing
engine.
So,
essentially,
what
we're
doing
is
we
have
a
parser
that
will
parse
the
language
into
a
dag,
so
every
single
query
in
influx
can
be
represented
as
a
directed
acyclic
graph
right
and
the
idea
there
is
that
we'll
be
able
to
have
any
number
of
query
languages
on
top
of
this
execution
engine
right.
A
So
obviously,
on
the
new
execution
engine
we
have
to
build
support
for
influx
q,
ow
right
for
better
or
worse.
We
have
to
continue
to
support
that,
because
a
lot
of
people
do
love
it,
even
though
I
think
the
functional
style
is
better.
We
also
want
to
build
support
for
tic
script,
which
is
the
language
that
we
designed
for
capacitor,
which
is
like
query,
processing
and
stuff
like
that.
A
But
the
idea
is
that
I
fql
is
going
to
be
the
one
language
that
unifies
the
platform
as
a
whole,
whether
you're
doing
interactive
batch
queries
or
you're,
doing
background
processing
or
ETL
tasks
or
monitoring,
alerting
and
then,
ideally,
what
we'd
like
to
do
later
this
year.
Is
we
like
to
add
a
prom
ql
layer
on
top
of
it?
A
So
you
could
actually
just
issue
prom
ql
queries
to
an
influx
server
and
it'll
know
what
to
do
and
who
knows,
maybe
other
other
query
languages
will
pop
up
as
well
the
the
thing
about
query
language
design,
language
design
in
general.
The
thing
that
I
realized
is
it's.
It's
really
like
an
aesthetic
thing.
You
can
get
a
group
of
engineers
reasonable
people
to
disagree
about
what's
good
in
language
design
and
they
could
all
be
right
at
the
same
time,
but
it's
a
matter
of
aesthetics
and
what
you
prefer
to
work
with.
A
So
in
my
ideal
world
we
can
have
languages
that
some
people
like
working
with
that
feel
more
like
expert
languages
or
whatever
and
other
languages
that
are
easier
for
beginner
use
cases
and
stuff
like
that.
We
want
to
support
as
broad
of
a
community
as
possible
and
ultimately,
what
we
want
to
do
around
influx
and
the
projects
that
we're
building
is.
We
want
to
support
the
Prometheus
ecosystem
as
a
whole
right
because
there's
a
whole
lot
of
good
work.
A
That's
going
into
you
know
it's
not
just
the
Prometheus
server
itself,
they're,
all
the
exporters
right
for
exposing
metrics
and
and
all
that
kind
of
stuff
there.
There
are
the
the
client
libraries
that
you
use
to
instrument
your
applications
and
I
think
that's
all
good
work
and
we
would
like
to
be
able
to
have
our
tools,
work,
seamlessly
and
really
well
with
those
tools
as
well.
So
we
already
support
scraping.
A
A
The
other
thing
I'd
like
to
see
move
forward
is
the
efforts
around
open
metrics.
So
basically,
this
is
creating
a
standard
that
people
can
agree
on
for
basically
defining
metrics,
not
sure
how
far
along
that
work
is
or
if
people
can
agree
on
it.
But
I
would
like
to
see
that
actually
get
to
some
level
of
completion
where
people
can
say.
Okay.
This
is
good
enough
for
a
v1
of
this
standard,
and
then
we
can
kind
of
form
around
that.
A
So
I'd
like
to
close
the
talk
with
some
future
work
that
we're
thinking
about
and
I'm
curious
to
get
feedback
from
people
in
the
audience
about
which
of
these
pieces
seem
interesting
or
worth
pursuing
right.
So
one
thing
I
mentioned
is
we
want
to
decouple
the
query
processing
from
the
storage
tier
and
what
will
kind
of
naturally
rise
from
that
is
in
theory.
We
could
build
connectors
to
Prometheus
servers,
so
you
could
actually
use
the
I
fql
query,
processing
tier
to
query
any
number
of
Prometheus
servers
and
influx
servers
and
actually
combine
those
results.
A
And
potentially
you
could
use
a
fql
to
query
Prometheus,
which
you
may
or
may
not
want
to
do.
I
guess
depends
on
how
much
you
love
prom
QL,
but
the
idea
with
this
new
this
new
engine
in
this
new
language
is
the
thing
that
we
want
to
achieve
is
want
to
do
the
same
thing
for
the
query
engine
that
we
did
for
telegraph
so
telegraph
as
a
data
collector.
A
What
we
did
was
we
defined
a
clean
like
there
are
input
plugins,
there
are
output
plugins
and
what
what
we
have
from
a
code
perspective
is
we
wanted
to
design
it
so
that
new
contributors
could
write
input
plugins
without
having
to
understand
the
entirety
of
the
Telegraph
code
base,
and
this
is
the
same
thing
that
I'd
love
to
see
in
the
query.
Engine
itself,
which
is
new
contributors,
can
come
in
and
they
don't
have
to
understand
the
the
entirety
of
the
database
and
all
these
different
complex
people
pieces.
A
They
would
be
able
to
define
input,
plugins
right,
so
input
plugins
could
be
an
influx
server,
it
could
be
s3,
it
could
be
a
sequel
server,
it
could
be
Prometheus,
it
could
be
a
third-party
service.
Anything
you
can
think
of.
There's
sets
of
functions
which
are
essentially
transformations
on
the
data,
and
ideally
people
would
be
able
to
add
new
functions
really
really
easily
and
then
finally,
there
are
output
plugins.
So
this
is
for
background
processing
casts
essentially,
but
the
idea
there
is
an
out
like
obviously
the
initial
output
plugin.
A
We
have
is
pipe
the
data
back
into
influx
right.
So
you
could
use
it
for
monitoring
and
alerting.
You
could
use
it
for
down
sampling
your
data,
but
you
could
just
as
easily
say,
have
a
Kafka
output
plug-in
or
an
s3
output
plug-in.
So
you
do
a
query
and
instead
of
writing
data
back
the
influx
or
returning
results
of
the
user.
In
the
background,
it
can
write
that
result
to
some
other
service.
A
A
What
he's
trying
to
do
with
this
is
make
it
so
that
there's
a
data,
interchange
format,
so
you
don't
have
to
marshal
it
and
unmarshal
it
and
reprocess
it
in
whatever.
So
the
idea
is
whether
you're
working
in
spark
or
pandas
or
any
of
these
other
things.
You
can
pass
data
around
without
having
to
do
all
this
expensive
copying.
A
So
I
think
it
would
be
interesting
from
a
Prometheus
init
influx
perspective
is
adding
Apache
arrow
is
basically
a
data
interchange
format
instead
of
what
it
uses
right
now,
which
is
it's
protobufs
over
HTTP
right
and
so
we've
already
started
on
the
work
on
the
arrow
stuff.
We
actually
there
wasn't
an
official
go
implementation
of
Apache
arrow,
so
we
started
work
on
it
and
Wes
had
noticed
what
we
were
doing
and
we
officially
contributed
it
back
to
the
ASF
is
like
the
official
go
implementation.
A
A
A
A
A
A
That
is
something
that
we
will
be
doing
later
this
year,
but
that's
also
one
of
those
things
where
it's
that's
a
multi-year
effort,
like
that's
decades,
work
so
so
yeah
right
now.
We
just
do
this
simple,
stupid
thing,
but
we
will
be
doing
stuff
like
that,
hopefully
in
the
future,
and
we
are
actively
hiring
for
the
query,
processing
roles.
A
A
So
the
question
is:
if
we're
looking
at
switching
to
influx
is
a
good.
Is
it
a
good
idea
to
wait
for
the
iql
engine
to
mature
before
making
the
switch
again
like
I?
Don't
I,
don't
think
running
in
flux
or
running
prometheus
is
like
an
either/or
proposition.
I.
Think
it's
totally
possible
to
run
both
well,
you
could
literally
have
Prometheus
running
is
essentially
your
data
collector
and
you
can
use
both
and
query
both
right
and
gravano
works
really
really
well
with
both.
A
Obviously
so
I
think
I,
honestly,
like
I,
think
the
lowest
risk
way
to
do
it
is
to
not
like
a
polling.
Replace
there's
always
like
the
riskiest
way
to
deploy
new
software
right,
like
the
best
way,
is
to
run
the
new
thing
in
parallel,
get
used
to
how
it
works,
get
used
to
how
it
operates
and
familiarize
the
team
with
it
and
actually
make
a
decision
from
there.
I
think
the
the
I
fuel
stuff
so
we're
Nathaniel
the
key
developer.
That's
writing!
It
is
actually
writing
the
formal
spec
this
week.
A
So
we
should
have
that
available.
So
the
whole
language
itself
is
going
to
be
fully
Specht
out.
We
should
have
that
available
within
the
next
week
or
two
so
yeah
there's
a
lot
of
work.
That's
happening
there
right
now.
I've
QL
is
not
like
something
that
we're
saying
is
production
ready.
It's
basically
like
this
is
test
code
to
get
feedback
from
the
community
right
now.
Our
recommendation
for
actual
production
workloads
is
to
continue
to
use
influx
QL
the
the
sequel
style
language.