►
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
TSDB: The Past, Present and the Future - Goutham Veeramachaneni, IIT Hyderabad (Advanced Skill Level)
TSDB is the storage engine that powers Prometheus 2.0 and brought most of the drastic improvements in speed and utilization across the board. But it is also a golang library that Prometheus imports and is a solid time-series database that anyone can use. It is undergoing active development and this talk will take you the journey, benchmarks, new features, ghastly bugs and moonshot ideas. This will also help the audience understand the library and hopefully get them to use and contribute to it.
About Goutham
Goutham is a student and a developer from India. His enthusiasm for Ops got him an internship on the infra team of a large company where he worked on Production infrastructure and built the company's monitoring system on top of Prometheus. That was his first encounter with production systems and Prometheus, and last summer, he started contributing to Prometheus and interned with CoreOS, working on Prometheus' new storage engine. He is now a maintainer for TSDB, the engine behind Prometheus 2.0.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
So
hello,
hi
I'm
here
to
talk
about
TS
DB,
but
what
is
TS
DB
THD
V
is
the
storage
engine
of
Prometheus
can
I
know
how
many
of
you
here
are
running
Prometheus
in
production
and
how
many
of
you
are
running
from
atheist,
2.0,
yep
you're,
all
using
TSD
be
the
internal
storage
engine
of
Prometheus,
so
I
am
Gautham
I'm.
Now
a
developer
at
the
fauna,
labs
I
was
quite
recently
a
student
I
contributed
to
TS
DB,
and
this
talk
is
about
the
past
present
and
the
future.
It's
actually
quite
funny.
A
I
just
submitted
a
talk,
hoping
I'll
figure
it
out
when
I
when
it
gets
accepted
and
it
did
get
accepted
so
I
think
I
put
together
a
fun
talk.
So,
let's
see
so
before
we
talk
about
TS
DB.
Let's
see
why
it
came
like
ye.
We
doubt
it
so
Prometheus
first
started
in
2012
when
a
bunch
of
soundcloud
engineers
felt
it
wasn't
a
good
monitoring
system,
the
first
version
of
the
storage
engine.
For
that
time,
series
database
or
the
monitoring
system
was
actually
completely
level
DB.
The
data
was
in
leveldb.
The
index
was
in
leveldb.
A
It
didn't
quite
scale
well,
so
they
went,
they
actually
went
ahead
and
rewrote
the
data
which
is
which
was
the
bottleneck
and
the
second
version
had
a
level
DB
index
with
the
custom
data
stores,
where
they
were
compressing
time
series
very
well
now
this
was
what
made
Prometheus
1.0
and
how
many
of
you
ran
from
me.
Theists
one
point
text,
some
version
of
one
point
X,
and
how
many
of
you
were
quite
happy
with
it.
Yep
like
we,
it
worked
really
really
well,
it
could
handle
millions
of
time-series.
A
Soundcloud
was
like
having
millions
of
time
series
and
what
Searle
does
1.0
went
through
a
lot
of
improvements
in
terms
of
performance
in
terms
of
issues,
and
there
was
like
more
than
one
enough
year
of
development
into
1.0
into
the
level
DB
index
in
the
custom
storage
engine,
but
then
came
cuban.
It
is
like
actually
wrote
the
stock
and
the
nests
of
abeyance
keynote
yesterday
and
I
was
like,
he
talked
half
the
stuff
I
wanted
to
anyways
with
kubernetes.
There
was
a
lot
more
series
like
there's
a
lot
more
dynamic
environments.
A
There's
a
lot
more
things
happening
and
this
kind
of
wasn't
working
out,
like
even
SoundCloud,
was
facing
issues
so
Fabian,
the
main
author
of
DSD
B.
He
actually
started
experimenting
on
the
side
and
he
came
up
with
tin
decks,
which
sounds
like
tinder,
but
it's
not
yeah.
It's
a
time
indexed
index.
Oh
it's
to
index
time,
series
data
and
actually
that
experiment
went
quite
well,
but
based
on
that
experiment,
he
went
ahead
and
rewrote
the
complete
thing,
not
just
the
index,
and
that
was
TS
DB
and
as
he
was
rewriting
this
a
funny
story.
A
So
I
in
my
internship
deployed
prometheus
and
I
wanted
to
contribute
so
I
started.
Bringing
people
and
Fabian
pointed
me
through
to
index.
Then
I
lost
like
most
contour,
open-source
enthusiasts
I
lost
interest,
but
six
months
later,
I
come
back
to
the
project
and
boom
index
became
TS.
Db
and
I
was
like
now,
I
have
to
contribute
and
that's
where
I
come
in
a
mid-weight
of
TS
DB
development
and
now
I'm,
one
of
the
maintainer
yep
TSD
B
stands
for
time
series
database.
But
what
is
a
time
series
time?
A
Series
is
just
a
set
of
values:
each
marked
by
time
like
weather,
in
Copenhagen,
at
10,
o
clock,
it's
11
degrees
at
12
o'clock,
it's
15
degrees
and
so
on
and
yeah,
like
the
number
of
people
in
this
room.
Probably
the
previous
talk
had
more
people.
This
talk
has
less
people
and
so
on.
Yeah,
and
you
just
don't
want
to
measure
a
single
time
series.
A
Do
you
want
to
measure
everything
like
CPU
memory,
ram
in
case
of
operational
metrics,
the
number
of
requests,
and
you
want
to
measure
it
by
instance,
but
like
it's,
it's
a
bunch
of
time
series
like
weather
in
Copenhagen
and
weather
in
Hyderabad
stock
market
prices,
and
all
this
they're
like
not
just
one
but
millions
of
them.
But
how
do
you
name
them
like?
How
do
you
name
the
number
of
people
in
Halton?
You
just
have
a
metric
name
and
a
bunch
of
key
value
pairs
quantifying
it
like,
for
example,
take
the
stock
price
of
Apple.
A
You
just
say:
stock
price,
ticker
and
exchange,
and
you
can
have
you
can
have
thousands
of
series
of
the
name
stock
price
because
there's
like
thousands
of
tickets
anyways.
Now
that
we
know
how
to
name
all
these
millions
of
time
series
we
to
query
them,
essentially
we
we
have
a
you
just
select
them.
You
just
say
you
specify
the
name
of
this
earth
metric
and
boom.
A
You
just
have
millions
of
samples
marked
by
time
and
millions
of
time,
series
anyways,
so
tht
be
the
new
engine
in
Prometheus.
2.0
is
actually
an
embedded
time
series
database,
so
Prometheus
imports
it.
If
you
right
go
and
if
you
want
to
do
some
number
crunching
that
like
requires
you
to
handle
time
series
data,
you
can
probably
go
ahead
and
import
Prometheus,
TS
DB,
it's
actually
very
simple
interface
like
I,
really
like
the
interface,
it's
just
like
I
mean
I'm
just
going
to
go
through
the
basics,
so
that
you
know
next
time.
A
A
So
we
have
some
sort
of
transactions,
but
we
will
talk
about
them
later,
so
you
create
an
offender
and
you
put
all
the
data
you
want,
like
you
just
add
all
the
data
you
want
to
that
offender
and
then
you
commit
it
boom.
The
database
will
have
your
data
and
if
you
want
to
query
it,
you
give
the
timestamps
the
minimum
time
and
maximum
time
for
what
like
for
the
samples
you
are
interested
in,
and
then
you
give
the
matches.
I
said
you
know
you
have
to
give
city
equals
Berlin
or
name
equals
this.
A
So
you
give
the
matchups
and
you
get
all
the
time
series
that
you
want
and
if
you
you
like
how
many
of
you
write
prompt,
QL,
how
many
like
for
how
many
of
you
is
this
obvious
like
it
should
be
for
everyone
because
essential
that's
what
we
do
and
the
matches
can
be
anything
actually.
So
it's
an
interface.
We
don't
specify
what
kind
of
matches
it
should
be
like
it
could
be
equality,
regulated
situation.
A
It
could
be
some
magic
that
you're
writing
it's
an
interface
and
that
kind
of
allows
you
to
select
all
kinds
of
data
at
once
anyways.
This
is
not
an
internal
stock.
Fabian
gave
an
amazing
talk
on
the
internals
of
the
database
in
prom
con
you
just
googled,
prom
corn
Fabiana,
and
you
will
get
the
talk.
This
is
about
what
is
great
like
and
what
are
the
issues
that
we
faced
and
what
is
the
future
of
THD
be
like
performance?
A
Trust
me.
Everybody
loves
the
graphs
yeah,
so
we
actually
benchmarked
Prometheus
with
the
new
database
and
the
old
one,
and
essentially
one
thing
we
did
is
we
had
a
lot
of
load,
but
also
on
top
of
the
load.
We
used
to
Reese
restart
half
of
the
pods
every
10
minutes,
like
that's
a
lot
of
churn,
it's
as
if
you
are
deploying
half
your
services,
every
10
minutes,
and
that
creates
a
lot
more
time
series
that
adds
a
lot
of
pressure
and
yeah
like
because
I
say
it's
better.
A
A
So
this
is
something
that
is
literal
that
literally
blew
us
away,
because
if
you
can
see,
prometheus
2.0
writes
very
little
but
like
Prometheus
one
point
or
it's
like
80
MB
per
second,
but
they're
scraping
the
same
data.
Why
is
that?
So
we
want
per
crash
resiliency
in
Prometheus.
So
what
what
one
point
or
does
this?
If
periodically
it
just
checkpoints,
all
the
data
it
has
in
memory
to
disk
and
if
it
crashes
it
restores
from
the
checkpoint.
A
Now
that
essentially
means
you're
writing
the
same
data
again
and
again
and
again
to
disk,
and
that
actually
has
a
lot
of
rights.
But
what
we
do
here
is
we
have
a
right
ahead,
lock,
which
we
will
also
see
has
a
lot
of
other
uses.
We
own
the
moment.
A
sample
comes
in,
we
write
it
to
disk
and
that's
the
only
thing
we
write
only
time
we
write
that
sample
to
disk
like
if
we
crash
and
we
restart,
we
just
read
the
whole
write
a
head
lock.
Instead
of
a
checkpoint,
it
worked
yep.
A
The
other
thing
is
the
on
disk
size.
Again,
as
fabian
explained
explained,
on
prometheus
1.0.
The
per
series
overhead
is
a
lot
here,
it's
like
very
little.
So
what
we
were
doing
is
every
10
minutes.
We
were
restarting
the
pods
right,
which
means
a
city
slash
only
for
10
minutes
and
in
10
minutes
it
turns
out
the
amount
of
data
we
store
is
less
than
the
data
required
to
have
a
single
series
in
1.0,
so
that
adds
a
lot
of
on
disk
size.
But
if
you
are
having
longer
series,
it
shouldn't
make
a
difference.
A
Yep
coming
to
query
latency
yeah,
one
of
the
things
you
should
observe
about
the
previous
graphs
is
that
all
of
them
are
consistent.
While
one
point
was
flaky,
and
that
is
really
important
for
queries,
we
want
consistent,
consistent
queries
and
that
actually
turns
out
to
be
very
good.
In
2.0,
anyways
now
comes
snapshots.
How
many
of
you
ran
1.0
in
production
and
wanted
backups,
and
how
many
of
you
were
frustrated
by
the
reply?
Stop
Prometheus
back
up
back
up
the
disk
and
restart
Prometheus
yeah.
That
was
the
like.
A
That
was
the
problem
we
had
in
1.0.
We
couldn't
support
backups
and
again,
with
the
structure
of
th
DB.
We
could
easily
support
ultra-fast
back
backups
like
snapshots,
it's
like
literally
taking
a
picture.
So
what
we
do
is
so
what
we
have
is.
Essentially
we
have
a
lot
of
immutable
blocks.
It's
like
SS
tables.
You
write
the
recent
data
into
memory
and
once
the
data
is
old
enough,
you
flush
it
to
disk.
Now
these
blocks
are
immutable.
Now
we
use
that
property.
What
we
do
is
we
have
a.
A
We
take
the
data
directory
and
all
these
immutable
blocks.
We
just
do
a
sim
card
link,
which
is
a
sis
call,
and
then
the
data
we
just
write
it
to
disk
turns
out.
The
SIS
calls
take
like
milliseconds
and
right
into
disk
of
the
in-memory
data
takes
like
maybe
10
seconds,
on
a
very
loaded
server
like
you
have
ultra-fast
backups,
which
are
online
and
then
the
other
thing
is
how
many
of
you
deleted
data
in
1.0.
A
So
essentially
some
of
the
use
cases
we
had
was
somebody
wrote
some
somebody
wrote
like
had
a
bug
and
then
you
had
like
a
million
time
series
with
a
user
ID
somewhere
and
suddenly
you
you
would
Prometheus
is
not
usable
anymore.
You
fix
that
bug,
but
your
Prometheus
is
still
not
usable
anymore,
because
you
have
too
many
series.
You
want
to
delete
those,
and
now
the
thing
is
4/5
like
for
five
days.
You
have
the
right
data
for
one
hour.
You
have
the
bad
data
and
then
for
five
more
days
you
have
the
right
data.
A
You
just
can't
clean
up.
Only
the
part
you
want,
you
just
have
to
throw
away
the
whole
thing
in
1.0
and
2.0
we
have
granular
deletes.
You
can
just
specify
I
just
want
to
delete
this
one
point
in
this
one
series
and
you
will
be
able
to
do
that
so
yeah.
So
how
do
we
do
that?
We
just
actually
instead
of
modifying
the
immutable
blocks,
we
write,
tombstones
and
tombstones
just
say
hey
for
this
series.
Just
don't
look
at
that.
A
A
Many
of
you
understand
what
is
isolation
in
acid
so
like
this
is
a
very
rare
case
in
Prometheus,
so
we
have
histograms
in
Prometheus
and
essentially,
when
you
scrape
a
histogram
is
like
50
different
series
and
you
have
to
update
all
of
them
at
once,
but
sometimes
that
doesn't
happen.
Let
us
see
why.
So.
This
is
a
time
series.
We
have
like
several
time
series
and
we
have
a.
We
have
a
scrape
that
is,
writing
data
to
all
these
time
series
and
so
far
the
goo
routine.
A
That
is
writing
the
data
has
written
to
these
time
series
you
can
have
a
reader.
You
can
have
a
courier
that
goes
from
below
and
it
it
just
slowly
reads
back
and
suddenly
you
see
a
partial
rate
boom
like
half
the
buckets
in
the
histogram
are
updated,
but
half
of
them
are
not
updated,
sometimes
breaking
your
alerts.
Now
this
is
a
hard
problem
to
solve,
but
because
we
have
a
lot
of
like
we
kind
of
rewrote
it
from
scratch,
it
kind
of
became
much
easier
to
solve
it
and
yeah.
A
The
PR
is
open
and
confidence
driven
development,
so
essentially
Brian
Brazil.
He
essentially
went
ahead
and
wrote
the
whole
thing
in
July,
2016,
2017,
yeah,
July,
16
2017,
and
it's
been
it's
just
there
and
yeah
I
just
wanted
to
talk
about
it.
So
I
went
ahead
and
rebased
the
whole
thing,
and
hopefully
soon
we'll
have
isolate
isolation.
A
Now
some
of
the
gotchas
of
be
building
a
database.
How
many
of
you
here
have
built
a
database
or
a
high-performance
service
yeah?
He
probably
knows
the
issue
so
Alex.
What
are
a
locks
so
essentially,
whenever
you
create
a
new
struct
or
whenever
you
create
anything
like
a
new
variable,
you
allocate
memory
to
it.
Yeah,
true,
you
might
throw
away
the
struct
or
the
variable
later
and
garbage
collection
kicks
in
so
you're
not
to
worry.
But
what
is
this
thing
that
is
happening?
A
So
these
are
all
compactions
that
are
happening,
so
we
are
essentially
merging
two
different
blocks
into
a
bigger
block
and
suddenly
there's
a
shoot
up
of
memory
that
that
is
being
used
and
that
shouldn't
happen.
Now.
Why
is
that
happening?
We
went
in
and
we
saw
turns
out
structs
taking
memory
like
several
bites
and
we
have
millions
of
chunks.
Essentially,
we
have
millions
of
structs
and
we
are
compacting.
A
We
allocate
all
these
drops
and
suddenly
for
a
few
seconds
we
allocate
gigabits
of
data
and
suddenly
yeah
GC
will
kick
in,
but
it
if
it
kicks
in
too
late,
the
oom
killer
will
kill
us,
kill
the
service
and
also
allocating
memory
is
very
CPU
intensive
and
it
just
slows
this
whole
thing
down
and
yeah
and
turns
out.
We
were
doing
this
all
over
the
place,
so
we
wrote
the
database,
it
works,
so
we
went
back
and
we
profiled
it
like.
A
A
Okay,
yeah
something
like
that.
So
yes,
essentially
you
have
a
struct
you
which
you
allocated
you
store
it
somewhere
like
you,
you
don't
throw
it
away,
you
store
it
somewhere
and
if
you
want
a
new
struct
in
use
the
previous
one
in
stop
allocating
it
again
and
again,
I
think
it
is
similar
to
that,
but
I
am
Not
sure
yeah,
essentially
what
we
had
was.
We
have
something
called
chunk
meta.
We
store
data
in
chunks
and
we
have
millions
of
when
you
have
millions
of
CDs.
A
You'll
have
millions
of
chunks
and
the
chunks
don't
store
much.
They
just
store
the
reference,
which
is
a
int
the
mean
time
and
max
time.
It
is
an
n64
again
64
bytes
and
a
pointer
to
a
chunk
again,
not
much,
but
when
you
add
it
up
into
millions
boom.
Suddenly
you
have
a
big
GC
spike
yeah,
so
you
use
something
called
sink
pool
in
golang.
So
essentially
you
have
a
pool
of
structs
like
yeah
sink
pool
is
very
nice.
A
You
just
have
a
pool
of
structs
and
whenever
you
want
a
new
one,
you
try
to
pick
it
from
the
pool.
If
you,
if
it's
not
there,
you
create
it,
but
once
you
want
to
throw
it
away,
you
don't
let
the
GC
eat
it.
You
give
it
back
to
the
pool
and
you
kind
of
reuse,
the
same
thing
and
we
were
creating
a
lot
of
these
chunks
very
fast,
but
we're
also
throwing
them
away
very
fast,
but
by
using
sink
dot
pool
by
using
by
pooling
we
saved
a
lot
of
this
Alex.
Yes.
A
So
this
is
some
of
the
example,
a
small
example
of
the
simple
and
golang.
If
you
want
to
use
bytes,
for
example,
this
is
what
you
do
so,
essentially
you
create
a
pool,
and
then
you
get
a
bite
from
the
pool.
If
it's
not
there,
it
creates
a
it,
creates
that
bite
and
once
you
are
done
with
that
bite,
you
put
it
back
yeah,
it
turns
out
yeah
we
had
to
fix
airlocks
at
a
bunch
of
places.
Yeah,
you
can
see
all
this
Alex
a
nice
PR
name
and
yes,
we
are
still
not
done
yep.
A
We
are
trying
to
reuse
memory
wherever
possible,
and
this
is
again
an
open
PR
that
is
there
yep.
This
is
a
funny
one.
This
is
an
interesting
one.
It's
called
Yolo
string,
so
what
happened
is
now
you
have
this
persisted
blocks
you're,
trying
to
read
the
data
from
the
blocks
now
a
date
data
has
a
lot
of
these
strings,
the
metric
names.
We
don't
want
to
allocate
those
strings
like
we
thought.
Okay,
too
many
are
locks.
We
don't
want
to
allocate
these
strings,
so
we
did
something
called
Yolo
string.
A
We
used
golang,
unsafe
and
so
how
many
of
you
can
understand
the
function
that
is
here,
yeah
I'm,
sorry
for
you
yeah!
So
essentially
it
says:
hey
take
this
byte
slice
is
a
string.
Please
assume
it's
a
string
and
yeah
it's
great.
It
is
a
string
for
us
great,
but
only
as
long
as
as
long
as
the
bytes
slice
is
there.
What
if
the
bytes
flash
goes
away?
What,
if
GC
the
garbage
collector
takes
away
the
bytes
list
boom?
A
If
you
try
to
use
the
string
you're
dead,
so
that
is
what
happened
when
we
tried
to
use
your
low
string
yeah.
So
what
we
do
is
we
a
map,
the
data,
so
we
actually
use
a
map
for
these
persistent
blocks
and
later
30
days
later
or
whenever
you
hit
your
attention,
we
delete
the
block,
but
then
you
are
deleting
the
block
and
you
have
a
courier
that
has
the
string
and
yeah.
If
you
delete
the
file,
we
lose
the
bytes
and
the
moment
we
lose
the
bytes.
A
We
axe
try
to
access
the
string
and
it
used
to
crash
we're
like
okay.
This
error
should
not
be
there
and
go
like.
Why
is
this
even
happening
in
gogo
is
supposed
to
make
sure
that
you
never
hit
this,
but
we
hit
this
because
we
were
stupid
enough
to
use
unsafe,
yeah
yeah,
and
then
there
was
this
PR
allocate
and
cash
strings
for
persistent
blocks
turns
out.
We
did
something
called
premature.
Optimizations
we
try
to.
We
thought
allocation
of
shrinks
is
too
much
but
turns
out.
A
It
is
not
yep
files
and
windows,
so
we
are
all
Mac
or
Linux
developers
and
we
go
to
2.0.
We
had
a
bunch
of
our
CSV
toss.
Everything
was
working,
we
finally
launched
2.0
and
somebody
comes
and
says
it
doesn't
work
on
Windows
and
you're
like
ok.
We
have
no
clue
why
it
doesn't
work
on
Windows,
but
again
it
being
an
open
source
project.
A
bunch
of
people
went
head,
debugged
it
and
turns
out
Windows.
A
Doesn't
let
you
delete
a
file
before
you
close
it
and
we
were
doing
it
in
a
bunch
of
places
we
like
yeah.
We
were
deleting
a
file
before
we
closed
it
and
we
are
also
having
a
lot
of
these
file.
Descriptor
leaks
and
stuff,
but
yeah
windows
development
is
a
pain
as
I've
learned
when
I
set
up
my
own
Windows
system
to
debug.
This
I
was
like
I
gave
up
halfway,
I,
don't
care
about
Windows,
yep
yep,
now
I'm
done
with
the
gorches
coming
to
the
future.
A
What
is
the
left
to
do
in
TS,
DB,
like
th
DV,
is
being
used
in
2.0.
We've
done
everything
we
wanted
to
do
like
we
fixed
isolation,
which
is
one
of
the
rarest
bugs
ever
and
stuff
like
that,
but
I.
Think
I'm,
like
I'm,
really
excited
about.
What's
coming
for
the
next
six
months
and
TSD,
be
one
of
them
is
backfilling.
How
many
of
you
wanted
to
backfill
into
prometheus?
Oh
that's
a
lot
of
people
and
what
was
the
answer
yeah.
A
We
said
nope
sometime
in
the
future,
but
not
now,
and
it
will
be
soon
and
again.
The
reason
we
didn't
want
you
to
let
you
backfill
your
data
is
because
each
series
was
a
file
and
if
you're
back
filling
into
millions
of
series,
you're
touching
millions
of
files,
it
just
doesn't
scale
in
1.0,
but
in
2.0
we
can
do
something
very
clever.
So
we
say
these
are
the
immutable
blocks.
Like
we
say,
the
first
block
has
data
from
t0
to
t1
t1
to
t2.
T2
has
the
second
block.
We
just
do
something
very
clever.
A
We
just
say:
hey
you,
take
your
backfill
data
and
plop
a
block
in
and
then
we
reuse.
So
a
block
is
an
interface.
It's
not
an
actual
thing,
so
you
can
have
anything
as
a
block.
So
what
we
do
is
we
have
a
big
block
that
encapsulate
all
this
and
merges
all
the
data.
So
this
is
something
that
I
want
to
work
on.
Next,
it
has
a
little
got
just.
How
do
you
d
do
placate
and
stuff,
but
I
would
love
to
see.
A
Someone
else
do
this
because
come
on
community
yeah
I
wish
too
many
things
to
do
and
the
other
things
right
ahead.
Log
base
replication,
so
we
have
a
right
ahead
log
and
it's
one
of
the
one
of
the
things
that
gave
us
a
lot
of
benefit.
It's
like
the
startup
times
are
instantaneous
like
it's
very
fast
to
restart
a
Prometheus
in
1.0.
You
had
to
go
through
the
checkpoint
and
a
lot
of
other
things
and
the
other
thing
that
the
right
ahead
lock
will.
Let
us
do
is
replication,
but
why
do
you?
A
Why
do
we
want
replication?
So
how
many
of
you
heard
of
remote
storage
in
Prometheus
I
mean,
if
you
use
remote
storage
in
Prometheus,
so
yeah
so
I'll
tell
you
how
remote
storage
in
Prometheus
works.
So
you
get
a
sample
Prometheus
scripts,
a
sample.
You
sent
the
sample
to
remote
storage.
Now
a
sample
is
six
eight
bytes
and
eight
bytes,
but
then
with
that,
you
send
also
all
the
strings
all
the
labels.
Everything
with
that.
A
So
essentially,
you
might
send
a
lot
more
data
than
you're
supposed
to
yeah,
but
with
the
right
ahead
log
we
just
replicate
the
right
ahead.
Lock,
we'll
make
it
an
interface
we'll
make
a
library
around
it
so
that
anybody
can
integrate
with
the
right
ahead
log
and
it's
much
more
efficient
that
way.
Yep
there's
a
proposal
open
on
Prometheus
dev
mailing
list,
just
Google
from
easiest
dev
right
ahead,
lock,
replication
and
it
would
be
if
you're
interested
in
it
I
would
love
your
thoughts
on
that
yep.
A
The
future
is
bright,
I
don't
have
the
goggles,
but
yeah
there's
a
lot
more
things
happening.
It's
a
library.
We
have
a
lot
of
users
using
using
th
DB
like
people
who
are
trying
to
do
their
own
analytic
stuff.
People
are
building
databases
on
top,
and
that
is
one
of
the
reasons
we
rebuild
th,
DB
and
I
want
you
guys
to
look
into
th
DB
C,
if
it's
applicable
for
your
use
case.
A
If
your,
if
you're
touching
time,
series
data,
you
probably
want
to
look
at
this
a
little
gotcha,
this
is
in
process
and
it's
a
single
node
one.
It's
not
a
replicated
one.
If
you
want
to
know
if
you're,
okay,
with
an
out
of
process
with
a
little
latency
influx,
is
much
better
or
maybe
just
use
Prometheus
and
one
more
thing.
One
of
the
use
cases
of
this
being
a
library
is
tano's.
A
How
many
of
you
saw
Avengers
here?
Thanos
is
the
good
guy
yeah.
So
Thanos
is
a
long-term
storage
for
prometheus,
so
what
it
actually.
So
what
every
other
remote
storage
solution
now
does?
Is
it
reads
from
this
remote
remote
storage
data
that
Prometheus
sends
to
that
thing,
but
because
th
TV
is
a
library,
anybody
can
write
us
a
thing
that
manipulates
the
data.
So
what
Tanis
does
is
it
has
Prometheus,
it
has
a
sidecar
and
it
has
these
blocks
immutable
blocks.
A
A
So
essentially
you
don't
need
to
care
about
replication
XYZ,
because
you
have
your
s3
or
GCS
caring
about
all
that,
and
one
of
the
wins
of
th
DB
is
the
store
which
is
essentially
written
because
th
DB
was
a
library.
If
it
was
like
a
custom
format
in
built
into
promises,
it
would
have
been
very
hard
to
implement
this
yep.
A
Also,
a
gentle
reminder.
We
have
Brom
Khan
coming
up
itself
is
actually
the
one
conference
I'm
looking
forward
to,
because
it's
just
so
small,
it's
like
220
people,
very
cozy,
very
dev,
very
like
and
it's
in
Munich.
It's
very
nice.
If
you
are
users
of
Prometheus,
if
you
have
use
cases
for
th
DB,
if
you
are
already
using
PSD,
be
thinking
of
using
th
DB
come
talk
to
me,
submit
talks.
We
have
the
CFP
open
for
another
two
weeks
or
so
it's
in
August,
it's
sunny
and
it's
in
Munich.
It's
there's
good
beer
yep.
A
A
Yes,
so
so
now
we
have
th
DB
and
it's
library.
So
we
have
this
block
size.
Options
turns
out,
there
are
advanced
user
options
and
we
don't
want
people
to
meddle
with
them
because
that
breaks
setups
and
that
actually
broke
setups.
Those
options
are
there
because
we
wanted
to
do
some
testing
locally.
Essentially,
we
scrape
every
point
every
point,
two
milliseconds
and
we
have
like
a
10-minute
block
size
because
we
want
to
hit
compaction.
A
A
A
So
that's
one
of
the
parts
of
the
few,
the
controversial
part
of
the
future
that
I
want
to
see,
because
so
essentially
I
want
to
have
the
time
series
to
be
an
interface
like
right.
Now,
the
time
series
values
are
only
floats
but
in
the
future
I
want
to
put
strings
in
there,
maybe
logs
and
stuff,
but
it's
all
like
well.
First
thing
is:
if
you
put
it,
if
you
make
it
an
interface,
you
kind
of
have
a
big
performance
finality
and
we
don't
want
that.
A
The
first
thing
is
th:
DB
was
built
for
Prometheus
oops,
oops,
yeah,
ts,
DB
was
built
for
Prometheus,
and
it
should
work
for
Prometheus
primarily
and
if
that
I'm
still
experimenting.
Like
trust
me
I'm
going
to
do
this
at
some
point,
but
if
it
turns
out
that
you
can
have
interfaces
for
values
rather
than
just
floats,
I
would
have
to
I
would
do
it
in
the
future,
but
sometime
in
the
future,
I
can't
guarantee
it
you're
welcome
to
talk
to
me.
If
you
want
to
experiment
with
that,
it
is
definitely
possible.
A
A
Yes,
only
if
the
blocks
are
an
s3
and
it
takes
some
time
yeah,
so
one
of
the
things
is
we
reduced
Alex,
we
reduced
memory,
usage
and
everything
before
the
storage
used
to
show
up
in
the
profiles.
Now
the
query
engine
is
showing
up
in
the
profiles
and
the
first.
The
thing
that
is
that's
another
open
PR
is
optimizing.
The
query
engine
now
going
back
to
Thanos
Thanos
puts
the
blocks
in
s3
and
the
block
only
goes
thereafter
to
ours.