►
From YouTube: Weave Cortex: Horizontally Scalable, Multitenant Prometheus as a Service by Tom Wilkie, Weaveworks
Description
Weave Cortex: Horizontally Scalable, Multitenant Prometheus as a Service - Tom Wilkie, Weaveworks
"In this talk we'll present Weave Cortex - a multitenant, horizontally scalable Prometheus as a service.
Our solution turns a lot of the Prometheus architectural assumptions on its head, by marrying a scale-out PromQL query engine with a storage layer based on DynamoDB and S3. We have disaggregated the Prometheus binary into a microservices-style architecture, with separate services for query, ingest, alerting rules and recording rules. By designing all these services as fungible replicas, this solution can be scaled out with ease and failure of any individual replica can be dealt with gracefully."
About
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Called
weave
scope-
and
this
is
about
virtual-
about
visualizing-
your
micro-services
applications
I
stopped
working
this
about
six
months
ago,
but
during
the
time
we
we
built
this
open-source
project.
It's
got
a
very
pretty
UI.
The
UI
team
in
Berlin
did
an
absolutely
fantastic
job,
but
mostly
this
year
we
build
a
hosted
version
of
this
called
weave
cloud
which
you
can
go
and
use.
If
you
like,
and
you
know,
running
a
service
meant,
we
had
to
monitor
it.
So
you
know
some
of
recommendations
of
my
friends.
A
I
was
told
to
check
out
prometheus
and
we
instrumented
scope
and
the
service
with
Prometheus,
and
we
absolutely
loved
it
and
we're
like
hey
about
about
six
months
ago.
We
write
hey,
you
know
we
should.
We
should
include
prometheus
and
a
Prometheus
service
in
weave
cloud,
because
it's
so
useful,
and
so
we
set
down
the
path
of
how
we're
going
to
build
a
scalable
version
of
prometheus
that
we
can
use
to
provide
a
Prometheus
as
a
service.
So
that's
what
how
it
got
here
in
the
process.
A
We've
moved
everything
on
to
kubernetes,
so
that's
kind
of
you
know
a
bit
more
synergy
with
this
conference.
I
guess
and
we
we
did
a
series
of
four
blog
posts
last
month,
I
think
maybe
the
month
before
about
our
experience
of
using
Prometheus
and
Kuban
at
ease
together.
So
you
know
they're
on
our
blog,
you
can
check
them
out.
They're
kind
of
interesting
I
mean
the
best
one.
Is
this
deploying
one?
This
is
kind
of
the
thing
not
often
talked
about
how
to
actually
like
set
up
and
deploy
a
production.
A
You
know
grade
Prometheus
on
your
Cuba
Nettie's
cluster
and
what
the
different
trade-offs
you
had.
You
have
in
the
monitoring
the
applications
everyone's
written
about
that.
So
there's
not
much
point
in
reading
that
one
to
be
honest
anyway,
I've
plugged
the
blog,
let's
just
dive
straight
into
the
design
of
the
system
and
like
every
good
engineer,
we
sat
down
and
we
collected
some
requirements
and
we
had.
We
had
Fabian
and
Bjorn
over
to
the
office
Fabian
biana,
the
other
Prometheus
core
developers
or
some
of
the
other
ones.
A
We
had
them
over
to
the
Weaver
Works
office
and
we
talked
to
them
about
this
idea
and
oh
really
encouraging
and
they
basically
completely
changed
the
design
in,
like
one
half
an
hour
meeting
which
was
fantastic,
we
started
off,
you
know
we
love
Prometheus,
we
love
the
the
query
language,
it's
really
concise
and
you
know
it's
really
powerful,
so
we're
like
we
definitely
want
to
be
completely
compatible
with
Prometheus
and
to
the
extent
where
you
go.
Look
at
the
code
base.
We
just
reuse
wholesale,
massive
swaths
of
Prometheus
code.
A
We
just
vendor
in
the
Prometheus
project,
into
our
into
our
binary
and
just
just
cool
into
it.
It's
fantastic.
We
also
mean
we're
a
very
small
engineering
team
that
we've
works.
We've
got
about
20
of
us
and
on
the
weave
cloud
project
and
on
this
project.
There's
like
five
so
on
this
project.
It's
actually
just
me
and
two
other
people
and
we've
cloud
and
the
hosted
version
scope,
maybe
two
or
three
more
people.
So
this
had
to
be
super
easy
to
operate
and
we
had
to.
A
We
had
to
build
this
thing
so
that
you
know
it.
It
was
just
it
just
had
to
be
easy:
it
had
to
scale
it
had
to
not
have
any.
You
know
single
points
of
failure
it
had
to
generally,
you
know
be
easy
to
upgrade,
because
we
do
continuous
deployment
and
we're
redeploying
this
system.
You
know
eight
ten
times
a
day.
We
you
know
we're
pretty
ambitious.
A
We
want
thousands
of
users,
I,
think
we've
got
like
ten
right
now,
but
we
want
thousands
and
when
it
makes
it
cheap
to
run
and
because
we
want
to
sell
it,
you
know
we're
a
company,
we
believe
in
open
source.
Everything
we
do
is
open
source,
but
we're
going
to
run
out
of
money
sooner
or
later,
and
you
know
by
then
we'll
hopefully
have
a
bit
of
revenue
to
to
pay
for
these
things.
A
So
we
do
want
to
sell
this
eventually
as
a
service,
and
you
know,
take
all
the
kind
of
operational
load
off
of
the
users
yeah
and
we
wanted
to
reuse
as
much
as
prometheus
as
possible.
Prometheus
is
one
of
the
best
piece
of
software.
I've
ever
worked
with
it's
one
of
the
nicest
code
bases
I've
ever
worked
with,
and
it's
just
really
high
quality,
and
so
we
wanted
to.
You
know
we
wanted
to
reuse
that
so
yeah
don't
worry.
This
is
all
open
source.
A
I
know
people
get
kind
of
a
bit
a
bit
worried
about
this
last
bit
here,
but
I
like
to
be
really
really
honest
about
these
things.
We
are
a
company.
We
are
trying
to
make
money,
so
we
do
want
to
sell
this,
but
it
is
all
open-source
we're
100%
committed
to
this.
We've
done
all
of
the
development
of
this
in
the
open,
so
I
published
her
with
Julis
and
I,
published
a
design
doc
under
the
old
name
on
the
Prometheus
developers.
Mailing
list-
oh
yeah,
so
you
come
well!
A
It's
not
that
interesting,
but
just
got
some
dates
on
it.
Yeah
we
started
the
design
dock
in
June.
We
put
the
first
jobs
up
about
a
month
later
into
our
cluster
and
it
kind
of
worked,
and
then
a
month
after
that,
I
presented
this
at
prom
Con
in
Berlin,
where
it
was
it
was
just
about
working
by
then
and
since
then
we
spent
a
lot
of
time,
making
a
lot
more
robust
and
today
I'm
talking
about
it
cubic
on
so
yeah.
A
If
you
want
to
check
out
the
design
dockets,
it's
kind
of
interesting
doing
the
whole
open
design
thing
where
you
get.
You
know
feedback
from
people
in
the
community
who
basically
know
more
than
I
do
about
this
stuff
was
a
really
interesting
experience
to
me
and
I'd
massively
encourage
it.
The
github
repo
that
we
did
the
development
in
was
open
from
day
0.
You
know
this
wasn't
a
we're
going
to
go
away
right,
thousands
of
lines
of
code
and
magically
open
source
it
and
make
a
big
PR
thing
out
of
it.
A
We
we
started
with
an
empty
repo
and
it
was
always
open
source.
In
fact,
I
think
it
started
as
a
branch
as
a
fork
of
the
Prometheus
repo.
So
it
wasn't
an
empty
repo
anyway.
So
next,
what
does
the
system
look
like?
Well,
we've
broadly
divided
the
problem
up
into
three
different
services.
You
know,
Mike
reservist,
is
are
all
the
rage,
so
we
thought
hey.
Why
not
do
that
and
we've
taken?
We've
got
this
retrieval
agent
that
you'll
run
in
your
cluster.
This
will
scrape
the
jobs.
A
A
This
is
like
the
only
bit
of
closed
source
code
in
the
whole
thing
right
and
it's
it's
mainly
closed
source
because
it's
embarrassing,
but
we'll
fix
that
at
some
point
it
was
nginx.
Now
it's
go.
You
know
it
will
probably
be
something
else
in
a
few
months
time,
but
now
we
really
get
into
the
into
the
Frankenstein
bits
and
we've
split
the
job
up
into
a
distributor,
and
this
is
the
thing
that
basically
acts
as
the
Frankenstein
front-end
is
not
cool
Frankenstein.
A
It's
called
cortex
as
the
cortex
front-end,
and
it
decides
where
to
route
the
Cuesta
to
basically
and
it
deals
with
things
like
replication
and
then
we've
got
the
in
jesters.
These
things
will
collect
samples
batch
them
up
and
when
they've
got
a
big
enough
batch
of
samples
called
a
chunk.
They'll
get
they'll
get
flushed
to
s3,
where
we
store
the
chunks
and
inverted
index
and
DynamoDB.
So
we
can
later
find
the
chunks.
A
So
let's
walk
through
that
one
more
time
just
to
hammer
this
home,
your
samples
start
in
your
jobs,
exposed
via
the
Prometheus,
slash,
metrics
endpoint
and
the
scraper.
The
retriever
will
come
along
and
scrape
them
into
a
little
batch,
not
the
same
as
a
chunk
but
a
little
batch,
and
then
it
will
send
them
over
via
the
front
end
to
one
of
the
distributors.
Now,
at
this
level,
we're
just
using
the
coop
Roxie
to
do
random
load-balancing,
so
we
don't
really
have
control
over
where
they're
going
to
hit.
A
That's
that's
desirable
because
that
means
you
know
these
distributors
are
completely
stateless.
So
if
one
of
the
misbehaves
or
we
want
to
scale
them
up,
it's
it's
super
easy
works
really
well,
but
and
once
they
get
to
the
distributor,
we
must
start
thinking
about
some
of
the
cortex
specific
logic.
So
what
we've
done
is
we've
used
concepts
from
Cassandra
a
distributed
database,
so
we
have
a
hash
ring.
I'll
go
to
the
whiteboard
minute
draw
this
out,
but
basically
each
sample
belongs
to
a
specific
time
series
and
each
time
series
belongs
to
a
specific
and
gesture.
A
So
we
use
this
routing
information
that
we
maintain
to
break
this.
This
batch
of
samples
up
into
you
know
different
samples
that
are
sent
to
different
injustice,
and
then
this
process
repeats
itself
and
eventually
I'm
sorry,
I've
got
an
animation
in,
but
I
think
it
looks
good.
Eventually,
you've
got
enough
enough
samples
in
an
gesture
to
form
a
chunk,
and
the
chunks
will
then
get
written
to
s3
and
indexed
in
DynamoDB.
It's
really
straightforward.
Now,
a
few
things
to
note
queries
also
handled
by
the
distributor,
julius
and
I
went
round
on.
A
The
in
jesters
have
to
have
an
in-memory
inverted
index
as
well,
and
so
that
you
can
find
the
queries
now
gives
you
some
examples
in
a
minute
of
how
that
works.
What
else
should
I
mention
here?
Yeah?
Why?
Why
have
we
got
this
separation
between
retriever
and
distributor
is
because
mainly
you
know,
your
code
is
going
to
hopefully
live
behind
a
firewall,
and
it's
going
to
be
really
hard
for
us
in
in
our
data
center
to
scrape
your
jobs.
A
So
another
thing
to
know
the
retriever
is
actually
just
an
open
source
Prometheus,
and
one
of
the
questions
before
the
talk
I
had
was
has
any
of
the
changes
we've
made
in
the
code
we've
developed
gone
upstream
and
yeah
in
the
in
retriever.
There's
a
generic
write,
API
or
spi
I,
guess
that
you
can
implement
the
cortex
is
one
of
the
implementations
of.
So
basically,
you
can
tell
Prometheus
to
send
your
samples
somewhere
else.
A
So
this
here
is
just
an
open
source
upstream
Prometheus
works
with
1.2
and
1.3,
and
why
are
we
doing
this
interesting
batching
model
here?
This
is
because
s3
doesn't
really
deal
very
well
with
with
16
byte
object.
You
couldn't
write
every
every
sample
test.
Three,
you
just
you.
You
know
you
just
destroy
it.
Similarly,
dynamite
does
deal
very
well
with
small
objects,
but
it's
incredibly
expensive.
A
A
This
would
serve
dynamo
d
I'll
get
into
more
detail
later,
but
DynamoDB
stores
an
inverted
index,
so
every
label
value
pair
has
to
be
written
into
dynamo,
DB
and
so
on
average
we
found
it's
between
8
and
10
users
who
are
using
our
system
have
that
many
label
value
pairs
on
each
metric.
So
if
you
multiply
that
out
for
every
sample
that
comes
in
the
front
end,
we
write
about
point
three
point:
zero,
three
over
right,
two
DynamoDB,
which
makes
this
system
cost-effective.
You
know
if
we
didn't.
A
If
we
didn't
have
this
system,
basically
we
would
be
charging
users
eventually,
one
day,
you
know
thousands
and
thousands
of
dollars
to
run
this
and
yeah
I
know:
let's
go
through
each
system
a
bit
more.
How
do
you
set
this
up
so,
as
I
said,
Retriever
is
a
vanilla,
open-source
Prometheus
go
and
download
it
from
docker
hub
or
prometheus
website.
All
the
service
discovery
scraping
that
you
know
and
love
about
Prometheus.
If
people
don't
know
what
I'm
talking
about
then,
hopefully
someone
else
will
cover
that
later,
but
I
can
go
into
in
detail.
A
If
you
like,
all
you
have
to
do
is
add
this
to
your
configuration
previously,
this
will
command
line
flanks,
but
now
it's
now
it's
in
the
config
and
you
can
put
some
some
credentials
in
there
and
you
put
an
API
endpoint.
The
whole
idea
here
is
that
you
can
also
you
know
if
you,
if
you
don't
wanna
use
cortex,
if
you
don't
use
weave
work,
so
you
want
to
host
it
yourself.
A
You
put
your
own
URL
in
there
or,
if
you
want
it
to
send
it
to
a
Kafka
or
or
something
else,
then
you
build
a
little
proxy
for
doing
that
and
you
put
that
in
there.
You
know
this
is
the
generic
completely
non
cortex
specific
bit
of
the
project
and
there's
some
PRS
down
here,
which
which
did
all
of
this
one
of
the
interesting.
A
One
of
the
big
differences
is
they've
considered
the
scalability
of
the
retrievers,
so
they
have
a
method
for
sharding
their
retrievers
out,
so
they
can
run
tens
of
retrievers.
You
know
hitting
millions
of
targets,
we
didn't.
We
just
didn't
think
of
that.
The
Retriever
is
not
really
a
single,
not
really
a
bottleneck
in
this
system.
It
is
a
single
point
of
failure
in
the
current
set
up,
but
the
idea
would
be
you
just
run
two.
So
that's
not
that
not
actually
we
do.
A
A
This
is
kind
of
the
maybe
the
most
hardcore
distributed
system.
Bits
of
this.
Have
you
guys
have
you
guys,
seen
Cassandra's
virtual
node
implementation,
yeah,
okay,
one
guys
nodding
his
head
great,
it's
based
on
it's
based
on
dynamo,
the
original
dynamo
paper
from
Amazon,
and
it
considers
this
token
ring.
You
consider
this
to
be
like
zero
to
two
to
the
64,
and
then
you
pick
each
each
in
gesture
that
comes
along
will
pick
up
a
bunch
of
random
points
in
this
ring.
A
You
know
just
just
sit
there
pick
hold
onto
random
numbers,
insert
them
into
the
ring
and
say
I
now
own
everything
from
this
random
number
to
the
next
random
number,
so
the
nice
property
of
this
system
is,
it
gives
you
a
completely
even
layout
of
distribution
like
the
more
random
tokens
you're
in
you
inject
the
more
even
the
distribution
between
and
jesters
become,
the
downside.
This
system
is
just
complicated.
You
have
to
do,
and
you
know
it's
Julius
and
I
found
out
in
the
past
few
months,
as
we've
been
implementing
this
particular
feature.
A
If
you
want
to
do
replication,
you
basically
start
with
a
hash
of
the
you
know.
In
our
sense,
it's
a
time
series.
It's
a
fingerprint.
You
start
with
the
hash,
you'd
look
for
the
key
closest
to
the
hash,
and
then
you
count,
you
know,
say
we're
doing.
Replication
of
three
you'd
count
three
steps
around
the
ring,
and
that
would
be
your
replica
set,
but
you've
inserted
multiple
tokens
for
each
and
gesture.
So
you
need
to
count
three
times
around
the
ring,
but
not
you
know
not
double
count
and
not
send.
A
You
know
not
treat
the
same
in
gesture
as
two
replicas
for
the
same
sample.
So,
okay,
so
now
we're
gonna
deegeu
pit
and
do
the
next
three
distinct
intestines.
But
what
happens
if
an
and
gesture
is
leaving?
You
know
because
we've
shut
it
down
because
we're
doing
the
role
and
upgrade
okay.
So
we
now
have
a
state
associated
with
each
and
gesture.
We
count
round
the
ring
and
if
we
encounter
a
leaving
and
gesture,
then
what
we
do
is
we
increase
the
number
of
replicas
we
want.
A
So
we
said
we
run
with
three,
but
with
we
hit
a
leaving
one,
we
ask
for
four
and
then
we
don't
send
the
sample
to
the
leaving
one.
Otherwise
it
will
just
never
get
a
chance
to
fully
flush
out
to
s3
and
then,
similarly,
you
have
a
mirror
of
this.
If
you're
doing
reeds,
if
you're
doing
reeds,
you
need
to
increase
by
one
again
and
then
do
your
read
from
an
extra
one.
Otherwise
you
might
potentially
miss
data.
A
So
just
one
one
last
point
on
the
distributor:
when
you've
got
your
finally
got
your
replica
set
your
three
or
four,
or
maybe
even
five
nodes,
that
formula
replica
set,
you
need
to
do
a
write
to
them
and
it's
really
inefficient
to
write
to
every
single
one
of
them
and
of
course
we
want
to.
We
want
to
tolerate
failure.
So
if
we
try
and
write
to
every
single
one
of
them,
wait
for
them
all
and
one
of
them
fails
kind
of
so
what
we
actually
do
is
we
write.
A
This
is
probably
what
I'm
implementing
so
right.
Now
we
just
write
to
all
of
them.
If
one
of
them
fails,
we
don't
matter
it
doesn't
really
matter.
We
just
make
sure
we've
written
to
at
least
n
over
two
plus
one,
so
at
least
a
quorum
of
them
and
the
whole
idea
of
writing
to
a
quorum
of
them
is
that
you're
guaranteed
that
a
subsequent
read
that
might
hit
a
different
quorum
will
at
least
have
a
single
overlapping
in
gesture.
So
that's
where
the
plus
one
comes
in
see
it's
kind
of
cooled
like
this.
A
In
a
previous
life,
I
started
a
company
called
a
kunu
which
did
the
virtual
node
implementation
in
Cassandra,
and
if
you
go
here,
there's
a
presentation,
funnily
enough
by
me,
like
six
years
ago,
discussing
exactly
the
same
topic
in
in
the
end,
the
idea
of
big
data
and
no
sequel.
So
it's
you
know,
there's
no
new
ideas
in
in
computer
science
right
anyway.
So
distributor
also
handles
queries.
We
in
Cassandra
and
in
other
dhts
this
this
this
ring
will
be
gossiped
around
the
cluster.
Using
the
gossip,
algorithm
and
Ciardi
tease.
We
didn't
bother
doing
that.
A
Let
me
see
so
dynamo
star
replication
we've
talked
about
console
yeah,
so
that's
the
distributor
in
our
system
were
actually
in
running
three
of
these
they're,
not
particularly
resource
intensive.
All
they're
doing
is
figuring
out
where
to
Ford
the
data
to
they
do
run
queries,
and
so
the
more
you
have
the
faster
your
queries,
but
you
know
we're
we're
in
the
early
days.
There's
not
a
huge
number
of
users
on
the
system.
A
Yet
so
three
is
enough:
the
ingest
is,
on
the
other
hand,
we're
running
six
now,
I
think,
because
these
are
the
ones
doing
the
real
heavy
lifting
and
the
chunking
scheme
is
really
kind
of.
The
key
are
sorry
about
the
whiteboard.
I
didn't
think
I
put
anything
in
the
left
anyway.
So
the
in
gesture
started
off
life
as
a
heavily
modified
version
of
Prometheus,
and
then
it
would.
We
would
push
data
to
it
and
I'm
not
going
to
get
into
a
push
versus
pull
debate
here,
but
we
were
pushing
data
to
it.
A
It
would
then
effectively
use
the
memory
series
storage
code
from
within
Prometheus
to
chunk
this
up.
It
used
exactly
the
same.
Chunking
schema
as
Prometheus.
This
was
developed
by
by
Fabian,
oh
no,
sorry
by
Bjorn,
and
it's
just
absolutely
fantastic.
One
of
the
most
interesting
pieces
of
code
I've
ever
had
to
stare
at
for
for
hours
on
end,
and
it's
it's
heavily
inspired.
A
You
want,
gives
a
fantastic
talk
about
how
it's
really
inspired
by
Facebook's
gorilla
paper
is
Facebook,
isn't
it
yeah
I
think
so
so
you
know
go
and
read
the
gorilla
paper
go
and
read
the
source
code.
It's
really
well
commented
and
one
of
the
one
of
the
reasons
we
we
use
the
same
chunk
format
as
Prometheus.
Is
it's
incredibly
efficient,
the
chunk
for
my
own
Prometheus?
So
we
take
these
samples.
A
We
batch
them
up
into
a
one
kilobyte
chunk
and
the
chunk
format
does
all
sorts
of
in-memory
compression
on
them
so
with
the
latest
var
bit
encoding,
there's
three
different
encodings
for
the
chunk,
but
with
the
latest
one
I
think
if
I
remember
correctly,
its
1.3
bytes
per
sample
and
the
sample
is
128
bits
so
is-
is
16
bytes.
So
that's
like
you
know,
12
13,
X
compression
in
memory
with
no
performance
overhead.
Basically,
it's
absolutely
fantastic
and
again
this
makes
the
system
cost-effective
to
run
so
yeah.
A
The
in
gesture
keeps
everything
can
memory
for
an
hour.
We
limit
the
we
limit,
the
chunk
sizes
to
one
kilobyte
or
one
hour,
whatever
we
hit
first-
and
this
is
mainly
in
case-
we
have
an
in
gesture
failure.
It
limits
the
amount
of
replay
and
the
amount
of
recovery.
We
have
to
do.
Oh,
oh
yeah,
the
invert
in
the
inverted
index,
so
this
was
actually
inspired
by
Fabian.
The
the
indexing
scheme
in
the
Prometheus
upstream
is
that
is
very
different
to
the
way
we
do
it.
Basically,
we
store
an
in
memory.
A
One
our
inverted
index
and
inverted
index
is
the
same
thing
that
elastic
search
or
Google
will
use
right.
It's
where
you,
you
basically
take
the
corpus
of
text
and
you
insert
for
every
word
in
that
corpus.
You
insert
an
entry
into
the
index
and
then,
if
you
want
to
do
queries
like
saying,
please
give
me
every
document
that
has
the
word
X&Y
in
it.
Then
you
just
do
a
query
for
extra
query
for
Y
and
a
set
intersection
on
that
on
the
results.
So
we
do
that
we
store
this
in
memory
at
this
stage.
A
A
A
You
know
one
of
the
big
differences
between
cortex
and
Prometheus
is
the
the
concepts
of
durability
and
and
how
long
we
can
retain
data
for
so
we
really
want
to
make
it
as
durable
and
and
have
the
best
retention
as
possible.
So
back
to
the
diagram
talks
about
the
distributors,
the
retriever
and
final
step
is
how
do
we
use
dynamodb
and
s3?
A
So
this
is
maybe
the
trickiest
thing
to
explain
and
I
really
haven't
come
up
with
a
good
way
of
explaining
it.
S3
is
easy.
We
just
saw
the
one
kilobyte
chunks
as
objects
in
s3.
That's
easy,
there's
just
a
bucket,
but
dynamodb
dynamized
v's
a
really
weird
database
who
here's
used,
dynamo,
DB
and
pro
against
it.
Okay,
other!
You
do
count
Julius,
because
you
wrote
some
of
this
right
say
it
looks
a
lot
like
Cassandra.
A
So
Cassandra
has
this
concepts
of
column,
families
and
rows
right
and
the
wrote
the
the
values
inside
the
row
don't
have
to
be
overlapping
between
different
rows.
So
it's
effectively
just
a
set
of
nested
dictionaries
that
are
ordered.
Okay,
dynamodb
thought
that
was
a
little
bit
too
complicated
for
people
to
get
their
head
around
I
think
so
we're
going
to
make
it
basically
a
list
of
items
documents
effectively
and
you
can
pick
two
keys
in
your
document.
A
One
will
be
the
hash
key
and
one
will
be
the
range
key
and
the
hash
key
is
used
for
distributing
data
between
nodes
in
DynamoDB.
You
know
I've
never
seen
the
code
for
an
urban
area
I'm
just
inferring.
This
is
how
it
works
from
their
documentation
and
the
range
key
is
used
for
ordering
items
on
an
individual
node
within
DynamoDB
and
what
this
means
for
user
is.
You
can't
do
range
queries
over
hash
keys
because
that
would
have
to
that
would
have
to
consult
every
single
node
in
your
DynamoDB
cluster.
A
It
would
be
terribly
unavailable
and
really
slow
and
you
can't
do
gets
on
range
on
range
keys,
so
you
basically
have
to
specify
a
specific
hash
key
and
then
a
range
of
you
know
start
and
an
end
for
a
range
key.
So
this
is
great.
Actually
we
we
we
form
our
hash
keys,
they're,
not
actually
strings
concatenated
by
colons,
there's
a
library
for
doing
really
cool,
multi-dimensional
array
encoding
into
into
bytes
and
go.
If
you
look
at
code,
it's
in
there
I
didn't
write
it,
but
it
guarantees
lexical
ordering
of
the
byte
array.
A
So
this
means
we
can
do
things.
We
would
put
the
label
name
the
label
value
in
the
chunk,
ID
the
chunk
ID.
We
just
generate
randomly
into
the
range
key,
and
so
we
can,
we
can
say
if
you
want
to
get
all
chunks
for
a
given
label
and
a
given
value.
We
just
do
a
range
key
from
label:
name:
label
value,
0
to
label,
name
label
value.
A
A
There
have
really
does
heavily
affect
how
the
system
scales,
but
it's
again
a
bit
more.you
satiric
and
we
partition
the
first
thing
in
all
of
our
data
stores
has
always
the
user
ID,
and
this
is
really
the
only
mention
of
multi-tenancy
and
the
whole
talk.
Is
you
stick
user
ID
in
the
primary
key,
so
that
was
easy.
A
I'll
come
back
to
explain
this
a
bit
better
later,
because
it's
kind
of
it's
tricky
this.
The
the
model
here
would
work
really
really
well
on
Cassandra.
We
just
didn't
want
to
run
our
own
Cassandra
cluster
I've
had
too
much
experience
doing
that
in
the
past,
so
yeah,
okay.
So
that's
the
finish.
That's
all!
There's
all
the
stages
in
the
system
there's
a
praying
cat,
because
I'm
going
to
do
a
demo
and
so
far
every
time
I've
tried
to
show
it.
It
hasn't
worked
because
of
the
Wi-Fi
I'm
going
to
say
at
least
right
then.
A
Everyone
can
see
my
screen.
This
is
this
is
actually
we
use
Prometheus,
obviously
to
monitor
cortex.
So
I'll
go
through
this
dashboard
in
a
minute,
but
let's
go
to
cloud,
but
we,
if
it
works,
I'll
show
you
the
Devon
stance,
so
we
don't
break
quad.
This
is
this:
is
our
product
we're
trying
to
sell
the
Wi-Fi
is
awfully
slow,
but
this
shows
you
the
topology
of
your
application,
which
is
kind
of
neat.
So
this
is
the
front-end
components
of
our
application
and
we
can
see
what
cortex
looks
like
on
this.
A
A
A
So
this
is
the
interface.
This
looks
a
lot
like
the
open-source
Prometheus
query
interface,
because
it
is
again
we
just
vendor
this
in
we've
added.
This
is
actually
something
we've
added
in
the
past
few
days.
So
I
don't
really
know
much
about
how
this
a
little
bit
works,
but
it's
just
a
little
banner
that
helps
you
navigate
the
metrics
inside
Prometheus.
So
let's
look
at
CPU
usage,
though
this
one
here
we
let
it
load,
no
two,
two
and
a
half
seconds
to
load
that
that's
pretty
poor.
A
But
this
is
the
standard,
Prometheus
interface
and-
and
you
know,
you've
got
the
grass
as
well.
So
this
just
shows
you
you'll
have
to
trust
me
that
this
isn't
actually
just
hitting
a
rule.
Prometheus
I,
guess
what
I
could
show
you
is
if
I
do,
but
we've
got
works
in
a
different
window
and
go
to
prod.
A
So
this
one,
if
you
look,
is
looking
at
dev
and
this
one
will
be
looking
at
prod
and
I,
put
the
same
expression
in
its
graph
and
executes
and
doesn't
work
fantastic.
Oh
no,
they
go.
So
if
you
look
between
these
two
things,
hopefully
I've
demonstrated
that
this
is
really
multi-talented,
because
it's
given
you
different
data
for
the
same
query
for
different
users,
so
yeah,
so
that,
basically
you
know
it
looks
just
like
it
looks
just
like
Prometheus
and
and
that's
by
design,
because
we
think
prometheus
is
awesome.
A
This
little
widget
up
here
is
really
cool.
David
wrote
this
recently,
but
it
basically
splits
the
metric
names
on
underscore
and
then
builds
a
little
tree
out
of
them,
and
so
you
can
say,
oh
I'm,
really
interested
in
cortex,
metrics
and
I'm,
really
interested
in
the
in
gesture
and
I
want
to
know
stuff
about
chunks
and
I
want
to
know
stuff
about
chunk
age,
and
it's
got
it
in
seconds.
I
want
it's
a
histogram
and
we
don't
help
you
with
that.
A
But
let's
just
do
a
count
and
do
a
rate
of
this
I
make
this
fullscreen
again
there
you
go
that
clue
is
a
bit
quicker.
You
know
normal
query.
Query
latency
on
this
thing
has
been
about.
You
know
in
the
order
of
maybe
500
milliseconds,
we
haven't
done
any
optimization
of
the
query
path.
Right
now.
We've
just
been
focused
on
the
right
path,
but
yeah,
that's
the
system
and
it
works.
Oh
I
was
going
to
show
you
like
how
to
install
it.
A
We've
got
this
cool
installer
that
installs
the
node
exporter
and
the
and
Prometheus
and
pointed
at
the
cluster,
but
there's
on
our
website.
This
is
a
dashboard
that
we
used
to
monitor
it.
So
I
talked
about
how
we've
got
chunks
and
the
ingest
has
build
these
chunks.
So
up
here,
we've
got
a
total
number
of
chunks
in
the
system
and
you
see
basically
because
we
flush
every
hour
and
because
we
released
this
version
at
the
same
time
as
everything
else
every
hour.
A
You
basically
see
this
huge
lump
as
we
start
flushing
and
duplicating
all
these
chunks,
and
you
see
this
down
here-
is
the
length
of
the
queue
for
the
flushing.
You
see
this,
this
go
huge
and
then
we
get
lots
of
errors
from
DynamoDB
and
and
then
we've
got
up
here.
We've
got
the
number
of
chunks
per
series,
so
this
should
basically
be
always
one
because
the
minute
a
chunk
gets
closed
because
it's
either
old
or
full.
A
We
try
and
flush
it,
but
again
every
hour
you
see
that's
that
spikes
up
and
slowly
dribbles
down,
and
then
you
know
Bob's
along
nicely.
We've
got
some
distributions
here
that
I
recently
discovered
a
completely
wrong,
but
basically
tell
you
how
full
the
chunks
are
when
we
flush
them.
So
most
chunks
are
completely
full
when
we
flush
them,
which
is
great,
that's
nice
and
efficient.
A
How
old
they
are
I
was
originally
trying
to
design
the
system,
so
there'd
be
one
hour
old,
but
actually
for
various
reasons,
because
we're
not
using
the
most
efficient
encoding
and
because
we're
pushing
much
more
data
into
the
system
than
we
thought
we
were.
The
chunks
tend
to
be
about
25
minutes
old
and
how
many
entries
in
them
they
tend
to
have
300
entries
in
okay,
the
presentation.
A
That's
the
demo
really
evaluation.
Did
this
work
well.
This
is
I
built
this
proper
graph
earlier
today.
This
shows
distributor
latency.
So
when
you
send
a
chunk
into
the
system,
how
longs
it
take
to
process
and
store
that
chunk
and
the
99th
percentile
is
a
hundred
and
twenty
milliseconds,
so
I'm
actually
pretty
happy
with
that.
The
this
is
the
50th
which
is
60
milliseconds
and
like
the
average
it
no,
that
must
be
the
average.
This
is
the
50th
down
here.
A
The
the
interesting
thing
here
is
this
is
only
ever
doing
pretty
much
in
memory
operations,
so
why
does
it
take
120
milliseconds
to
do
in
memory
operations
right
and
if
you
look
at
the
individual
in
jester's
the
latency
for
a
chunk
append
for
a
batch
of
samples
append?
It
is
actually
you
know,
six
or
seven
milliseconds,
so
most
of
the
time
I
think
is
being
spent
a
because
we're
writing
to
all
three
and
jester's.
A
When
really,
we
only
need
to
kind
of
write
to
return
and
let
the
third
one
kind
of
kind
of
go
along
later,
but
also
like
the
that
calculation
I
was
saying
in
the
ring
when
you
need
to
try
and
work
out,
the
replicas
set
I
think
is
quite
expensive
and
I
think
we
could
probably
improve
that
algorithmically.
So
it
wasn't
so
expensive
queries.
So
this,
oh
yeah,
so
to
give
you
some
context,
this
was
on
a
system
with
about
a
million
series
in
it.
A
This
was
about
a
hundred
thousand
samples,
a
second
being
added
to
the
series.
So
a
single
Prometheus
could
do
this
just
as
well
to
be
brutally
honest,
but
this
was
only
running
on
six
nodes
or
five
nodes,
so
it's
not
a
particularly
large
system
and
yeah
about
a
hundred
milliseconds
hundred
twenty
milliseconds
at
the
99th
percentile
queries
is
a
different
story
right
now.
My
next
job
is
to
optimize
the
hell
out
of
this,
but
the
peak
of
this
is
at
a
hundred
milliseconds.
So
that's
not
too
bad,
I,
guess
and
down
here.
A
This
is
generally
I
think
when
people
just
weren't
using
it
and
so
hundred
milliseconds
for
queriers
is
probably
probably
reasonable.
The
big
problem
actually
is
just
the
queries
generate
quite
a
lot
of
data.
They
have
to
fetch
quite
a
lot
of
data.
They
have
to
process
quite
a
lot
of
data
and
the
response
contains
quite
a
lot
of
data.
So
when
this
goes
through
the
horrible
author
thing,
I
was
the
authenticating
front-end
I
was
referring
to
earlier.
A
A
Why
not
just
run
my
own
Prometheus?
Well,
we
do
right,
that's
how
we
monitor
cortex,
and
you
probably
should
as
well
prometheus
is,
as
I
said,
one
of
the
best
pieces
of
software
out
there
Prometheus
intentionally
has
a
very
different
story
around
high
availability
to
cortex,
so
the
story
in
prometheus
is
run
and
then
you've
got
the
most
reliable
system
you
ever
will
possibly
want,
and
that's
great
that
works
really
well
and
that's
that's.
How
Google
operate?
That's
how
we
operate
and
that's
how
most
people
should
operate.
A
There
are
some
downsides
to
running
your
own
Prometheus.
If
you
do
have
a
failure,
you
might
get
a
few
gaps
in
the
graphs,
although
that
should
be
too
hard
to
fix.
One
of
the
other
things
like
you
might
want
to.
You
know,
share
the
access
to
the
Prometheus.
You
know
we'll
probably
do
some
boarding
that
you
might
want
to
share
in
the
future.
So
you
need
access
control.
You
need
authentication,
you
know,
we've
cloud
does
all
of
that!
That's
kind
of
what
we
hope.
A
You'll
pay
for
Prometheus
you
know,
is
not
trying
to
be
everything
to
everyone.
So
rightfully
isn't
doing
that
and
the
other
nice
thing
about
this
design
that
really
contrasts
it
with
prometheus
is
we
have
virtually
infinite
retention
of
your
data
because
we're
not
actually
storing
it.
S3
+
DynamoDB
are
storing
it
and
that
they
are
incredibly
reliable
systems.
You
know
s3
seems
to
serve
most
internet
sites
on
the
web
now
most
web
sites
on
the
internet
now.
So
this
is
kind
of
maybe
the
biggest
difference
between
Prometheus
and
cortex
is
data.
A
Attention
prometheus
is
designed
to
run
on
a
single
machine
designed
to
be
super
reliable.
It
is
designed
not
to
depend
on
external
services
and
therefore
is
limited
to
how
much
data
it
can
store
on
that
machine,
and,
typically
you
see
people
we
run
with
about
a
three
month:
data
retention,
I'm
hoping
once
I
start
working
on
query
optimization
I
can
start
parallelizing
the
query
execution.
A
You
know
and
start
having
a
pool
of
query
workers
and
and
doing
tricks
like
that
and
I'm,
hoping
that
our
query
performance
will
be
able
to
exceed
that
of
three
theaters
as
well.
But
having
said
that,
for
me,
fifth
query
performance
is
pretty
good.
So
that's
a
tough
target
and
we
want
to
have
a
very
different
story
about
high
availability,
so
not
just
having
really
good
durability
and
retention,
but
should
like,
should
you
know
one
of
the
availability
zone
in
our
Amazon?
They
tend
to
go
down.
A
We
we
want
to
have
a
different
story
and
a
better
story
around.
You
know
not
having
holes
in
your
graphs
during
that
process.
There
is
a
lot
left
to
do.
This
is
an
open
source
project.
It's
there's
only
two
or
three
of
us
working
on
it.
So
when
I
presented
this
slide
in
two
months
ago,
it
none
of
it
was
crossed
out
and
since
then,
we've
we've
done
a
whole
bunch
of
code.
Cleanup
we've
done
a
whole
bunch
of
up
streaming.
A
The
replication
between
in
jesters
went
in
about
a
month
ago,
the
life
cycle
of
in
jesters,
joining
and
leaving
the
cluster
without
the
cluster
going
down
and
exploding
that
went
in
about
a
week
ago
commit
log
and
and
set
for
query
services
stuff
we're
going
to
work
on
next
and
jonno.
The
other
chap
working
on
this
is
currently
implementing
recording
rules
and
alerting
and
to
kind
of
round
everything
off
so
yeah.
A
Oh
yeah,
we
probably
should,
like
figure
out
a
way
to
build
for
it
and
implement
rate
limiting
and
stuff
like
that,
because
right
now,
if
you
sent
me
a
million
samples
a
second,
you
probably
take
those
down
wait.
That's
me,
that's
Tom
any
questions.
If
you
want
to
try
it
out,
I'll
put
the
slides
online,
so
you
can
copy
their
code
of
that.
But
this
is
the
way
of
kind
of
getting
started,
go
and
sign
up.
A
A
So,
actually,
in
the
past
four
days,
the
load
on
the
system
has
gone
up
by
about
10x,
so
I
very
much
have
been
frantically
scrambling
around
behind
our
booth
scaling
the
system
up
to
deal
with
that
so
yeah.
We
wouldn't
like
the
whole
point
of
this
system.
Is
each
customer
doesn't
get
their
own
Prometheus
because
we
want
to
have
you
know
we
hope
to
have
thousands
of
customers
and
we
don't
want
to
manage
thousands
of
Prometheus's.
So
instead
of
you
know,
sharding
by
customer,
where
sharding
I
didn't
talk
about.
A
This
did
I
I'll
completely
missed
it.
We
shard
by
customer,
ID
and
metric
name,
and
we
hashed
that
into
the
ring.
So
the
idea
is
this
should
be
very
horizontally
scalable
at
multi-tenant,
so
you
know
if
a
customer
does
come
along
and
try
and
us,
as
we
should
hopefully
just
scale
the
system
up
and
and
survive.
A
I
mean
the
performant
yeah.
The
challenge
isn't
going
to
be
the
performance
of
the
chunking
scheme
or
of
anything
like
that.
The
problems
is
our
current
bottleneck
is,
is
actually
how
quickly
we
can
flush
stuff,
the
s3
and
and
how
how
we
use
DynamoDB,
because
DynamoDB
is
a
statically
provisioned
system.
A
A
Yes,
each
batch
that
comes
in
gets,
we
run
a
little
algorithm,
which
decides
which
sample
belongs
to
which
in
gesture,
and
then
we
split
it
up
into
multiple
different
batches
and
send
them
on
so
so
there
is
a
write
amplification
going
on
there.
Oh
yes,
so
if
one
ingestion
is
down,
we
actually
only
need
you
know
if
we're
writing
2/3,
which
is
what
we
currently
do.
We
only
need
2
to
act
the
right.
A
A
A
Yes,
so
one
of
the
things
we
want
to
do
because
that's
through
itself
is
replicated
behind
the
scenes.
We
want
to
have
a
background
process
which
is
going
along
coalescing
these
chunks
and
maybe
forming
slightly
bigger
chunks,
to
give
you
better
performance
for
larger
queries
as
well.
We
haven't
got
round
to
that,
and
it's
not
particularly
important
right
now,.
A
Yes,
11930
bility
is
a
lot
better
than
cortex
as
availability.
So
it's
not
a
limiting
factor.
Right
now,
I
mean
our
idea
will
be
eventually
when
we're
properly
selling
this
commercially,
we
will
run
it
in
two
different
locations
and
you
know
we'll
have
a
proper,
hey,
CheY
and
so
on,
but
but
right
now
you
know
it's
still
pretty
reliable.
It's
you
know.
If
you
consider
a
single
Prometheus
node
is
as
reliable
as
the
machine.
It's
on
I
think
we'll.
A
A
Yeah,
yes,
we
did.
The
main
thing
was
we
just
didn't
want
to
run
too
many
big
complicated
distributed
systems.
We
believe
you
know
we're
offering
this
as
a
service,
because
we
we
want
to
use
systems
that
are
offered
as
a
service.
We
want
somebody
else's
the
the
operation.
This
isn't
be
someone
else's
problem
and
we're
happy
to
pay
for
it.
A
On
the
other
hand,
like
the
interfaces
internally
are
designed
such
that
you
could
swap
this
out
for
Cassandra
I
mean
Cassandra
is
really
the
one
that
we've
mainly
kind
of
talked
about,
but
there
would
be
nothing
to
stop
you
using
big
table
or
HDFS
or
maybe
even
react
or
something
like
that.
The
only
interesting
kind
of
side
effect
here
is
the
chunk
size
is
quite
small,
so
one
kilobyte
is
easy,
is
small
even
for
something
like
s3
and
in
it
kind
of
fits.
A
It
kind
of
slips
between
the
the
small
value
databases
like
Sandra
Hoover,
which
are
very
good
at
storing
values
of
a
few
bytes
in
size
and
the
large
value
stores
like
s3,
because
that's
just
an
easy
problem
and
there's
kind
of
a
gap
around
the
one
kilobyte
Park
of
like
what
do
you
do
with
one
kilobytes
they're
a
little
bit
too
big
to
put
in
Cassandra.
You
probably
just
put
me
into
a
little
bit
too
big.
A
You
know,
you'd
have
to
tune
your
compaction
settings
and
they're
a
little
bit
small
for
something
like
cluster
or
HDFS
or
so
on.
So
so
that's
kind
of
an
interesting
problem.
I
think
you
know,
hopefully
the
coalescing
system
that
I
was
referring
to
when
we
get
round
to
writing.
That
will
allow
us
to
build
much
larger
chunks
and
do
some
DJ
ping,
and
that
will
maybe
make
it
more
applicable
for
something
like
HDFS
or
Gloucester,
or
something.
A
We
also
do
that
because
it
makes
it
more
cost
effective
because
we
don't
have
to
pay.
If
we,
if
we
get
a
hit
in
memcache,
the
right
performance
for
s3
is
is
not
as
nice
as
we
like,
and
it
has
some
weird
characteristics.
You
you,
you
can
kind
of
tell
what
it's
doing
them
behind
the
scenes
and
it's
clearly
like
forming
ranges
of
keys
and
splitting
them
off
and
and
giving
them
to
other
machines,
because
you
see
really
weird
step
changes
in
the
latency
of
s3
as
you're
running
the
system.