►
From YouTube: Blazin’ Fast PromQL - Tom Wilkie, Grafana Labs
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Blazin’ Fast PromQL - Tom Wilkie, Grafana Labs
PromQL, the Prometheus Query Language, is a concise, powerful and increasingly popular language for querying time series data. But PromQL queries can take a long time when they have to consider >100k series and months of data. Even with Prometheus’ compression, a 90 day query over 200k series can touch ~100GB of data. In this talk we will present a series of techniques employed by Cortex (a CNCF project for clustered Prometheus) for accelerating PromQL queries -- namely query results caching, time slice parallelisation, aggregation sharding and automatic recoding rule substitutions. But there’s more: we will show how you can use this technology to get these improvements with Thanos and Prometheus. Finally, we will cover optimisations to the PromQL engine by the Cortex team, and how these have already been merged upstream to benefit the whole community.
https://sched.co/UaWT
A
Good,
oh
hello,
everybody.
My
name
is
Tom
sorry
I'm
starting
a
bit
late.
I
was
having
a
good
chat
at
the
front
right
then.
So
today,
we're
going
to
talk
about
blazing-fast,
prom
ql.
This
is
a
project
that
I've
been
working
on
now
for
about
a
year
or
so
and
yeah
hopefully
take.
How
long
do
we
have
do?
You
know
I've
now,
35,
okay,
good
with
questions
nice.
Thank
you.
So
yeah
take
off
now
to
explain
what
we've
done
here
so
a
bit
of
an
introduction.
A
As
I
said,
my
name's
Tom
I'm
really
honored
to
be
basically
giving
one
of
the
first
track
talks.
A
cubic
on
this
means
that
all
of
my
anchorman
memes
are
going
to
be
nice
and
fresh
I.
Think
in
three
days
time,
you're
gonna
be
really
sick
of
anchorman
quotes,
yeah
I
work
for
a
company
called
Cortana
labs,
who
here
uses
kevanna,
Wow,
excellent,
yeah,
I,
I'm,
technically
the
VP
product,
but
I'm.
Actually
just
a
software
engineer
by
trade
at
least
I.
Don't
do
many
PRS
anymore
I
started
the
cortex
project.
A
Cortex
is
one
of
these
CNCs
projects
to
do
a
horizontally.
Scalable
from
easiest
I
also
started
something
called
Loki
about.
A
year
ago
we
announced
a
cube
comp
and
I'm.
Yeah
I
was
invited
to
be
a
prometheus
maintainer
about
two
and
a
half
three
years
ago
as
well,
which
is
pretty
cool.
If
you
want
me
on
Twitter
on
github,
there
you
go,
and
that
is
not
my
photo.
So
this
talk
is
not
about
cortex,
but
I
just
want
to
give
a
brief
intro,
because
it
will
be
relevant
to
the
rest
of
the
discussion.
A
So
cortex
is
this
CN
CF
project.
We
started
it
three
years
ago
and
it's
to
build
a
horizontally
scalable
kind
of
clustered
Prometheus.
So
we
do
a
high
availability.
We
do
replication.
We
do
all
of
these
kind
of
things
and
really
cortex
is
about
about
giving
you
like
solving
some
like
key
problems
with
previous.
So
if
you
have
multiple
independent
kubernetes
clusters
and
you
run
independent
Prometheus
in
them,
you
want
to
have
like
a
global
view
of
all
your
metrics
like
that's
when
you
might
consider
using
I
think
the
audio
is
cutting
out.
A
Okay,
yeah,
okay,
we're
Karam
yeah,
you
could
also
use
other
projects
like
Thanos
exists.
Souls
very
similar
problems,
there's
an
excellent
talk
by
Bartok
in
the
audience
here
and
my
set
off
from
pom
con
two
weeks
ago,
where
we
kind
of
compare
the
two
systems,
but
they're
yeah
they're,
set
out
to
solve
the
same
problem
we
also
like
in
Prometheus.
If
ever
you
have,
if
you
guys
use
Prometheus
who
use
it
for
me,
theater,
hopefully
with
everyone
right.
Otherwise,
why
are
you
in
this
talk?
A
But
if
you've,
if
you've
experienced
like
sometimes
you
know,
you
might
need
to
upgrade
your
Prometheus
server,
you're
gonna
end
up
with
gaps
in
your
graphs.
From
that
restart,
you
know,
maybe
you
need
to
upgrade
the
hostess
running
on
a
sonnet
like
that.
So
this
is
another
thing
we
try
and
solve
with
cortex.
We
can
do
like
hae-joo
ping
again
look
at
this
prom
con
talk,
but
the
reason
I
want
to
talk
about
cortex
and
I
want
to
put
this
talk
in
in
context.
A
Get
those
two
words
correct
is
that
you
know
in
a
cortex
cluster
or
sorry
in
a
Prometheus
cluster.
We
have
this
kind
of
rule.
You
should
never
really
query
more
than
about
a
hundred
thousand
time
series
in
a
single
query-
and
this
is
you
might
hear,
people
talk
about
cardinality
and
this
is
kind
of
what
they
mean.
Like
don't
query
too
many
time
series.
A
Naturally,
when
you
start
using
when
you
start
aggregating
together,
multiple
Prometheus
worth
of
metrics,
and
so
it's
not
enough
for
us
just
us
to
say:
don't
do
that
like
we
have
to
actually
come
up
with
solutions
for
people
who
do
that
because
they
do
it
all
the
time
and
that's
really
where
this
talk
comes
from
like
when
running
cortex.
You
end
up
querying
millions
of
metrics,
normally
and-
and
you
have
to
therefore
do
something
about
that
so
yeah.
A
The
rest
of
the
talk
isn't
really
about
cortex
I'm,
sorry
about
that
and
if
you're
here
to
hear
about
cortex
I
understand
the
next
talk
in
room.
15
is
about
cortex,
but
yeah
I'm
gonna
move
on
now.
So
what
we
do
to
execute
these
queries?
We
actually
just
use
the
upstream
Prometheus
query:
engine
the
prom
ql
query
engine
we
actually
in
cortex
use
the
same
storage
format,
and
this
is
like
I
think
if
you're
familiar
with
Thanos,
if
you're
familiar
with
other
Prometheus
compatible
systems,
this
is
really
common
like
they.
A
It's
really
easy
to
vendor
and
really
easy
to
reuse.
This
Prometheus
query
engine,
so
we
just
do
that.
But
one
thing
you
should
know
about
the
Prometheus
query
engine.
Is
it
it's
effectively,
single-threaded,
so
for
a
single
request?
It's
going
to
execute
on
a
single
core.
It's
gonna
a
great
together
all
that
data
decode.
It
aggregate
it
do
whatever
maths
you're
doing
all
on
a
single
core
and
for
those
of
you
have
been
following
along
cause,
are
actually
not
getting
much
faster.
A
In
fact
like
we're
getting
into
a
world
of
multi-core,
you
know
already
in
the
world
of
multi-core
really,
and
so
in
fact,
cores
are
getting
a
bit
slower,
and
so,
if
you've
got
a
query
engine,
the
only
ones
on
a
single
core
and
you
want
to
make
it
faster.
You've
really
got
to
look
at
ways
of
paralyzing
those
queries
and
that's
what
the
rest
of
this
talk
is
really
about.
A
The
other
thing
we
did
with
the
the
query
engine
inside
cortex
is:
we
made
it
kind
of
stateless
and
we
made
it
horizontally
scalable.
So
if
your
queries
are
slow,
just
run
more
of
them
is
the
idea
accessory
doesn't
really
work
right.
So,
if
you
send
us
lots
of
very
small
queries,
then
we
compare
allies
them
between
different
query
engines.
A
You
know
we
can
scale
up
if
you
have
a
high
concurrent
query
load,
but
everyone
sitting
here
using
graph
Arnall
right,
you,
you
load
up
and
you
maybe
show
five
or
10
graphs
on
your
dashboard
and
if
you're
really
aggressive,
you
might,
you
know
refresh
them
every
few
seconds,
but
that's
still
only
gonna
be
like
single
digits,
maybe
not
even
single
digits,
QPS
and
so
parallelizing
them
is.
You
know,
running
multiple
queries
and
then
load
balancing
across
those
queries
isn't
really
going
to
make
it
much
faster.
A
We've
actually
got
to
find
a
way
of
paralyzing
individual
queries.
You
can't
really
just
keep
scaling
up
the
query
and
putting
on
bigger
bigger
boxes
with
faster
processors.
Does
it
really
exist
anymore?
Yes,
so
this
talk
is
really
about
how
we
handle
big
queries
like
if
you
come
along
and
you
ask
you
ask
to
do
a
query
over
a
few
weeks
worth
of
data
over
a
few
million
time
series.
How
are
we
going
to
handle
that?
A
But
before
we
get
into
that
first
thing
we
did
like
every
good
software
engineer.
We
just
went
and
put
as
much
caching
as
possible
into
the
system.
So
we
went
in
we
we
have.
We
have
a
separate
indexing,
cortex
and
a
separate
chunk
store
of
storing
the
compressed
time
series
data
and
we
just
went
and
stuck
em
caches
in
front
of
them.
So
the
chunks
was
easy
because
chunks
are
immutable.
A
A
The
index
is
a
lot
harder
to
cache,
because
the
index
is
always
changing
and
you're
always
adding
new
stuff
to
the
index,
and
if
you,
if
you
get
the
cache
wrong-
and
we
did
have
a
bug
that
took
us
a
long
time
to
figure
out
where
we
turned
out,
we've
got
the
caching
wrong.
You
end
up
seeing
gaps
or
missing
data
in
your
queries,
so
you've
got
to
be
really
careful
with
with
caching
kind
of
accesses
to
indexes
like
this,
but
anyway
it
works
now
and
this
this
this
made
a
big
difference.
A
This
like
now
80
90
%
of
queries,
will
just
hit
the
index.
We
still
have
to
process
them
on
a
single
thread,
but
at
least
we're
not
putting
a
load
on
the
database
and
we're
not
putting
load
on
the
chunk
store
and
we're
effectively
just
saving
that
capacity
for
when
someone
does
come
along
and
run
one
of
these
really
big
queries
anyway.
Now
we
actually
get
to
the
kind
of
exciting
bit
who
here
has
heard
of
a
project
called
trickster
by
Comcast?
Oh,
not
so
many
okay,
well
tricks!
A
There's
this
really
cool
thing
that
that
caches
the
results
of
the
query
after
you've
executed
the
query.
They
basically
take
the
result.
They
stick
it
in
a
cache
and
then,
if
you
ask
the
same
query
twice
they
return
that
result.
So
this
is
an
awesome
idea
right.
This
relieves
even
more
pressure
from
the
database
and
from
the
query
execution,
and
we
wanted
to
do
this
now.
A
Unfortunately,
when
we
did
this
over
a
year
ago
now
the
trickster
code
wasn't
a
reusable
and
there
was
a
few
things
we
didn't
particularly
like
about
it,
and
you
know:
I
didn't
invent
it.
So
I
kind
of
wanted
to
write
my
own
I
wanted
to
do
it
in
a
different
style.
I
wanted
to
make
it
more
reusable
and
I
wanted
to
do
some
more
interesting
things
with
it.
So
I
built
my
own
thing.
We
called
it
the
query
front-end
and
if
the
query
front
ends,
job
was
to
cache
query
results.
A
We
also
use
it
as
an
opportunity
to
do
a
few,
more
cool
things
and
I'm
going
to
just
like
deep
dive
into
the
query
front-end
for
five
minutes
and
talk
through
what
it
does
to
process
a
request.
So
imagine
you
have
a
request.
Http
durations
seconds
count
right,
so
we're
rating.
It
will
probably
summit
this.
What
this
is
basically
telling
me
the
QPS
of
this
shipping
system.
A
The
first
thing
we
can
do
we're
going
to
take
the
query
into
the
crew
front
end
and
we're
going
to
align
the
steps
right.
What
this
means
is
in
in
the
prometheus
world.
You
ask
a
query,
say
you
ask
it
from
the
Prometheus
UI,
it's
just
going
to
say
from
now
from
right
now
this
second,
this
millisecond
back
to
let's
say
an
hour
ago,
every
15
seconds
give
me
the
value
of
this
query.
A
That's
how
we
plot
the
graph,
and
then
you
run
it
again
and
it'll
say
from
right
now,
every
15
seconds
give
me
the
value
of
this
graph,
and
so
those
those
of
you
who
have
kind
of
seen
this
you'll
see
like
if
your
graphs
are
a
refreshing,
constantly
they're
kind
of
they're
jumping
around
a
little
bit.
They're,
not
they're,
not
constant,
because
if
they're
refreshing
a
slightly
different
interval
to
the
step,
you're
actually
going
to
be
getting
back
different
data.
A
A
A
Does
this:
when
trickster
first
came
out,
there
was
a
bit
of
back
and
forth
over
whether
this
was
even
a
good
thing,
but
what
we
did
because
I
work
for
co-founder
labs,
we
just
changed
Ravana
to
always
issue,
step,
aligned,
queries
as
well,
so
actually,
if
you're
using
graphing
on
top
of
Prometheus
and
on
top
of
cortex,
you
won't
notice
any
difference.
It'll
behave
exactly
the
same,
and
so,
if
you're,
using
graph
honor
to
send
queries
to
cortex
this
step
alignment
will
be
a
no
op
right.
A
It
won't
do
anything
if
you're
doing
it
from
your
own
user
interface
or
you're,
doing
it
from
a
different
user
interface
that
doesn't
do
this
step
alignment,
then
cortex
will
subtly
adjust
the
query
by
a
few
seconds
to
make
the
result
more
cashable,
and
this
is
where
we
start
to
diverge
from
what
what
trickster
does
we
start
to
do
our
own
thing?
We
actually
take
that
resulting
query
and
we
split
it
up.
We
split
up
into
different
sub.
Why
can't
you
use
the
term
sub
queries
because
they
exist
in
Prometheus?
A
A
However,
you
want
we're
reusing
this
exact
code
base
for
a
different
system
where
we've
chosen
to
split
it
up
by
our
instead
of
by
day,
but
we
chose
to
split
it
up
by
day,
and
this
is
because
the
index
we
use
in
cortex
is
day
aligned
every
day
you
get
a
fresh
index,
so
by
splitting
up
by
day,
we
don't
get
any
kind
of
read
amplification
on
to
the
index.
Basically,
the
same
load,
and
now
we've
got
for
in
this
example
for
independent
queries,
we
can
actually
go
and
look
them
up
in
the
cache.
A
Sometimes
you'll
get
a
hit.
You'll
go
like
oh
I.
Ask
you
today
Squier
a
few
days
ago,
and
that
day's
results
already
in
the
cache
done.
Don't
need
to
do
anything
more,
but
actually
what's
more
interesting.
Is
the
I
don't
know
whether
you
can
see
my
cursor?
You
can't
the
one
kind
of
towards
the
bottom
on
the
right,
there's
kind
of
a
partial
hit
like
we
looked
in
the
cache
for
a
day.
A
We
found
it,
but
it
didn't
contain
a
whole
24
hours
worth
of
data
and
maybe
only
contained
20
hours
or
18
hours
worth
of
data.
So
in
that
case
we
actually
have
to
like
construct
a
kind
of
a
new
partial
query
for
the
data.
That's
missing,
then
the
final
thing
is
not
made
obvious
here.
Is
we
never
cache?
A
We
never
returned
recent
cache
data,
so
it's
configurable
again.
We
set
it
to
10
minutes,
but
it
will
never
use
the
cache
for
the
last
10
minutes
of
data.
You'll
always
go
to
the
storage
for
10
minutes
of
data.
This
is
because
cortex
is
a
push
based
system
from
easiest
pushes
metrics,
the
cortex
it
might
get
slightly
behind.
You
know,
might
get
behind
by
a
minute
or
two,
and
so
that
that
last
10
minutes
of
data
is
still
changing.
It's
still
being
updated.
It's
still,
not
not.
You
know
not
not
constant.
A
Yet
anyway,
once
we've
constructed
this
list
of
queries,
we've
filtered
out
the
ones
that
already
cached
we
found
the
kind
of
partial
cache
hits
in
the
partial
cache
cache
hit.
Logic
is
actually
kind
of
fun
because,
like
you
can
have
like
an
hour
in
the
middle
of
a
24
hour
block
and
then
another
like
30
minutes
in
the
middle
of
that
block,
and
you
have
to
kind
of
decompose
them
into
almost
an
arbitrary
number
of
sub
queries,
which
is
kind
of
cool.
A
Anyway,
then
we
actually
dispatch
them
in
parallel
off
till
you're
off
to
our
back-end
right
and-
and
this
is
how
we
get
the
speed
up,
because
we're
going
to
execute
this
one
for
day,
query
we're
gonna,
actually
execute
it
as
three
separate
queries,
some
with
cached
results.
This
is
how
we
can
start
to
paralyze
start
to
use
that
scale
out.
Query
back
a
query
service
and
actually
process
bits
of
this
query
in
parallel,
there's
a
bit
more
to
it
in
cortex,
there's
a
bit
more,
we
do.
A
We
actually
have
a
queue
that
they
go
into
first.
This
is
because
cortex
is
a
multi-tenant
system.
We
have
a
few
hundred
clients
use
sharing
a
cortex
cluster
with
isolated
data.
We
really
don't
want
one
client
to
come
along
and
mass
refresh
on
their
browser
and
affect
other
clients
group
performance,
so
we
actually
per
quarter
per
client
and
we
do
come
a
really
naive,
really
basic
quality
of
service
between
clients.
A
Anyway.
So
that's
that's.
How
the
query
front-end
works.
I
wanted
to
I
wanted
to
talk
about
something
I
thought
was
pretty
cool.
This
is
that
this
idea
came
up
about
two
months
ago.
A
month
ago.
Nothing
we've
done
here
is
actually
cortex
specific.
But
what
happens
if
I
wanted
to
use
this
system
use
the
query
front-end
with
a
system
that
isn't
cortex
could
I
use
it
with
Thanos.
That
said,
could
I
do
just
use
it
with
Prometheus?
A
It
turns
out
you
can
we
did
this
PR
the
date
on
here?
Third
of
June?
Ok,
so
it's
like
four
or
five
months
ago.
We
did
this
PR
to
make
it
so
that
we
can
avoid
that
queueing
logic
and
just
dispatch
the
queries
directly
to
any
Prometheus
compatible
API,
so
be
it
panels,
be
a
Prometheus,
even
m3,
even
like
in
flux,
anything
that
has
a
Prometheus
compatible
API.
You
can
now
put
the
query
front
end
in
front
of,
and
it
will
cache
the
results.
A
A
A
A
Right,
oh
there's,
nothing!
Nothing
on
my
sleeves
I
do
have
a
make
it
bit.
Bigger.
I
do
have
a
graph
on
or
running
in
the
background
that
I
kind
of
set
up
beforehand,
but
there's
nothing
special
there.
What
I'm
gonna
do
I'm
gonna
run
the
cortex
query
front-end,
and
it's
just
this
this
one
command
here.
A
They
go
so
there's
prebuilt,
config
file
that
sets
this
up
for
you,
so
you
don't
have
to
worry
about
it
and
and
how
many
of
you
know
about
that
demo,
dot
robust
perception
die.
Oh,
this
is
like
one
of
the
most
useful
websites.
If
you
ever
want
to
demo
Prometheus
to
anyone,
it's
run
by
a
a
small
Irish
startup
run
by
one
of
the
Prometheus
maintainer,
and
he
runs
this
and
maintains
this
Prometheus
and
it's
got
loads
of
data
in
it
and
you
can
send
it
queries,
and
he
doesn't
mind
me
doing
this.
A
So
I'm,
just
gonna
use
that
as
a
demo,
so
I'm
gonna
point
the
court
with
a
query:
front-end
app
his
Prometheus
over
the
Internet.
There
we
go
now
I'm
gonna
switch
to
the
graph
honor
make
that
a
bit
bigger
now
I've
got
two
data
sources
in
here.
I've
got
Prometheus
pointed
directly
at
robust
perception
and
I've
got
a
cached
version
of
Prometheus,
pointed
at
my
local
machine,
which
is
then
pointed
at
robust
perception,
but
just
to
show
this
working
we'll
go
to
the
non
cached
one
first
and
because
I
can
never
do.
A
Histogram
functions
live
I'm.
Just
gonna
copy
out
the
slides
paste
that
in
now
this
is
only
running
over
the
last
hour
of
data,
so
take
600
milliseconds,
even
though
the
servers
running
somewhere
else
in
the
world.
So
this
is
really
quick,
but
we
we
need
to
come
up
with
something,
that's
a
bit
more
aggressive,
so
you
can
really
see
the
caching
effect
and
the
paralyzing
effect
a
parallelism
effect.
Let's,
let's
do
a
bit
longer
if
we
do
three
hours
now,
it's
still
pretty
quick.
A
This
is
all
now
in
the
page
cache
in
that
Linux
VM,
still
pretty
quick
at
6
hours.
This
is
actually
the
problem
when
you're
doing
like
query,
optimization
work
on
Prometheus
resis
is
pretty
quick,
it's
quite
hard
to
actually
notice
any
difference.
So,
let's
just
let's
do
a
day.
Nope,
it's
got
faster.
Let's
do
seven
days
here
we
go
now.
It's
actually
kind
of
hard
to
do
some
work,
two
and
a
half
seconds.
That
is
quite
quick
and
the
Wi-Fi
here
is
probably
faster
than
my
hotel
anyway.
A
So
doing
this
request
directly
against
Prometheus
took
two
and
a
half
seconds
now.
What
I'm
gonna
do
is.
Do
this
request
against
the
local
query
front-end
on
my
laptop?
This
is
gonna,
there's
no
cache
because
I've
just
started
it.
So
it's
gone
in
a
completely
empty
cache.
This
is
going
to
split
the
requests
up
into
seven
requests
and
dispatch
them
in
parallel
or
to
a
single
premiere
Prometheus
instance.
If
I
do
that,
you'll
see
it
takes
1.4
seconds,
this
is
already
faster.
This
is
no
caching.
A
This
is
twice
as
fast,
even
though
there's
no
cash,
just
through
the
parallelism
effect
interesting,
you
got
on
one
of
the
prometheus
maintained.
That's
had
a
bet
with
me
that
this
wouldn't
be
any
faster
because
it's
still
going
to
a
single
machine,
but
what
this
is
really
showing.
You
is
even
just
by
dispatching
multiple
independent
queries
to
the
same
Prometheus
note.
It
can
start
to
use
more
of
its
cause,
each
of
those
queries
being
processed
on
a
single
core.
A
It
can
start
to
use
the
multi
cores
on
the
box
and
it
gets
faster,
but
then
the
really
cool
stuff
happens
now
that
I've
run
that
query
one
once
the
results
of
this
query
are
sitting
in
the
cache
on
my
local
machine,
and
so,
if
I
refresh
this
takes
200
milliseconds.
This
is
now
10
times
faster
than
it
was
directly
hitting
the
Prometheus.
So
you
know
I
think
that's
pretty
cool.
A
Now,
let's
continue
yes
great
Odin's
Raven,
but
but
that's
not
the
end.
That's
this
is
where
I
know
I
finished
the
talk
when
I
last
gave
it
I
kind
of
stopped.
There
told
people
you
know,
go
and
deploy
this.
It's
one
command.
You
go
and
deploy
entire
in.
You
know
inside
your
company
and
stick
it
between
the
Gravano
and
the
prometheus
you'll,
just
relieve
a
load,
a
load
off
of
the
Prometheus.
All
your
users
will
be
happy
because
they've
refiner
dashboards
will
be
much
faster
and
we're
done
right
wrong.
There's
one
more
thing.
A
This
still
means,
if
you're
executing
a
query
for
the
last
sub
24
hours
worth
of
data.
If
you're
executing
a
query
for
the
last
few
hours
worth
of
data
and
you're
hitting
millions
tens
of
millions
of
time-series,
it's
still
not
going
to
be
fast
enough.
You
know
internally
we're
like
we
really
want
queries
to
be
instant,
like
anything
around
100
millisecond
mark
is
considered
to
be
basically
instant,
and
so
our
aim
is
to
be
able
to
process
kind
of
millions
tens
of
millions
of
series
in
the
hundreds
of
milliseconds.
A
This
is
crazy
right,
because
if
you
do
the
math
I
don't
know
you're
going
to
be
pulling
in
gigabytes,
if
not
tens
of
gigabytes
of
data
in
sub-second
and
aggregating
all
together
like
this
is
really
hard,
and
this
approach
I've
got
which
I
kind
of
call
kind
of
tight
and
a
called
time-based
parallelism.
This
isn't
going
to
help
in
that
approach,
so
we
need
to
need
a
different
form
of
parallelism
right.
A
We
need
a
different
way
of
guarding
the
query
executing
across
multiple
course
in
combining
the
results
so,
but
the
sake
of
the
rest
of
this
talk
there's
only
five
more
minutes
to
go.
For
the
sake
of
the
rest
of
this
talk.
Imagine
all
Prometheus
queries.
Look
like
this,
and
this
actually
isn't
too
much
of
a
stretch
like
this
pattern,
probably
covers
80%
of
queries
to
your
Prometheus
server
and
the
intuition.
A
This
is
just
MapReduce
right.
This
is
just
the
same
theory
behind
MapReduce.
These
sums
are
independent.
They
can
be
executed
independently.
As
long
as
each
of
these
partial
queries
runs
over
independent
data,
then
this
is
kind
of
accurate.
You
know
I
quoted
the
plus
here,
because
you
know
you're
not
actually
adding
these
together.
You
need
some
kind
of
merge,
reduce
operation
and,
of
course,
like
this
shard
equals
1
won't
work
either,
like
Prometheus,
doesn't
support
this
out
the
box.
A
We
just
wanted
to
shorten
the
row
lengths
in
the
index
because
we're
overloading
our
big
tables,
but
it
enabled
this
extra
thing
and
again
by
picking
and
aligning
the
sharding
that
you
do
at
the
very
front
end
with
the
way
we've
laid
out
our
index.
We
managed
to
do
this
without
any
amplification
of
the
read
load
on
the
index.
Anyway,
we
started
doing
this.
Actually,
I
didn't
start
doing
this
Oh,
who
is
not
in
the
audience.
A
No
Owen
and
Cyril
have
kind
of
been
working
on
this,
and
so
I
can't
take
the
credit,
but
just
to
give
you
an
example,
this
works
there's
code,
there's
a
PR
that
exists
and
we're
still
like
we
haven't,
got
it
into
production
yet,
but
but
it
works,
and,
to
give
you
an
example,
here's
a
query
over
quite
a
few
time
series.
This
is
running
one
of
our
larger
clusters
and
there's
you
know
thousands
of
containers
and
and
I
think
each
machine
has
like
16
cores.
A
So
this
is
kind
of
this
is
not
millions,
but
this
is
hundreds
of
thousands
of
of
time-series
and
because
it's
kind
of
we
turn
the
caching
off.
We
just
want
to
highlight
the
effect
of
the
parallelization.
This
query
is
artificially
slow,
but
but
it
really
demonstrates
the
effect
so
without
the
without
the
caching
and
without
the
parallelization.
A
This
query
was
taking
about
ten
seconds
in
our
dev
environment
and
once
we
turn
the
caching
and
parallelization
on
the
exact
same
query
on
the
exact
same
data
was
about
three
times
faster,
but
we
were
hoping
it
would
be
16
times
faster,
but
we'll
take
three
combined
with
everything
else.
I
think
that's
a
pretty
big,
pretty
big
improvement,
and
once
you
turn
caching
on
once
you
deploy
this
in
environment
with
enough
resources
and
and
hundreds
of
replicas
of
the
query
service.
A
A
We
haven't
done
this
yet,
but
we
want
to
take
this
technology
and
apply
it
to
Thanos
right
in
a
Thanos
environment.
You
can
actually
guarantee
certain
sets
of
data
are
isolated
because
they're
stored
at
the
edges
in
the
individual
Promethea.
So
you
know
once
we
start,
you
know
once
we
start
merging
this
with
some
of
the
Thanos
work,
we
can
actually
hopefully
push
down
these
partial
aggregations
out
to
the
edges
and
only
ship
back.
A
So
what
are
we
gonna
do
in
the
future?
Well,
we've
kind
of
you
know:
I,
don't
know
whether
you've
realized.
What
we
kind
of
built
is
a
query
planner
for
prom
QL,
like
if
anyone's
hears
from
with
SQL
databases,
we're
really
kind
of
looking
at
the
query,
we're
picking
it
apart,
we're
figuring
out
which
bits
can
be
executed
in
parallel.
You
know,
so
we
really
want
to
kind
of
tease
this
apart,
a
bit
more
and
separate
kind
of
the
planning
from
the
execution
of
the
queries
so
that
we
can
do
even
more
interesting
optimizations.
A
One
of
the
things
I
really
want
to
do
is
like
averages,
can't
be
can't
be
reallocated
right.
You
get
a
different
answer.
If
you
do
that,
but
if
you
decompose
an
average
into
a
sum
divided
by
account,
you
can
parallelize
the
individual
sum
and
count
do
the
binary
operation
centrally
and-
and
you
know,
different
tricks
like
this
I
think-
will
make
even
more
different
Prometheus
queries,
parallelizable
and
faster,
and
then
this
is
one
we've
been
talking
about.
I,
don't
quite
know
whether
this
is
actually
going
to
work.
A
Yet
we
haven't
done
the
work,
but
one
of
the
weak
points
with
Prometheus
is,
if
you
do
like,
you
want
a
year's
worth
of
data,
and
you
want
to
do
like
rate
30
days.
You
know
over,
you
know,
get
a
rolling
one
month
window
over
a
year's
worth
of
data.
Every
single
point
is
going
to
have
to
consider
a
whole
months
worth
of
data,
and
this
gets
really
expensive.
A
So
we
want
to
kind
of
decompose
a
30
day
rate
into
like
a
series
of
one-day
rates
and
then
combining
this
with,
like
maybe
memorizing
the
cache
or
something
hopefully,
reuse,
overlapping,
one-day
rates.
I
think
this
is
going
to
be
really
cool,
and
this
is
this
is
where
we're
going
to
take
this
I
hope.
Anyway.
A
A
How
do
we
even
start
to
identify
which
queries
are
running
slow?
It's
a
good
question
in
production
we
log
every
single
query
out
to
out
two
loci
actually
funnily
enough
and
then
we've
built
some
scripts
that
process
those
logs
link
them
to
traces
and-
and
we
can
just
kind
of
literally
look
over
the
last
few
weeks
worth
of
queries,
yet
the
ones
that
were
in
that
1%
little
slowest
and
then
go
and
view
the
traces
for
them.
But
this
is
kind
of
custom.
A
Now
too,
we
log
every
query,
because
we've
got
a
logging
infrastructure,
but
if
you
don't
have
that
or
if
you
don't
want
to
log
that
much
volume
you
can
set
tell
accrue
front-end
only
to
log
queries
over
a
certain
amount
of
time
or
log,
slow
queries
and
yeah.
That's
basically,
we
do
log
analysis
to
identify
slow
queries
over
there.
A
A
A
So
even
if
you've
got
the
window
open
for
a
few
minutes,
you
probably
issued
you
know:
20
queries
for
each
panel
and
you're
gonna
have
a
huge
hit
rate
just
in
the
moment,
but
between
different
sessions.
Maybe
not
you
know
between
different
sessions.
It
depends
how
big
your
team
is.
It
depends
who's.
Looking
at
these
queries,
you
know
one
of
the
things
the
father
has
its
own
inbuilt,
alerting
system.
I'm,
not
you
know,
I,
don't
use
it
myself,
but
we
use
it
internally
or
grow.
A
Fana
labs-
and
that
is
just
constantly
doing
the
same
query
over
and
over
again
like
every
few
and
no
tens
of
seconds
or
something,
and
when
we
introduce
the
caching
system
that
saw
a
huge
increase
in
performance
that
saw
a
huge
cache
hit
rate,
but
it
does.
It
is
work
like
dependent,
like
I.
Think.
If
you
only
ever
go
to
a
graph
are
no
graphs
like
once
a
month,
then
no
there's
not
gonna,
be
any
cache.
It
right,
like
it's
really
for
people
who
are
using
it
more
actively
than
that.
Does
that
answer
question?
A
A
A
A
So
the
question
was
we:
we,
we
use
Prometheus,
X
or
chunks,
which
is
like
I
forgotten,
the
name
of
the
paper
really
famous
there
gorilla
paper
yeah.
We
basically
use
the
gorilla
encoding
in
chunks
for
each
individual
time
series
in
Prometheus
and
in
cortex
and
in
Thanos
it's
quite
a
popular
system.
We
we
don't
use
the
same
format
for
the
query
results
we
actually
like
use
protobufs.
A
It's
interesting
that
you
mention
I
haven't
thought
of
it
like,
because
some
of
these
query
results
are
very
large,
like
we've
had
to
configure
our
memcache
is
to
like
store
values
of
hundreds
of
megabytes
of
protobufs
and
yeah.
I,
see
no
reason
why
we
couldn't
encode
them
into
the
same
X
or
chunk
format.
Yeah.
That
makes
a
lot
of
sense
even
better
right.
A
The
X
or
chunk
format
when
you
have
very
very
repeatable
timestamps,
like
with
very
even
differences
between
them,
gets
really
efficient
and
in
query
results
you
definitely
have
very
even
gaps
between
your
values,
I
might
I
might
actually
go
and
look
at
look
at
doing
that.
I
think
it's
a
really
good
idea.
Thank
you
gone
then.
A
We
get
this
every
time
every
time
so
that
one
of
the
panels
maintainer,
is
sitting
just
here,
Bartek
sorry,
so
the
the
question
was
I've
mentioned
cortex
and
Thanos
multiple
times
and
there's
there
any
plans
to
merge
these
projects
and
categorically
I
can
say
here.
Yes,
no
I,
don't
know
like
that's
like
the
Thanos,
like
what
I
you
know,
Bartek
and
I
both
live
in
London.
We
both
enjoy
going
for
a
beer
together,
I
think
I
would
love
to
work
more
with
him
and
more
closely
with
him,
making
things
like
the
query.
A
Front-End
work
with
honest
we've
actually
recently
embedded
Thanos
into
cortex,
so
cortex
can
now
store
blocks
in
object
stores
in
exactly
the
same
way
that
us
can-
and
this
opens
up
more
opportunities
to
collaborate
on
like
extending
Thanos
performance
for
like
the
store
gateway
and
stuff
I.
Like
the
fan
esteem,
you
know
it's
a
large
overlap
with
the
Prometheus
team
and
with
the
cortex
team
and
I
look
forward
to
collaborating
more.
If
that
means
the
projects
merge
great.
A
If
it
doesn't,
that's,
probably
also
fine
yeah
yeah,
then
they
differ
they're,
very
different
deployment
models
and-
and
that's
not
going
to
change
right
so
but
no
good
question.
We
get
it
every
time.
One
more
question
I
think
I've
got
time
I'm
going
to
keep
going
until
someone
stops
me
actually
at
the
back
over
there.
A
A
So
that's
that's
really
the
key
for
like
things
that
can
be
used
with
like
events
and
logs
and
things
that
can
use
new
metrics
like
if
we
put
that
query
into
a
metric
label,
the
cardinality
of
that
metric
would
blow
up
really
quickly
and
then
you
need
something
like
this
to
query
it
yeah.
Oh,
the
question
was:
why
don't
I
put
the
we
log
slow
crews?
Why
don't
we
put
it
in
a
metric
and
it's
all
about
cardinality
yeah,
any
more
questions,
no
more!
How
am
I
missing
anyone
Chris
go
on.
A
A
A
But
yeah
the
question
was:
do
we
have
like
an
explained
on
queries
like
we?
Don't
we
don't
have
at
least
a
user
accessible
one
I,
don't
think
there's
anything
can
Prometheus
that
does
this
either
I
think
it's
a
great
idea
like,
as
you
said,
we
have
Yaeger
so
internally
we
can
see
why
queries
are
slow
because
we
just
go
and
look
at
Yaeger
and
I.
A
So
if
they
want
to,
they
can
go
through
and
you
can
actually
export
Yaeger
traces
as
big
blobs
of
JSON
and
then
you
could
just
you
know,
give
an
API
for
a
customer
to
go
and
get
that
and
then
they
could
load
it
into
their
own
Yaeger
instance
and
visualize.
It
ya
know
we
don't
we
don't
have
it
but
I
think
it's
a
good
idea,
and
especially
when
the
query
planner
starts
getting
more
and
more
complicated,
and
you
know
eventually
we're
gonna
have
to
deploy
various
different
heuristics.
A
We're
already
doing
this
for
like
deciding
what
to
query
in
the
index.
You
know
there's
basically
no
point
in
querying
very,
very
large
dimensions
in
the
index,
so
we
kind
of
already
use
some
basic
heuristics
when
we're
hitting
our
inverted
index
to
kind
of
figure
out
what
to
query
and
yeah,
like
there's
definitely
feedback
where
you
could
give
to
the
user.
Saying
like
just
don't
bother
touching
this
label.
It's
got.
You
know
it's
got,
30
million
entries
in
it
doing
reg
X's
over
that
I'm
never
going
to
work
but
know
it's
a
good
idea.