►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Every way we DDoSed our Cortex - Goutham Veeramachaneni, Grafana Labs
Cortex is a distributed version of Prometheus with a lot of moving parts. We have a pretty good getting started guide with enough information to get a working cortex cluster that can ingest data and answer queries. But there is limited material on the day 2 operations: Capacity planning, query performance debugging, and general health monitoring. In this talk, we will take you through the debugging workflow, the typical knobs that should be tweaked for optimal performance, the mixin for cortex that covers the dashboards and alerts, and in general how to approach debugging and maintaining an existing cortex cluster.
https://sched.co/UagC
A
So
hello,
folks,
this
is
going
to
be
about
cortex
or
how
to
performance
tuned
cortex
for
scale.
How
many
of
you
know
what
cortex
is?
Okay,
so
you're
in
the
right
room?
How
many?
If
you
are
already
using
cortex
nice
you're,
gonna
love
this
talk,
so
this
is
the
cortex
deep
dive.
I've
actually
changed
the
title
a
bunch
of
times
on
my
slides
and
in
the
end,
I
ended
up
with
every
way
we
need
washed
our
cortex
or
how
to
make
sure
your
cortex
will
scale
with
your
usage.
A
So
a
little
bit
about
me,
I'm
Gautham,
I'm,
actually
I
started
as
a
Prometheus
maintainer
and
then
I
started
working
on
cortex
and
refine
elapsed
and
now
I'm
a
cortex,
maintainer
and
I.
Also
co-author
Loki,
along
with
Tom
Loki,
just
hit
one
point
over,
which
is
amazing
and
Loki,
is
also
built
on
top
of
cortex.
So
whatever
I
am
saying
for
cortex
also
applies
to
Loki,
so
that's
nice.
A
So
this
is
the
architecture
of
cortex,
so
I'm,
just
gonna
like
I'm,
not
going
to
do
all
the
components
and
all
the
ways
it
can
break
all
the
ways
you
can
tame
this,
but
I'm
gonna
give
you
a
framework
on
how
you
can
fix
things
if
you're
running
cortex
or
in
general
any
system
that
is
a
database
or
a
distributed
database.
So
for
those
of
you
who
are
not
familiar,
cortex
is
a
horizontally
scalable
distributed
multi-tenant
version
of
Prometheus,
it
has
all
the
buzzwords,
and
so
the
beauty
of
cortex
is
its
configurability.
A
A
So
again,
this
means
you
can
tune
every
single
piece
of
cortex
individually
and
we
have
a
bunch
of
compostable
components
that
actually
so
essentially,
we
have
a
caching
component
that
has
memcache
radius
in
memory
on
disk
and
then
you
can
use
the
same
cache
component
in
six
different
places.
So
that's
like
60
or
70
flags
right
there.
A
So
there's
a
bunch
of
things
that
you
can
configure
it
in
cortex
and
I
actually
wanted
to
control
minus
minus
help
and
you
scroll
through
for
like
30
minutes
but
yeah,
so
I'm
gonna
actually
tackle
two
things:
the
right
path
and
the
read
path
and
how
to
actually
make
sure
you
were
configuring
cortex
for
you
for
scale,
so
the
right
path.
So,
if
you
can
see
this
see,
my
pointer
so
essentially
Prometheus
writes
to
something
called
the
gateway
that
does
authentication
that
adds
the
tenant
idea.
A
Tenant
header
sends
it
to
the
distributor
the
distributor
sends
it
to
the
industry.
The
injector
actually
takes
all
these
samples,
individual
samples
builds
that
builds
them
up
into
bigger
chunks
and
then
writes
it
into
BigTable,
GCS
and
and
a
cache.
So
this
is
this
is
a
small
subset
of
the
right
path
that
you
see
and
I'm
just
gonna
cover
this
bit
of
the
right
path.
Now
so
again,
suddenly
you're
operating
cortex
and
suddenly
you
get
paged
saying
the
rights
are
failing
or
Prometheus
is
failing
to
write
data
or
the
Prometheus
remote
write
is
falling
behind.
A
A
So
how
many
of
you
have
heard
of
the
read
method
so
read
method?
Is
this
method
of
how
to
instrument
your
distributed
applications,
especially
if
they're
RPC
systems?
So
it
says
you
need
to
track
your
request,
errors
and
duration.
So
again,
as
I
said,
we
have
a
gateway.
This
is
the
cortex
component
that
talks
to
the
distributor
that
talks
to
a
net
CD
that
oxygen
in
gesture,
the
in
gesture
rights
to
memcachedb,
BigTable
and
stuff,
like
that.
A
So
essentially,
this
is
like
a
tiered
application,
where
a
talks
to
me
talks
to
C
talks
to
D,
and
you
have
the
QPS
and
the
latency
on
the
occupation
on
the
left
and
latency
on
the
right.
The
beauty
of
this
is
if
something
goes
to
red.
For
example,
you
see
the
gateways
red
500s
are
being
thrown,
distributor
is
red,
but
the
injection
is
green.
So
immediately
you
can
understand.
The
distributor
is
what
is
throwing
these
letters
and
that
you
need
to
look
at
the
distributor
distributor
logs,
not
the
ingest
of
logs.
A
So
again,
the
same
thing
applies
for
latency
you're,
seeing
okay,
that's
like
40
milliseconds
later
in
C
on
rights.
Right,
it's
a
super
quick
there's,
a
40
millisecond
agency
on
the
distributor
and
the
industry
is
only
having
a
5
millisecond
latency,
which
means
the
distributorship.
What
is
adding
the
latency?
So
if
you
actually
have
distributed
systems,
if
you
can
build
these
very
consistent,
templated
and
dashboards,
that
will
actually
give
you
at
one
glance
what
is
wrong
with
your
system?
What
is
right
with
your
system?
A
So
this
is
a
cortex
right
dashboard,
where
we
have
everything
tiered.
So
one
thing
is
everything
that
I'm
talking
about
here
is
open
source,
I,
just
open
sources.
Today
morning,
I'm
going
to
give
you
I'm
going
to
walk
you
through
how
you
can
get
these
own
dashboards
and
alerts
for
you
or,
if
you're
running
cortex,
and
the
next
thing
is
okay,
everything
is
green,
but
just
slow.
So
we
have
a
SLO
of
one.
A
Second
99.9%
of
our
writes
need
to
pass
within
one
second
and
if
I'm,
seeing
okay,
suddenly
something
is
being
limited
or
like
I'm
being
rate
limited
or
I'm.
Just
seeing
high
latency
I
have
a
scaling
dashboard
and
the
scaling
dashboard
is
super
important
because
we
run
cortex
as
a
service
and
initially
I
set
up
a
cortex
provision
date,
blah
blah
blah,
and
then
the
sales
team
went
ahead
and
signed
up
a
bunch
of
customers
and
I
forgot
to
upgrade
my
cortex
or
like
scale
it
up
and
everything
started
to
slow
down.
A
Things
started
to
own,
so
I
built
this
dashboard.
That
does
two
things.
One
is
workload
based
scaling
and
the
other
is
resource
based
scaling.
So
we'll
talk
a
little
space
scaling,
which
is
the
obvious
one.
Essentially
there
are
twelve
replicas,
but
if
you
look
at
the
CPU
usage
and
CPU
request,
you
need
thirty
replicas.
So
I
can
just
take
a
take
one.
Look
at
this
dashboard
and
see.
Okay,
I
know
exactly
what
is
wrong
here
and
the
other
thing
is.
This
is
our
dev
system.
This
is
our
dev
namespace,
so
it's
intentionally
under
provisioned.
A
Similarly,
there
we
have.
This
were
closed.
Work
load
based
scaling.
So
if
you
are
familiar
with
cortex,
do
you
have
this
in
gesture
component,
for
example,
and
the
interest
component
stores
the
recent
6
to
12
hours
of
data
that
so
essentially
you
send
all
the
samples
to
the
in
gesture.
It
batches
them
up
to
6
into
6
hour
chunks
and
then
writes
the
chunk
through
big
table
or
dynamo
DB
just
to
reduce
Reaper
write
amplification
and
to
have
compression
and
everything.
A
So
we
actually
limit
the
number
of
active
cities
and
in
gesture
can
do
because
we
notice
that,
regardless
of
6
hours
or
12
hours,
the
memory
being
used
by
an
in
gesture
is
dependent
on
the
number
of
active
cities.
It
has
and
we
limit
the
amount
of
active
cities
to
1
million
to
one
and
half
million
cities.
But
it's
all
dependent
on
your
on
your
use
case
and
how
tightly
you
want
to
run
your
system
so
because
the
recent
data
is
in
the
in
jesters.
A
The
courier's
also
hit
the
in
jesters
so
for
their
queries,
and
this
means,
if
this,
if
the
in
gesture
has
two
miniseries,
the
in
gesture
will
add
some
latency
to
your
queries.
If
the
industry
is
just
managing
smaller
number
of
cities,
its
the
queries
are
going
to
be
quicker
and
even
during
rollouts
we
need
to
actually
transfer
this
live
in
mem
data
toward
to
the
new
ingest.
So
that
came
up
so
that
we
don't
partially
flush
things.
So
the
transfers
are
also
faster.
A
Having
said
that,
you
need
to
get
comfortable
with
a
number
and
then
stick
to
that
number.
So
we
Brian
from
V
works.
He
runs
the
cortex
with
4
million
series
with
64
gig
ram,
while
we
run
it
at
1
million
Series
with
16
gig
ram.
So
these
are
all
trade-off
that
you
need
to
make,
but
just
make
sure
that
you're
scaling
along
with
usage.
So
the
moment
you
tell
your
users,
hey,
we
have
this
cool
system
called
cortex.
You
can
send
all
your
matrix
here.
They'll
be
stored
forever.
A
People
will
send
you
a
ton
of
matrix
so
make
sure
that
you
have
alerts
on
all
of
this
and
then
make
sure
you
can
scale.
So
you
can't
just
stare
at
dashboards
all
day.
You
need
to
actually
make
sure
that
your
paged
for
this,
so
we
have
alerts
for
everything.
So
essentially,
the
other
thing
we
do
is
we
make
sure
we
catch
the
recent
two
days
of
data
in
memory
in
cassadee,
so
that
most
of
your
queries
are
going
to
be
on
the
recent
two
days.
A
So
if
you
open
a
mode,
it's
going
to
be
last
24
hours,
law
or
last
48
hours
and
most
pores
are
just
last
one
are
so
we
noticed
that
99.9%
of
our
queries
are
actually
only
looking
at
the
recent
two
days
of
data.
It's
the
rare
query.
Over
30
days
that
looks
at
the
older
data,
so
we
wanted
to
make
sure
that
everything
is
in
cache.
So
it's
super
fast
and
also
it
reduces
the
load
on
the
remote
system
like
GCS
or
BigTable.
A
So
we
have
a
alert
that
says:
okay,
based
on
this
time
series
and
these
many
chunks
being
generated.
You
need
these
many
mem
caches.
Do
you
have
enough?
Do
you
have
like?
Are
you
sure
you're
only
running
at
most
1.5
active
time,
series
per
ingested
and
stuff
like
that?
So
the
good
thing
is
the
first
three
alerts
are
actually
warning
alerts,
so
you
don't
get
paste
so
I
get
a
slack
notification.
Whoever
gets
who's
on
call.
A
A
A
So
this
is
a
paging
alert
because
again,
as
I
said,
the
recent
six
hours
of
data
is
in
memory
in
the
in
gesture
and
we
don't
want
the
investors
to
own.
So
if
suddenly
somebody
is
sending
to
me
too
many
metrics
and
the
interest
in
memory
is
just
growing.
We
get
we
get
paid,
saying.
Okay,
the
industry
memory
is
growing,
you
might
have
data
loss.
A
The
next
thing
is,
we
also
have
SLO
sand,
SLS
and
SL
eyes
for
everything,
including
our
read
and
writes,
so
you
should
watch
beyond
stock
on
how
to
do
SL
loss
right.
So
we
have
our
six,
our
error
budget,
one
our
error,
budget
and
budget.
So
we
have
all
this
error,
but
it's
written
down
and
s
ellos
are
public.
It's
just
that
our
queries
should
be.
99.9%
of
our
query
should
be
faster
than
2.5
seconds
and
successful
and
99.9%
of
our
write
should
be
successful
and
more
than
slower
than
one
second.
A
So
we
have
all
these
SLO
based
or
alerts.
So
if
something
is
wrong
in
the
right
path,
I'll
get
alerted
within
15
minutes
and
then
I
can
go.
Take
a
look
at
my
my
writes,
dashboard
and
then
I
can
start
debugging
things
by
looking
at
logs
or
traces
and
stuff
like
that.
So
this
is
actually
the
easy
bits:
the
right
path.
How
do
I
get
access
to
all
these
nice
alerts?
So
we
actually
open
sourced
our
JSON
at
in
the
morning.
A
So
yes,
so
this
repo
has
actually
the
JSON
it
for
generating
all
the
amell's
for
deployments.
The
JSON
it
only
currently
supports
the
GCS
or
BigTable
and
Cassandra
backends,
not
the
Olympius
back-end,
what
we
are
going
to
add
the
AWS
back
and
remove
some
graph
on
a
specific
bits
and
then
put
it
into
the
cortex
project
open
source.
And
if
you
want
to
look
at
the
alerts
it's
in
the
cortex
mixin,
so
the
readme
has
instructions
on
how
you
can
generate
these
alerts.
But
again
these
are
our
windows.
A
So
it's
like,
if
you
like
again,
there's
this
nice
blog
on
how
to
do
alerting
on
SLO.
So
we
use
that
blog
and
say
our
latency
threshold
is
2.4
seconds
for
read,
and
then
we
have
a
bunch
of
alerts
like
if
the
in
gestation
haldi
it's
a
critical
alert,
critical
alert
pages
flushes
are
being
stuck
the
pages,
but
if
requests
are
failing
for
15
minutes,
we
have
warnings,
so
things
like
that.
A
So
we
have
both
warning
and
critical
alerts
and
warnings,
don't
page,
but
it's
the
job
of
the
on-call
person
to
keep
an
eye
on
that.
So
all
of
this
is
open
source.
All
of
this
you
can
get
so
go,
get
some
alerts
and
monitoring
and
also
check
out
the
nice
way
of
how
to
package
dashboards,
allow
it's
recording
rules
and
the
amyl
together
in
the
form
of
JSON,
if
you're
not
familiar
with
it.
So
we
recently
hired
an
HDD
outage.
A
We
had
an
outage
for
two
hours
where
we
just
couldn't
accept
rights,
that's
because
I
actually
suddenly
decided
it
didn't
have
a
leader
and
the
funny
thing
is
I.
As
I
said,
we
have
this
nice
dashboard.
That
was
talking
to
the
Gateway
talking
to
the
distributor
talking
to
the
HCD.
We
recently
added
eight
CD,
but
did
not
forgot
to
add
this
panel
in
the
in
the
dashboard.
A
So
when
rights
were
failing,
we
had
no
clue
why
they
were
failing,
but
that's
why
you
do
post-mortem
so
that
you
can
fix
your
mistakes
and
now
we
actually
have
the
panel.
Let
me
see
where
that
other
dashboard
is.
So
if
you
look
at
it
during
the
time
of
the
outage,
we
just
couldn't
write
stuff
and
our
it
CD
just
became
read
everywhere
so
yep
we
make
mistakes,
but
we
do
post
mortems
and
we
fix
them.
A
The
read
path
is
super
tricky
because
there's
just
too
many
components
and
the
tricky
part
is
the
queries
vary
in
size.
Sometimes
somebody
sends
a
query
that
worries
a
million
time
series
over
50
days
of
data
and
that's
bound
to
be
slow
and
then
there's
other
queries
that,
like
that's
life,
you
can't
make
it
any
faster
than
20
seconds
I
mean
we
are
trying
to,
but
you
don't
want
to
be
paced
for
that.
A
On
the
other
hand,
if
somebody
sends
a
simple
query
that
just
looks
at
one
time
series
about
five
days
of
data,
if
that
is
taking
ten
seconds,
you
want
to
be
paged
for
that.
So
we
don't
have
a
good
way
to
page
on
the
size
of
the
query
and
latency.
So
we
page
for
everything
and
turns
out.
We
are
fast
enough,
quick
enough,
so
that
we
are
really
paced
like
I'm
on
call
right
now,
I
think
I'm.
A
Actually
one
call
in
the
Indian
timezone,
so
I'm,
not
on
call
right
now,
I'm
on
call
tonight,
but
yeah.
If
you
look
at
the
read
path,
you
get
a
graph
on
a
sense,
a
query
to
the
query
front
and
the
query
front
and
talks
to
the
courier,
the
courier
talks
to
in
gesture
the
databases
and
a
bunch
of
caches.
A
So
this
is
the
reads:
dashboard
I
just
want
to
quickly
walk
you
through
the
Reeves
dashboard,
but
once
you
have
the
reads
dashboard,
so
it's
like
the
Gateway,
the
query,
front-end
the
caches
and
then
the
courier
and
stuff
like
that
is
the
same
thing.
So
for
debugging
queries
like
this
in
a
distributed
system,
especially
something
that
is
latency
sensitive,
you
need
the
right
tools
and
I
found
Jaeger
to
be
super-powerful.
So
first
thing
is
you
need
your
dashboards
and
you
need
your
alerts
to
tell
you
something
is
wrong,
but
to
actually
debug
what
is
wrong.
A
You
need
Jaeger
and
I
really
really
like
Jaeger,
and
it's
also
simple,
simple
to
configure
Jaeger
in
to
cortex.
You
just
need
to
add
a
bunch
of
environment
variables
like
these.
Three
internet
variables
are
needed.
Actually,
only
the
first
one
is
needed,
but
we
do
dynamic
sampling
and
we
add
some
extra
metadata
to
each
of
our
containers.
So
we
add
a
bunch
of
extra
flags
and
this
is
a
variable
that
we
import
and
we
just
added
to
our
container
wherever
required.
A
A
Yeah
the
cortex
reads
dashboard,
so
this
is
like
the
Gateway
or
the
front-end,
the
courier
they
ingest,
memcache,
lookups
and
BigTable.
So
this
over
the
last
24
hours
are
actually
intentionally
did
a
huge
query
that
took
a
long
time,
1
and
1/2
minute,
and
this
won't
immediately
page
me
with
this,
because
it's
just
one
query:
you
don't
want
to
be
paged
for
a
single
query.
If
I
consistently
do
these
queries
and
it
consistently
so
that's
when
I
will
get
paged.
A
That's
when
I
will
exhaust
my
error
budget
and
I'll
get
paged
I'll
be
like
ok,
so
the
gateway
is
seeing
one
and
half
minute
latency.
The
front-end
is
seeing
one
enough
minute
latency.
So
it's
probably
not
the
front
end
or
not.
The
Gateway
caches.
Ok,
cassia's
are
quite
quick.
The
query
is
seeing
significant
latency
one
minute,
so
the
front-end
is
adding
500
500
milliseconds
of
latency.
The
in
gesture
is
also
kind
of
slow,
so
the
query
er
makes
several
calls
to
the
in
gesture.
A
So
it's
kind
of
hard
to
see
where
the
latency
is
coming
from
and
in
the
end
we
can
see
that
big
table
itself
is
taking
21
seconds
average
to
respond
to
each
lookup.
So
it
probably
means
BigTable
is
overloaded,
but
that's
because
I
know
that
exactly
why
BigTable
is
overloaded
and
stuff,
because
I'm
I've
been
doing
this
for
18
months.
Just
with
one
look:
I
can
tame
this,
so
this
is
the
reads.
Dashboard
that
can
end
up
at
a
high
level.
Tells
me
what
is
what
is
slow?
A
The
good
thing
is,
you
should
all
for
those
of
you
who
have
not
heard
of
the
read
method.
You
should
go
check
out
the
read
method,
because
that
tells
you
how
to
build
these
consistent,
templated
dashboards
and
then
I
have
the
cortex
queries
dashboard.
That
actually
tell
me
tells
me
exactly
a
higher
level
data
about
the
queries
that
are
being
run.
Ok,
so
suddenly
I
see
that
my
query
queue
had
60
60
things
piling
up,
which
probably
means
that
is
what
that
was
significant.
A
So
we
saw
that
the
front-end
was
having
one
and
half
minute
latency,
but
the
queries
were
her
career.
Was
having
one
minute
latency,
so
the
500
milliseconds
was
being
added
by
the
query.
So
that's
a
so
that's
how
you
can
figure
things
out
and
you
can
see.
Okay,
my
friend
end
cache
hit
rate
is
super
high
I'm,
seeing
a
bunch
of
cache
misses
cache
hits,
so
you
can
see
how
different
caches
are
performing,
how
the
churn
rate
is
and
a
bunch
of
query
related
things
that
actually
helps
you
understand
is.
A
Are
these
queries
big
or
are
these
queries
slow?
You
can
see
that
most
queries
are
only
taking
ten
twenty
thousand
chunks
a
query,
but
currently
there
was
a
query
that
was
taking
190
Cait
songs.
Probably
there
was
a
huge
cache
miss
in
the
chunk
cache
and
that
is
causing
extra
load
on
the
big
table.
So
again,
this
is
another
dashboard
that
helps
me
figure
things
out
and
in
the
end,
I
also
have
a
memcache
dashboard.
A
That
tells
me
exactly
what
the
eviction
rate
is
commander.
It
is
what's
the
latency
memory
being
used
in
all
of
that,
so
I
use
these
three
dashboards
to
kind
of
understand
at
a
high
level.
What
is
going
on
and
if
I
can
figure
out
at
one
glance,
if
that
doesn't
work,
and
usually
it
doesn't
I
use
Jaeger.
A
So
how
many
of
you
are
familiar
with
Jaeger
already,
how
many
of
you
are
seeing
Jaeger
for
the
first
time
in
action?
Okay,
so
you're
gonna
love
this
demo,
so
I'm
just
gonna
pick
a
service,
the
cortex
pony
front,
end
and
I'm
gonna
say
I
want
the
API
prom.
This
is
the
query.
Endpoint
and
I
want
to
see
all
the
requests
that
took
more
than
one
second.
The
mint
duration
is
one
second
and
I'm,
seeing
a
bunch
of
requests.
So
let
me
look
at
this
one
that
took
30
seconds.
A
So
let
me
zoom
in
a
little
bit
and
then
expand
this.
So
this
is
a
live
demo
again.
I
have
a
perfect
trace
before,
but
let's
see
why
this
query
is
taking
37
seconds.
Hopefully
the
demo
gods
gods
are
with
me,
so
you
can
see
that
most
of
the
latency
is
being
introduced
by
the
prompt
queue
eval
function
in
this
and
if
I
look
at
the
prompt
queue
level
function.
A
A
The
demo
is
not
with
me
this
time,
but
yes,
so
I
can
see
that
this
particular
query.
It
took
16
seconds.
Let
me
see
if
the
cache
there
was
a
hit
or
not
okay,
he
tried
to
look
at
look
if
this
has
been
requested
before
there
was
a
cache
miss,
so
it
tried
to
bend
into
the
retry
middleware,
and
then
we
have
this
taking
16
seconds.
If
you
look
at
it,
prompt
you'll
prepare
function
is
taking
super.
It's
super
quick,
but
the
prompt,
you'll
see
restored.
Gate
function
is
taking
14
seconds.
A
A
So
in
this
function
you
can
see
that
something
is
taking
12
seconds.
Out
of
that,
you
can
see
that
most
of
the
function
is
being
done
by
this
query.
Paytas
function,
that's
taking
11
seconds
if
I
expand
that
I
can
see,
and
a
little
bit
more,
that
BigTable
is
actually
taking
seven
or
eight
seconds
to
respond
to
each
query.
Each
a
table-
lookup
I,
mean
each
row
lookup
and
that
is
adding
a
significant
delay
to
my
latency.
So
I
can
go
back.
Look
at
BigTable,
see
that
it's
under
provision
and
then
provision
BigTable.
A
So
this
way
by
using
Jager
by
getting
alerted
by
my
alerts
missing,
is
a
loss.
I,
look
at
dashboards
to
end
of
try
to
figure
out
what's
wrong
and
if
it's
not
clear,
I
then
use
Jager
to
kind
of
just
dive
into
exactly
what's
wrong.
So
I
spent
seven
or
eight
months,
just
like
in
a
query,
optimization
optimization
sprint,
so
I
made
sure
that
your
the
code
is
instrumented
with
the
right
functions
so
that
for
an
operator
it's
easy
to
figure
out
what's
wrong.
A
So
again,
if
you're
on
the
read
path,
the
things
that
you
need
to
look
out
for
are
your
cache
is
doing.
Well.
Is
your
your
back-end
source
doing
well
and
if
both
of
them
are
doing
well,
you
probably
are
only
running
10
queries
to
handle
200
queries
per
second,
you
just
need
to
make
sure
that
you're
not
queuing
up
your
queries.
You're
scaling
things
up
again,
use
the
scaling
dashboard
to
scale
things
up.
A
A
Just
make
sure
that
you
limit
everything
so
again,
there's
a
bunch
of
limits,
but
the
idea
is
you
keep
the
default
limits
super
low,
so
essentially
we're
on
cortex
as
a
hosted
Promethea
service,
and
if
you
sign
up
you
get
super
low
limits,
so
only
a
small
prometheus
will
be
able
to
send
data
to
it.
But
the
moment
you
hit
these
limits,
we
get
an
alert
and
then
we
reach
out
to
you.
We
understand
your
use
case
or,
and
we
will
tell
you,
okay,
it's
going
to
cost
you
this
much.
A
Are
you
ready
for
us
and
then
I?
So
then
I
increase
his
limits
or
increase
increase
the
tenant's
limits.
So
there's
a
bunch
of
limits
that
you
need
to
take
care
of.
One
is
what
is
the
samples
per
second
at
each
distributor
can
and
what
is
the
number
of
maximum
series
that
each
injector
should
have
per
user,
and
sometimes
somebody
has
a
histogram
that
sends
us,
like
300,
400
thousand
matrix
for
a
single
metric
name,
and
we
don't
want
that.
A
That's
usually
a
bad
practice
and
we
want
to
tell
our
user
say
we're
limiting
this,
because
this
is
a
bad
practice.
Are
you
sure?
So
that's
the
max
series
per
metric
and
then
there's
some
query
level
limits
that
you
need
to
use
or
when
you're
querying
you
need
to
make
sure
that
they're
not
querying
100
million
time
series
or,
like
5
million
times
the
views
over
30
days
of
data.
You
need
to
make
sure
that
they
are
not
moving
all
the
queries
for
everyone
else.
So
there's
a
bunch
of
query
limits.
A
We
will
see
that
so
again,
I
said
make
sure
that
every
single
user
has
a
default
low
limit,
but
make
sure
you
increase
that
thing,
so
cortex
is
able
to
limit
overwrite
the
limits
per
user
and
how
it
like
again,
we
use
JSON,
add
to
template
this
out
because
we
have
hundreds
of
users
and
we
don't
want
to
set
every
one
of
them
by
hand.
So
we
have
this
overrides
variable
that
you
can
just
add.
Okay,
initially,
everyone
is
a
small
user.
A
By
default
and
suddenly,
okay,
they
say:
okay,
we
have
two
Prometheus's
or
we
have
a
high
cardinality
Prometheus.
We
graduate
them
to
a
medium
user.
We
just
add
it
to
this
variable
and
there's
a
config
map.
That's
updated
that
cortex
takes
up.
This
is
all
at
runtime
and
you
can
see
that
you
can
make
them
a
medium
user,
but
also
give
them
specific
configuration.
For
example,
we
had
a
user
with
a
specific
use
case
that
required
the
user
to
send
metrics
with
40
label.
A
Names
like
our
default
is
20
for
a
good
reason,
and
then
there
was
this
user
trying
to
send
everything
in
a
cubed
et
sanitation
in
this
metric
for
a
good
reason,
so
we
reached
out
to
them
and
told
them.
We
are
dropping
this
metric
because
it's
too
many
labels
and
then
they
explained
that
to
us,
and
then
this
actually
like
you,
can
see
how
easy
it
is
a
it
is
for
us
operators
to
manage
this
complexity.
We're
like
ok,
he's
a
big
user,
but
he
access
extra
extra
limit
added.
A
That
is
overridden
and
stuff
like
that,
and
if
you
actually
look
at
the
config
map,
it
looks
like
this
for
user
29.
This
is
a
these.
Are
the
limits
and
stuff
like
that.
So
how
do
we
do
that
tearing?
So
we
have
these
variables
again?
All
of
this
is
open
source
in
the
JSON.
It
you
I
think
if
you
are
going
to
have
teams
or
tens
of
teams
using
cortex
in
your
organization,
make
sure
that
you
have
some
kind
of
manageable
config
management
like
you,
don't
want
to
spend
three
days
to
do
or
something
small.
A
So
we
have
this
tier
small
user,
medium
user,
big
user,
and
if
you
have
a
lot
of
money,
you're
a
super
user
yep
and
that
is
actually
not
the
end
of
the
things.
So
there's
a
bunch
of
query
limits.
How
many
of
you
attended?
Tom's
talk
yesterday
on
blazing
fast
from
ql,
nice?
Ok,
so
that's
a
significant
portion!
So
I
highlighted
something
like
this,
because
when
I
first
found
it,
it
totally
blew
my
mind
and
then
I
was
like.
Who
did
this
magic
and
turns
out?
A
It
was
me
long
back
long
while
back
so
there's
the
same
limit,
which
is
max
query
length,
744
odds
of
seven
days
and
max
query
lines:
500
500
days,
certain
two
different
deployments.
So
what
is
happening?
What
is
this
max
query
length
so
the
max
period?
So
when
you
get
a
range
query
or
when
you
add
a
prompt
ql
query,
it
is
the
length
of
the
data
that
you
look
at
so
that
if
it's
like
rate
three
minutes
or
like
rate
five
minutes
in
an
instant
query,
it's
five
minutes
of
data.
A
If
it's
red
five
minutes
over
30
days
in
a
range
query,
it's
five
minutes,
thirty
days
plus
five
minutes
of
data.
So
that's
the
max
query
length
and
you
want
to
make
sure
that
a
single
user
doesn't
query
too
much,
but
you
also
want
to
make
sure
hey
they're,
storing
13
months
of
data
in
context
they
want.
They
should
be
able
to
query
it
back
of
course.
So
how
do
you
do
this
so
turns
out?
These
are
two
different
query.
A
Workloads
arrange
request
that
takes
over
90
days
for
rate
five
minutes
is
actually
quite
simple
to
do,
while
rate
over
90
days
is
hard
to
do
because
in
a
single
window
you
need
to
look
at
all
the
90
days
of
data,
while
in
fact,
with
the
rate
with
the
first
example,
you
actually
only
look
at
five
minutes
and
then
five
minutes
and
then
five
minutes
and
then
five
minutes.
So
the
memory
requirement
the
CPU
requirements
are
quite
low,
but
for
the
other
one
is
super
high.
A
So
you
want
to
be
able
to
limit
your
your
users
to
only
do
maybe
like
30
over
30
days.
Give
me
the
increase
over
the
last
month,
so
that
I
can
do
some
building,
but
also
be
able
to
do
give
me
a
rate
five
minutes
over
the
last
13
months,
because
I'm
storing
that
in
months
of
data,
so
to
enable
that
we
have
the
free
front
end.
A
So
what
the
peripheral
ten
does
is
when
you
have
a
30-day
request,
it
splits
it
up
into
31
day
requests
and
runs
it
in
parallel,
so
that
the
queries
are
much
faster.
So
if
you
actually
compare
cortex,
v
or
vanilla,
Prometheus
cortex
is
much
faster
one,
because
we
do
caching
to
be
grossly
massively
paralyze
your
queries.
If
you
actually
send
a
year
long
query,
it's
actually
run
in
parallel
for
three
in
like
in
365
different
20
years.
A
Does
it
make
sense?
So
we
are
making
sure
the
limit
on
the
query
R
and
on
the
query
front
end
is
different.
That
allows
you
to
do
long
queries
but
not
super
intense
queries.
You
also
want
to
have
cardinality
limit
on
the
query.
Sometimes
people
send
like
5
million
time
series
and
they
want
to
query
all
of
them
back
together
and
that
will
in
your
career,
so
there's
this
cardinality
limit
and
max
samples
that
you
need
to
look
out
for
and
after
that
you
also
have
caching.
A
So
these
are
two
important
things
that
we
did
so
this
is
one
thing
is
because
all
the
recent
data
is
in
the
in
jester's.
You
also
need
to
query
the
in
jesters
and
the
query:
the
query
the
store
and
then
merge
those
two
two
things
in
the
in
the
queriers
and
we
actually
have
hundreds
of
QPS
and
every
single
query
was
actually
going
to
the
in
gesture.
A
That
means
the
in
gesture
was
locking
up
because
it
was
always
trying
to
select
the
data
and
stuff
like
that,
so
the
in
gesture
was
adding
a
lot
of
latency
to
the
query:
query
layer,
so
we
decided
okay.
But
a
lot
of
these
queries
don't
need
to
look
at
the
in
gesture,
because
the
interest
only
has
recent
data
and
these
queries
are
old
based
on
the
dashboards
because,
again
remember,
we
are
always
splitting
things
up
into
one
day.
A
Queries
so
if
somebody
opens
a
30
day
dashboard
and
has
five
panels,
the
first
five
queries
will
look
at
the
in
gesture
and
the
other
29
queries
will
not
look
at
the
in
jester's,
but
you
need
to
tell
the
courier's
how
much
data
the
in
gesture
has.
So
we
have
six
are
chunks,
so
we
set
it
to
12
hours
just
in
case,
so
anything
any
query:
that's
older
than
12
hours
will
not
hit
the
in
gesture.
A
The
other
thing
is
we
do
a
lot
of
index,
caching,
so
the
indexes
in
BigTable
Cassandra
DynamoDB,
you
name
it
and
it's
it's
a
lot
of
data.
We
do
a
lot
of
index.
Caching,
but
caching.
The
index
is
actually
quite
hard
because
you
cache
the
index
and
then
you
write
write
to
the
datastore
again,
which
means
the
index.
A
The
cache
is
invalidated,
so
we
used
to
do
a
lot
of
jumping
to
make
sure
that
we
can
make
the
cache
is
fresh
and
stuff,
but
once
we
did
that
we
still
couldn't
cash
for
more
than
fifteen
minutes.
So
if
the
same
query
is
run
in
in
less
than
fifteen
minutes,
it
used
to
hit
the
cache
for
the
index,
but
now
we
have
another
flag.
That
is
added.
That
says.
We
know
that
any
index
that
is
older
than
36
arts
is
stale
index
that
it's
never
going
to
be
written,
so
just
cache
that
forever.
A
So
if
you
actually
cache
that
and
reduce
the
loader
injustice,
your
queries
are
going
to
be
really
really
fast,
so
that
actually
made
a
lot
of
check
a
lot
of
difference,
so
yep
in
summary,
make
sure
you
have
the
right
alerts
and
dashboards
that
help
you
when
you
get
paged
and
don't
get
paged
for
every
single
thing
make
them
all
or
I
can
make
99%
of
the
alerts
warning
alerts.
We
paste
only
on
s
ellos.
We
pasted
on
like
some
very
critical
things
like
a
3d
master.
A
It
really
mark
master,
not
X,
like
it's
really
not
having
a
leader
for
five
minutes.
That
should
be
a
paging
alert,
so
just
make
sure
the
number
of
things
that
page
is
extremely
low,
but
there
are
things
that
you
should
look
at.
We
make
them
all
warning
alerts,
and
so
we
do
at
the
fauna
we
do
weekly
on
call
so
I'm
on
call
for
a
week
and
while
I'm
on
call
half
my
job
is
to
clean
up
clean
up
the
conflict.
A
Do
this
tech
tip
clean
up
and
I
keep
a
look
on
this
warning
alerts
and
make
sure
that
I
mean
they
don't
fire
anymore
again?
Have
proper
esta
loss
and
I
love
these
SLO
alerts,
because
once
we
move
to
s
ellos
everything
changed
because
we
have
stricter
celos.
Now
before
we
used
to
page
for
every
random
thing.
A
A
Jagr
is
amazing:
it
saved
my
life
before
Jaeger
people,
basically
guessing
looking
at
the
stars
and
guessing
okay.
This
might
be
slow
because
of
this
we
had
no
context,
but
now
and
then
we
used
to
deploy
something
add
a
bunch
of
matrix.
It
used
to
be
marginally
faster,
but
we,
it
didn't,
fix
anything
you're
like
okay.
A
We
are
at
point
three
percent
faster,
that's
nice,
but
now
with
Jaeger
I
know
exactly
what
is
slow
or
which
function
is
slow,
but
even
while
doing
that,
just
don't
add
yogurt
traces
everywhere,
like
have
them
at
higher
level
functions,
and
only
if
that
function
is
flow
like
if
you
can't
debug
things
then
go
to
a
lower
one,
because
again
or
like
traces
are
explained,
if
expensive,
just
don't
have
too
many
spans
or
every
single
function
called
in
there
and
then
don't
forget
to
limit
stuff.
This
is
a
multi
grand
environment
and
yeah.
A
Just
make
sure
everyone
has
the
right
limits
and
also
have
a
framework
to
make
sure
that
managing
this
limiting
figuration
is
easy
like
for
us.
We
have
tears
instead
of
individual
limits
for
each
tenant.
We
have
tears
and
we
have
like
hundreds
of
customers
that
yeah
I,
like
the
current
conflict
management
solution
that
we
have.
All
of
that
is
open
source
by
the
way.
So
you
can
use
it
directly
or
you
can
look
at
it
and
you
can
build
your
own.
A
A
So
the
idea
is
the
injector
has
six
hours
of
data
in
memory
and
when
a
new
in
gesture
comes
up,
it
transfers
the
six
hours
of
data
into
that
and
then
it
it
dies
and
stuff
so
that
you
don't
put
extraordinary
load
because
you
need
to
flush
a
million
series
or
four
million
series,
and
each
series
has
six
different
rows
that
need
to
be
so.
You
need
to
do
six
million
writes
if
you
want
to
like
kill
ingest,
so
this
process
is
kind
of
fragile
and
you
can
only
do
it
one
at
a
time.
A
A
Query
run
time.
So
the
okay,
that's
a
good
question,
so
we
don't
use
remote,
read
so
people
directly
query
cortex,
so
they
just
set
their
data
source
in
graph
honor
to
be
the
cortex
endpoint
that
we
give
them
with
authentication.
So
they'd,
never
there's
like
there's
no
Prometheus
on
the
query
path.
A
Okay,
so
good
question,
so
the
act
so
we
have
cortex
and
then
we
have
a
Prometheus
scraping
this
cortex
that's
doing
or
alerting,
and
then
we
have
a
global
alert
manager,
cluster
or
in
two
different,
maybe
three
or
four
different
clusters.
They
serve
this
indictment
at
each.
That
is
talking
to
each
other
that
so
we
have
a
global
or
minor
cluster,
and
this
Prometheus
is
sending
alerts
to
that
alert
manager.
And
that's
that's
paging.
Us.