►
From YouTube: Intro to Scaling Prometheus with Cortex - Tom Wilkie, Grafana Labs & Ken Haines, Microsoft
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Intro to Scaling Prometheus with Cortex - Tom Wilkie, Grafana Labs & Ken Haines, Microsoft
Have many, disparate Prometheus instances scattered around your organisation? Want a single, centralised place to store and query all your metrics? Don’t want to manually shard your metrics across instance? Want faster queries and indefinite retention? If you answered “yes” to any of those questions, Cortex might be for you. Cortex is a horizontally scalable, highly available Prometheus-compatible TSDB. Cortex allows users to deploy a centralised, globally aggregated view of all their Prometheus instances, storing data indefinitely and scaling to hundred of millions of series. In this talk we will discuss a bit of Cortex's history, Cortex's architecture and how to get started with Cortex. Cortex is an Apache-licensed CNCF incubation project.
https://sched.co/ekHh
A
A
I'm
joined
today
by
by
ken
ken
was
one
of
the
first
people,
I
guess
other
than
myself
to
to
start
running
cortex
over
at
ea
three
years
ago.
Ken
is
now
at
microsoft,
where
he's
a
principal
software
engineer
and
he's
also,
he
was
one
of
the
first
cortex
maintainers
as
well
myself,
I'm
the
vp
product
at
grafana
labs,
I'm
a
prometheus
maintainer
one
of
the
original
authors
of
cortex.
I
also
started
a
project
called
loki,
which
is
our
horizontally
scalable
kind
of
prometheus
inspired
log
aggregation
system.
A
A
A
The
things
I
love
about
prometheus
are
that
this
pool
based
monitoring
system
that
uses
kind
of
dynamic
service
discovery
to
to
find
your
jobs.
I
really
love
this
because
it
just
kind
of
works.
For
me,
you
know,
especially
if
you're
running
your
jobs
in
a
kubernetes
cluster.
It
just
finds
them
it
scrapes
them.
It
gives
you
that
kind
of
monitoring
experience
you
really
you
really
want.
A
It's
got
this
incredibly
powerful
query
language
and
multi-dimensional
data
model,
which
makes
it
so
easy
to
do
kind
of
ad
hoc
analysis
and
and
compare
application
level,
metrics
and
and
system
level
metrics
together.
In
one
query,
it
also
allows
you
to
do
some
kind
of
really
powerful
stuff
in
alerting
like
error,
budget
and
slo
style
alerting
and
finally,
it's
open
source,
it's
incredibly
resource
efficient
and
it's
really
easy
to
operate
so
in
general.
Prometheus
is
more
than
enough
for
most
people
in
this
kind
of
model.
A
A
A
You
know,
especially
if
these
clusters
are
disconnected
this
pull-based
model,
that
prometheus
has
really
encourages
you
to
deploy
the
prometheus
server
co-located
with
your
jobs
like
next
to
your
jobs
in
the
same
cluster,
and
so
what
we
typically
see
is
when
people
start
to
service
customers
in
europe
and
and
in
asia.
They'll
deploy
their
application
in
a
in
a
region
there
and
and
deploy
some
prometheus
with
it,
and
at
this
point
right,
you've
got
your
grafana
pointing
at
one
of
them.
How
do
I
start?
You
know?
A
How
do
I
start
monitoring
my
other,
my
other
regions?
A
lot
of
people
will
just
deploy
multiple
grafanas
one
of
the
things
you
can
do,
I'm
sure
most
people
are
familiar.
You
can
actually
use
grafana
with
multiple
data
sources
and,
what's
even
cooler
is
you
can
sub
out
the
data
source?
You
can
template
out
the
data
source
in
your
dashboard
to
allow
you
to
switch
between
different
regions.
A
A
The
challenge
here
comes
when
I
want
to
look
at
my
global
metrics
in
a
single
query.
Let's
say
I
want
to
look
at
my
global
cpu
usage.
You
know,
maybe
I'm
doing
some
capacity
planning
and
I'm
deciding
where
I
should
deploy
more
more
resources,
it's
hard
to
do.
If
I
have
to
connect
to
three
different
prometheus
servers
to
to
do
that,
but
all
is
not
lost
and
this
can
actually
be
achieved.
You
can
actually
achieve
a
solution
to
this
without
cortex,
and
so
I'd
be
remiss.
A
If
I
didn't
mention
federation
federation
is
prometheus's
answer
to
how
to
build
kind
of
global
observability.
You
deploy
effectively
another
prometheus
server
on
top
of
your
existing
ones.
That
prometheus
server
goes
out
and
scrapes
the
the
edge
prometheus
servers-
and
you
know,
gives
you
that
central
place
with
all
that
data.
A
The
challenge
here
for
one
is
your:
your
global,
prometheus
server
has
to
be
able
to
connect
to
and
scrape
metrics
from
your
edge
prometheus
servers,
and
this
means,
if
you
don't
have
like
a
fully
connected
kind
of
set
of
vpn
tunnels,
you
probably
need
some
way
of
exposing
these
prometheus
servers
to
the
internet.
You
probably
need
some
way
of
securing
them
and
controlling
who
can
access
them.
A
A
So
the
solution
we
recommend
in
the
prometheus
space
is
that
actually
you
don't
propagate
the
raw
samples
to
the
central
global
federation
server,
but
you
only
propagate
pre-aggregated
data,
so
you
can
imagine
you
know
at
each
at
each
edge
node
you
might
have
metrics
per
pod,
but
then
you
might
only
propagate
metrics
per
service.
You
know
for
per
deployment
per
stateful
set
to
the
federated
server,
and
this
is
one
way
of
kind
of
controlling
the
cardinality
in
that
in
that
federated
server.
A
A
A
All
in
this
is
quite
a
powerful
technique
and
it's
very
simple
and
reliable,
but
it
can
be,
it
can
be
quite
tricky
to
master.
So
this
is
really
where
cortex
comes
in.
You
can
deploy
instead
of
a
global
federation
server,
you
can
deploy
a
central
cortex
cluster
and,
as
cortex
is
horizontally
scalable.
You
can
scale
that
cluster
up
to
take
all
your
raw
metrics,
having
basically
the
union
of
all
the
metrics
in
all
the
edge
prometheuses
all
in
a
single
place.
A
Cortex
itself
is
also
highly
available
through
replication
and
we've
basically
gone
to
a
lot
of
effort
to
accelerate
queries
and
to
to
make
this
system
functional.
For
this
use
case,
the
the
horizontal
scalability
in
the
cortex
cluster
also
allows
you
to
grow
and
shrink
the
cluster,
as
you
add,
or
remove
edge
locations,
and
generally
it's
it's
really
designed
for
that
kind
of
global
visibility
into
your
metrics.
A
A
A
We
add
a
lot
of
distributed
systems
glue
to
make
that
database
horizontally
scalable.
The
other
big
difference
with
cortex
is
it's
push
based,
so
you
don't
have
to
open
up
firewall
ports
and
worry
about
securing
every
single
one
of
your
edge
locations.
You
just
have
to
worry
about
securing
the
cortex
cluster
and
you
can
have
prometheus
natively
push
its
metrics
using
its
remote
right
system
directly
to
cortex.
A
Alongside
the
the
glue
we've
put
in
for
horizontal
scalability,
we
use
something
called
a
distributed.
Hash
table
we've
also
added
replication.
This
allows
cortex
to
tolerate
failures
in
the
nodes
without
ending
up
in
gaps
in
your
graphs.
This
also
means
it's
super
easy
to
kind
of
do
a
rolling
upgrade
with
zero
downtime
in
a
cortex
cluster
and
then
once
you've
got
all
your
data
in
one
location,
it's
pretty
it's
pretty
obvious
you're
going
to
want
to
store
it
and
you're
going
to
want
to
store
it
for
a
long
time.
A
Cortex
is
offloads
a
lot
of
the
long-term
storage
aspects
to
object
stores,
and
so
really
you
connect
it
to
an
s3
bucket
and
it
will
go
and
put
the
blocks
of
data
in
there
for
long-term
storage.
Prometheus
itself
is
totally
totally
useful
for
long-term
storage
and-
and
you
know,
as
long
as
you've
got
a
big
enough
disc
and
you
take
regular
backups.
You
can
store
data
for
years
in
in
prometheus,
but
in
cortex
we
care
about
a
lot
about
durability.
We
care
a
lot
about
replication.
A
Multiple
different
users
can
be
isolated
from
each
other,
with
their
own
data
sets
that
only
they
have
access
to,
and
we
go
to
a
lot
of
lengths
to
make
sure
that
not
only
is
the
data
isolated,
but
also
from
a
performance
perspective,
one
user
can't
run
big
queries
and
can't
kind
of
start
sucking
up
all
the
resources
of
a
single
cluster.
There's
a
lot
of
kind
of
sophisticated
quality.
A
Sorry
about
that,
there's
a
lot
of
sophisticated
quality
of
service,
a
lot
of
sophisticated
kind
of
limit
and
quota
management
all
baked
into
cortex,
so
that,
if
you're,
a
central
observability
team
within
a
large
organization,
you
can
have
different
teams
within
your
company
share
the
same
cortex
cluster
and,
as
we
said,
this
is
no
longer
a
sandbot
project.
Instead,
in
fact,
I
have
to
update
this
slide.
Cortex
is
now
a
cncf
incubation
project.
It's
apache
licensed
it's
open
source.
A
It's
got
a
vibrant,
maintainer
community
and
really,
I
would
encourage
you
at
this
stage
to
to
go
and
get
involved
and
try
it
out
a
bit
of
history.
I
started
the
cortex
project
with
a
chat
called
julius
who
who
started
the
prometheus
project.
We
started
it
almost
four
years
ago
now,
over
four
years
ago,
now
wow
we
initially
used
dynamodb
for
a
lot
of
the
storage
requirements.
A
After
that
I
added
support
for
google
bigtable,
and
that
was
about
when
ken
started
using
cortex
at
ea.
I
understand
those
still
on
dynamodb
shortly
after
that
we
added
support
for
cassandra
and
really
had
the
ability.
Now,
to
start
running
cortex
on
premise
and
out
of
the
clouds
and
really
one
of
the
things
I'm
very
proud
of
with
cortex
is
I
I
feel
like
we
got
the
right
path,
the
the
the
scalability
of
the
right
path
in
cortex.
A
So
we
put
a
lot
of
time
and
effort
into
parallelizing,
sharding
and
and
generally
finding
ways
to
horizontally
scale,
queries,
and
I
I
feel
like
we
I've
given
talks
at
kubecon
before
about
the
techniques
we've
used.
I'd
really
encourage
you
to
go
and
look
at
those,
and
I
feel
like
one
of
the
things
that
really
sets
cortex
apart
is
the
focus
for
the
past
two
or
three
years.
We've
had
on
query
performance
and
on
caching
and
on
parallelization
and
on
sharding
we
joined
the
cncf
sandbox
in
2018..
A
If
you
follow
our
instruction
on
the
website,
you'll
be
able
to
use
cortex,
you
know
in
half
an
hour,
we
did
our
first
1-0
release.
Well
only
hopefully
1-0
release
earlier
this
year,
and
that
really
for
me
marks
the
start
of
when
I
think
non-maintainers
of
cortex
can
really
start
to
lean
on
cortex
and
anger.
A
We've
been
seeing
a
lot
of
adoption
of
cortex
over
the
past
year
in
some
quite
large
companies,
and
very
recently,
a
month
ago,
we
launched,
what's
called
the
block
space
storage
engine
in
cortex.
So
this
is
the
same
storage
engine
that
thanos
uses
it's
the
same
code.
Marco,
who
did
this
in
cortex,
is
also
a
thanos
maintainer
and
and
really
marks
a
lot
of
collaboration
between
the
cortex
and
thanos
projects.
A
A
B
All
right,
let's
jump
into
a
demo.
What
we're
going
to
show
here
over
the
next
few
minutes
is
collecting
metrics
from
various
prometheus
instances
and
sending
them
to
a
central
cortex
installation
which,
when
then
we
can
do
global
aggregates
and
reports
with
via
grafana
all
right.
So
what
we've?
What
I've
set
up
on
my
local
docker
is.
I
have
a
three
node
cortex
installation
and
an
instance
of
grafana
running
on
a
docker
network.
B
That's
separate
from
this
prometheus
and
node
exporter
instance
they're
running
on
their
own
to
simulate
they're
running
in
a
particular
region,
while
your
cortex
installation
is
centrally
located
somewhere
else,
not
really,
particularly
where
just
that
the
prometheus
can
send
data
up
to
it.
So
our
cortex
ring
is
fully
up
and
running.
Let's
just
double
check
hit
refresh
a
couple
times
there,
so
we
get
recent
data,
so
we've
got
three
nodes
in
the
in
the
member
list
ring
and
they
are
all
active
and
have
their
tokens
registered.
B
B
B
B
The
main
pieces
here
that
are
important
for
the
demo
is
they're
all
pointing
to
this
url
to
remote,
write
data
to
so
everything
they
collect
they'll
push
up
to
that
url,
which
is
where
cortex
is
listing
to
write
the
data
into
its
storage
to
differentiate
each
region,
we're
adding
a
region
tag
to
each
one.
So
this
one's
from
region,
one
there's
one
that
says
region
two
three
and
we'll
see
that
on
the
grafana
dashboard
here
momentarily.
B
So
if
we
go
back
to
our
or
we
flip
over
to
grafana
here
to
global
overview,
I
created
this.
This
demo
board
to
just
showcase
just
some
some
sample
metrics
that
were
coming
in
from
the
node
exporters,
so
I've
been
collecting
region
one
for
quite
a
while,
and
we
just
extended
out
to
two
and
three,
so
those
lines
are
just
starting
to
show
up
now.
B
So
these
these.
This
is
one
one
query
on
one
data
source:
that's
just
highlighting
that
you
can
now
see
that
there
are
multiple
time
series
coming
in
from
the
different
regions.
You
can
see
all
that
in
one
view,
and
in
one
query,
so
this
also
means
you
can
do
aggregates
on
this
data
as
well,
which
is
what
the
total
is.
The
total
graph
shows
as
well.
B
It
should
it
should
have
been
a
good
inspiration
of
how
straightforward
it
is
to
take
any
one
of
your
prometheus
installations
from
any
of
your
environments
and
now
point
it
to
a
central
cortex
installation,
for
you
just
store
the
data,
query
the
data
and
aggregate
it
with
other
installations
from
a
single
dashboard.
Thank
you.