►
From YouTube: Cortex: Multi-tenant Scalable Prometheus - Bryan Boreham, Weaveworks & Jacob Tlisi, Grafana Labs
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Cortex: Multi-tenant Scalable Prometheus - Bryan Boreham, Weaveworks & Jacob Tlisi, Grafana Labs
Cortex is a time-series data store based on Prometheus.
Cortex adds:
- Scalability: run across dozens of servers to handle millions of samples per second.
- Availability: if one server fails then work will be redirected to others.
- Multi-tenancy: store data from different groups or customers, segregated so a user from one tenant cannot see data from another.
- Durability: use cloud stores (such as S3) to reduce the chance of data loss.
This session will provide an overview of Cortex, an update on recent news from the project, and a deeper look at the recently added administrator feature to query across multiple tenants.
A
Hello
and
welcome
to
the
cortex
maintainer
session,
let's
introduce
ourselves.
First
of
all,
you
got
myself
brian
borum.
I
work
for
weaveworks,
I'm
a
maintainer
of
two
cncf
projects,
cortex
and
cni,
and
work
on
a
bunch
of
other
ones,
weave
net
etc.
One
of
my
hobbies
flying
and
I
I
put
dislikes
out
of
memory
errors.
I
hate
those
jacob.
B
A
So
we're
gonna
give
you
a
little
overview
of
the
cortex
project.
Some
news
bring
us
up
to
date.
Gonna
do
a
demo
of
a
recent
feature
and
talk
about
the
roadmap
forward
and
we've
left
plenty
of
time
for
questions.
So,
if
you're
watching
this
live
type
them
in
to
the
box
on
your
screen
and
we'll
get
to
them
at
the
end
of
the
video.
A
A
A
A
We
will
distribute
the
incoming
data
across
as
many
ingestor
processes
as
you
need.
So
you
can
run
a
handful
of
these.
You
could
run
20,
you
could
run
100
and
each
one
can
store
a
few
million
series
depending
on
how
big
your
machines
are
and
what
the
ingester
does
is
it.
It
captures
the
the
metrics
that
are
coming
in
compresses
them
and
eventually
sends
them
off
to
the
store
from
the
store.
We
have
a
querier
process
and
again
that
can
be
scaled.
A
A
We
spread
it
out
across
different
ingestors,
depending
on
where
it
came
from,
and
we
hold
it
tagged
with
that
identity
in
the
store,
so
it
always
has
its
identity,
it's
its
tenant
id,
and
that
means,
when
you
come
to
query
it,
you
will
get
the
data
just
for
that
one
tenant.
That's
why
we
we
stress
it's
really
good
for
those
situations
where
the
the
data
is
separate
amongst
different
customers,
different
teams.
They
just
want
to
see
their
own
data.
B
So
there
are
a
number
of
organizations
that
are
adopting
or
have
adopted
cortex.
You
can
see
from
this
list.
You
may
recognize
some
of
these
names
of
current
or
past
cortex
users.
The
ones
highlighted
in
yellow
are
new
users.
Since
the
previous
kubecon
in
terms
of
contributing
to
cortex,
we
have
a
hundred
people
who
have
contributed
in
the
past.
Hopefully
we
can
get
that
a
bit
higher.
So
we
can
say
we
have
hundreds
of
contributors.
B
We
have
seven
maintainers
across
three
different
companies
and
our
workflow
is
mainly
driven
through
github,
specifically
github
issues.
Github
prs
pretty
standard
there
in
terms
of
larger
features.
We
do
have
a
proposal
system
where
you
create
a
proposal
using
a
markdown
document
and
then
do
a
pr
against
the
main
github
cortex
repository.
B
B
We
are
looking
to
grow
our
community,
please
if
you
are
interested,
join
our
the
cncf
slack,
go
to
the
cortex
channel,
say
hi.
If
you
want
to
contribute
you,
don't
necessarily
have
to
start
with
go
code,
you
can
start
with
docs
or
the
helm
chart.
For
instance,
if
you
want
to
work
on
the
go
code
also
feel
free,
we're
looking
for
contributors
there
as
well.
B
If
you're
interested
in
learning
more
about
the
cortex
community,
I
highly
recommend
you
go
see.
Gotham's
talk
this
thursday
on
the
cortex
community
story
in
terms
of
project
news,
so
we've
implemented
a
number
of
items
on
our
roadmap,
specifically
per
tenant
retention.
So
this
is
basically
deciding
how
long
you
want
to
keep
data
points
for
each
specific
tenant
queries
over
multiple
tenants,
which
I'm
going
to
show
you
a
demo
of
a
bit
later
bulk
loading
data
from
prometheus.
B
So
taking
your
blocks
from
your
prometheus
node
and
uploading
them
to
an
object,
storage
system
for
cortex
to
use
and
then
horizontally,
scalable
alert
manager.
So
this
is
basically
the
alert
manager
component
of
cortex
is
now
scalable
in
a
horizontal
manner,
whereas
previously
horizontally
scaling,
it
didn't
actually
have
that
much
of
an
effect.
It
was
mainly
just
for
h.a
some
items
that
we
implemented
that
weren't
on
the
roadmap.
B
B
So
block
storage
is
now
live.
This
has
been
going
on
for
a
while
now
on
the
cortex
project,
where
we've
basically
migrated
the
way
you
store
data
from
using
a
nosql
backend
to
an
object,
storage
backend.
So
as
a
project
we
do.
We
now
recommend
block
storage
as
the
primary
way
of
using
cortex.
We
no
longer
recommend
the
chunks
back
end.
Why?
Well?
It
uses
the
prometheus
native
tsd
code
format,
so
it
standardizes
across
the
ecosystem
and
allows
us
to
reuse
code
from
prometheus
and
thanos.
B
Also
object.
Storage
is
generally
cheaper
or
easier
to
use
than
nosql.
It's
definitely
cheaper
from
commodity
offerings.
In
terms
of
running
on
prem
that
you
would
have
to
run
your
own
tco
calculations,
but
we
found
it
to
be
more
cost
effective,
broadly
and
then.
The
third
thing
we
should
mention
is
that
it
is
a
bit
faster
on
the
read
path,
especially
for
pretty
standard
rate
queries.
It
also
is
a
bit
more
expansive
in
terms
of
its
prompt,
ql
compatibility,
because
you
can
now
do
anonymous,
name
queries
so
some
next
steps.
B
There
are
some
limits
that
we
had
in
chunks
that
we
haven't
ported
over
to
blocks.
Yet
the
main
thing
I'm
thinking
of
is
the
number
of
actual
discrete
samples
per
quarry
being
able
to
limit
that
on
a
per
tenant
basis
was
pretty
useful
in
chunks.
We
can't
do
that
currently
in
blocks,
but
we
are
going
to
look
to
implementing
that
in
the
near
term.
B
The
other
major
limitation
with
blocks
is
that
it
isn't
as
good
at
retain
at
ingesting
data.
That's
old,
so
blocks
are
caught
on
a
two-hour
time
frame
and
if
you
go
beyond
that,
it's
a
bit
wishy-washy.
But
after
a
certain
point,
it's
very
difficult
to
actually
write
data
to
blocks.
You
have
to
upload
it
using
a
backfill
mechanism.
If
you
don't
get
it,
we
did
in
within
about
an
hour
and
a
half
of
when
it
was
first
created.
B
So
let's
talk
about
multi-tenant
queries.
This
feature
landed
in
1.70
cortex.
It
was
implemented
by
a
co-worker
of
mine,
christian
christian
simon.
So
if
you
are
interested
in
this
feature
be
sure
to
thank
him
so
the
way
this
works
is
we've
talked
about
how
tenants
are
basically
isolated
databases
within
cortex,
so
you
can
have
a
prom
ql
database
for
multiple
tenants
and
the
data
between
those
tenants
is
isolated
from
one
another.
This
allows
you
to
have
different
users
in
your
organization
have
different
sets
of
data
that
are
relevant
to
them.
B
B
So
this
is
my
grafana
instance:
let's
take
a
look
at
some
of
the
data
sources
currently
configured
to
it,
so
we
have
three
data
sources:
team,
brian
team,
jacob
and
then
team
brian
plus
team,
jacob,
underneath
the
hood
team.
Brian
is
basically
just
a
grafana
data
source,
pointing
at
a
cortex
instance
running
on
my
local,
the
slash
prometheus
endpoint.
So
this
denotes
a
prometheus
api
is
underneath
this
endpoint
and
a
custom
header
set
x,
scope
or
d.
This
is
how
a
tenant
is
referred
to
in
cortex.
B
B
So
let's
take
a
look
at
what
those
data
sources
look
like
check
out
the
up
metric,
which
you
may
be
familiar
with
if
you
use
prometheus,
so
we
have
three
series
in
the
team:
brian
data
source.
We
have
an
up
metric
for
the
prometheus
job,
the
cortex
job
and
the
minion
job
looks
like
they're
all
doing
fine,
let's
check
out
team
jacob
for
the
same
query.
B
So
we
have
two
series,
one
for
grafana
and
one
for
agent,
which
is
the
grafana
cloud
agent,
basically
just
a
scrape
and
right
version
of
prometheus
without
the
time
series
database
built.
Then.
So,
let's
take
a
look
at
this
team.
Brian
plus
team,
jacob
data
source
again
the
header
is
obfuscated,
but
let's
just
show
you
what
this
would
look
like.
So
it's
the
same
as
before:
team,
brian
and
team
jacob,
but
they're
separated
with
a
pipe
operator.
B
B
You
see
the
team
brian
series
here
and
the
team
jacob
series
here,
so
these
are
stored
in
separate
tenants,
but
we're
able
to
view
them
as
one
consolidated
query
again
the
way
one
of
the
ways
this
works
is
we
basically
take
the
tenant
id
and
turn
it
into
its
own
label
selector,
and
you
can
actually
use
this
label
selector
in
your
query.
So
if
we
were
to
do
count
up,
I
send
an
id.
B
We
would
see
that
team
brian
has
three
series
for
up
and
team.
Jacob
has
two.
As
you
can
see,
this
is
just
a
really
powerful
way
of
courting
across
multiple
tenants
and
if
you
want
to
expose
tenants
as
a
way
of
delivering
a
prometheus
database
by
isolating
data
for
your
users,
but
then
still
have
that
global
view.
This
lets
you
do
that
in
a
really
seamless
way.
B
B
So
I
have
this
go
processes
dashboard.
It
has
two
variables
up
at
the
top
tenant.
You
can
see
team,
brian
and
team,
jacob
and
job,
so
we
can
see
all
of
the
applications
we
saw
on
the
previous
screen.
This
is
a
bit
of
a
contrived
example.
Each
of
these
dashboards
is
an
aggregation
of
all
of
the
jobs
and
if
we
change
some
of
these
drop-down
values,
we'll
start
to
see
some
of
these
graphs
change
a
bit.
B
B
And
you
can
see
here
if
we
just
select
rafana
the
variables
change,
you
can
already
get
a
sense
that
what
multi-tenant
queries
allow
you
to
do
is
have
dashboards
that
are
specific
to
specific
teams
and
data
sources
that
are
useful
for
specific
teams,
but
then
still
get
that
global
view
across
the
board.
If
you
need
it
again,
it's
it's
definitely
an
organizational
tool
that
you
can
use
if
you're
interested
with
cortex.
B
So
now
that
we've
seen
this
demo,
let's
talk
about
some
challenges
to
the
future
of
the
project.
So
cortex
has
multiple
interconnected
services.
It
requires
a
significant
amount
of
tuning
and
balancing
and
the
tooling
is
a
bit
early
stage
or
focused
on
specific
use
cases.
For
instance
the
jsonnet
library
is
very
mature,
but
it
focuses
on
the
microservices
use
case.
B
Once
you
have
this
in
place,
there's
nothing
stopping
you
from
scaling
further
moving
towards
a
more
complex
deployment,
but
it's
just
getting
to
that
initial
easy
deployment,
something
we're
going
to
focus
on
in
the
near
future
in
terms
of
feature
development.
Some
things
on
the
roadmap
include
exemplars,
so
this
is
being
worked
on
in
upstream
prometheus.
It's
essentially
a
way
of
taking
observations
or
data
points
recorded
by
your
prometheus
instrumentation
and
associating
them
with
trace
ids.
B
So
if
you
have
a
high
latency
query
request:
that's
instrumented
in
logs
in
prometheus,
you
can
later
take
that
metric
and
then
look
up
an
actual
trace
id
of
a
high
latency
request.
The
other
thing
working
on
is
improved
scalability.
This
is
going
to
be
pretty
much
a
constant
with
the
project.
We
always
want
to
push
more
active
series,
higher
ingestion
rate,
higher
quarters
per
second,
it's
just
something
we
want
to
do
as
a
distributed
time
series
database,
then,
in
terms
of
tooling
and
packaging,
we
have
a
helm
chart.
B
We
have
a
jsonnet
library
fleshing
those
out
making
them
a
bit
more
powerful
and
expansive
and
encompassing
the
single
binary
use
case.
For
instance,
that's
something
we're
looking
to
do
just
make
those
those
libraries
a
bit
more
consumable
so
that
new
users
can
get
started
with
the
project
more
easily.
B
So
if
anyone
has
any
questions,
feel
free
to
ask
me
and
brian
would
be
happy
to
answer.
You
may
also
be
interested
in
joining
the
cortex
office
hours
this
tomorrow
at
11,
15
or
the
community
story
session
by
gotham
or
the
shuffle
sharing
session
by
tom,
but
hope
you
guys
enjoyed
the
talk
and
happy
to
hear
any
questions
you
might
have.