►
From YouTube: Cortex: Intro and Production Tips - Bryan Boreham, Grafana Labs & Alvin Lin, Amazon Web Services
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Cortex: Intro and Production Tips - Bryan Boreham, Grafana Labs & Alvin Lin, Amazon Web Services
Cortex is a time-series data store based on Prometheus. Cortex adds: - Scalability: run across dozens of servers to handle millions of samples per second. - Availability: if one server fails then work will be redirected to others. - Multi-tenancy: store data from different groups or customers, segregated so a user from one tenant cannot see data from another. - Durability: use cloud stores (such as S3) to reduce the chance of data loss. This session will provide an overview of Cortex, an update on recent news from the project, and a run-through of top 5 tips for running Cortex in production.
A
First
of
all,
a
little
introduction.
I
am
nowadays
a
distinguished
engineer
at
grafina
labs.
I
moved
from
wave
works.
You
may
if
you've
followed
cortex
for
a
while,
you
may
have
seen
me
there
acrofana
labs
as
well
as
cortex.
I
work
on
loki
and
tempo,
which
are
kind
of
projects
that
were
created
out
of
cortex
for
logging
and
tracing
respectively.
B
A
Thanks
for
that
alvin
and
welcome
to
the
team
now
to
look
at
what
we're
going
to
be
talking
about
in
this
session,
I'm
going
to
give
an
overview
of
cortex
first
of
all,
just
run
through
how
it
works,
because
I
know
a
lot
of
people
come
to
these
sessions.
Without
that
background,
I'm
going
to
go
through
some
news,
what's
been
happening
recently
on
the
project
and
then
alvin
is
going
to
run
through
how
aws
are
working
with
the
cortex
project,
I'm
going
to
touch
on
some
future
work
and
then
flip
over
to
live
q.
A.
A
A
Cortex
is
built
to
be
a
centralized
service,
highly
available,
very
scalable
and
and
also
durable.
It
will
hang
on
to
your
data
for
a
long
time.
It
can
be
used
for
many
teams
or
users.
It
was
originally
designed
to
create
a
prometheus
as
a
service,
but
you
can
you
can
use
it
within
an
organization
and
and
keep
multiple
people's
data
separate
so
who
uses
cortex?
Well
a
couple
of
the
big
prometheuses
of
user
names.
There
grafana
labs
weave
works
where
cortex
was
created.
A
So
what
does
cortex
do?
Well,
I've
drawn
a
very
big
picture
diagram
here
over
on
the
left.
You
have
prometheus,
which
is
doing
its
thing
of
of
gathering
metrics
what
we
call
sometimes
scraping
from
exporters
from
programs
from
machines
wherever
your
metrics
are
coming
from
and
then
sending
those.
So
this
is
cortex
is
the
cortex
logo
in
the
middle
which
gathers
collects
the
metric
stores
them,
and
then
you
have
some
kind
of
dashboard
over
on
the
right,
where
you're
querying
the
data
and
viewing
it.
A
We
have
prometheus
on
the
left
and
we
have
a
store
on
the
right.
So
this
is
going
to
be
a
something
like
an
aws
s3
or
a
google
cloud
storage,
some
kind
of
blob
store.
You
can
also
get
this
kind
of
thing
on
premise:
your
your
storage
vendor
emc
dell,
someone
like
that
will
have
usually
described
as
an
s3,
compatible
store
that
you
can
use
the
same
way
and
that's
what
that's?
What
makes
the
data
durable
and
and
pretty
low
cost
to
store
thousands
and
millions
of
metrics
over
time?
A
A
You
could
run
a
couple
for
resilience
or
you
could
run
a
hundred
to
handle
a
high
load
and
distributors
pass
the
data
on
to
ingester
processes
and
what
they
do
is
they
collect
up
the
data
and
they
make
what
we
call
a
block.
The
block
is
about
two
hours
of
data,
which
might
be,
I
think,
in
terms
of
a
gigabyte,
send
that
data
up
to
the
store
at
that
two-hour
mark
distributors
and
the
investors
use
a
key
value
store.
A
So
that's
something
like
xcd
or
console
to
decide
which
ingester
is
holding
handling,
which
data
and-
and
if
you
add
more
to
scale
up,
then
they
automatically
figure
out
how
the
the
re
we'll
go.
Sharding
shards
of
data,
different
slices
of
data
going
to
different
machines
is
handled
automatically
by
cortex
and
one
more
thing
on
the
ingest
path.
A
Once
we've
created
these
two
hour
blocks,
this
process
called
the
compactor,
will
download
a
set
of
blocks
and
create
a
12-hour
block
and
then
even
a
24-hour
block.
So
we
crunch
the
data
down.
We
remove
any
duplicates
from
the
data
and
we
create
these
bigger
blocks,
which
are
a
bit
more
efficient
to
work
with
over
time.
A
A
Those
are
passed
to
querier
processes
that
do
the
actual
work
of
of
fetching
the
data
and
interpreting
the
prom
ql
and
again
you
you
might
want
to
scale
that
out.
You
might
want
to
have
100
of
those
to
handle
your
load
and
there's
another
set
of
processes
called
the
store
gateway.
Those
are
the
ones
that
fetch
the
data
from
the
store
and
and
again
they
have
one
of
these
key
value
stores
to
decide
who's
doing.
A
A
Okay,
let's
move
to
the
project
update
since
the
last
time
we
did.
This
talk,
there's
been
four
cortex
releases
with
293
contributions
from
59
authors
working
at
22,
different
companies.
So
that's
great
round
of
applause.
Thank
you
to
all
our
contributors,
the
maintainer
team,
alvin
alvin,
lin
who's,
our
newest
maintainer
works
at
amazon,
web
services,
myself
gotham,
marco
peter
and
tom
wilkie.
The
original
creator
of
of
cortex
all
work
at
grafana
labs
and
jacob
lisey
has
now
moved
to
google.
A
The
releases
that
we've
done
just
quickly
going
through
the
highlights
1.8
gave
us
per
tenant
retention
limits.
1.9
gave
us
shuttle
sharding,
which
is
a
feature
to
as
you
as
you
get
into
hundreds
of
processes
to
limit
the
the
extent
to
which
failures
will
impact
any
one.
Customer
or
user
1.10
brought
us
exemplars,
which
we're
showing
an
example
of
here
these
little
blobs
that
if,
if
they
are
collected
by
your
program,
your
application
that
you're
monitoring
that
you
can
set
it
up.
A
B
B
B
A
B
B
B
B
Aside
from
providing
new
features,
we
were
also
keeping
an
eye
on
the
operation
aspect
of
cortex.
During
one
of
our
weekly
ops
review,
we
noticed
the
metrics
from
cortex
related
to
block
storage
operation.
Failure
was
spiking
up,
so
we
looked
deeper
and
found
that
the
matrix
was
unintentionally
incremented
for
expected,
http
400
error.
B
We
quickly
submit
the
pr
to
fa
fixed
and
runs
cortex
using
block
storage
because
it
is
cost
efficient,
but
cortex
does
not
support
serious
deletion
for
block
storage.
We
heard
many
feedbacks
for
many
for
many
customers
that
they
wanted
to
be
able
to
delete
series,
for
example,
they
might
wanted
to
decrease
their
cardinality.
B
B
B
B
A
I
want
to
mention.
We
had
two
cvs.
We
had
two
security
vulnerabilities
reported
and-
and
we
do
have
a
process
whereby
we
ask
people
to
disclose
those
privately
to
the
maintainer
team,
and
then
we
work
on
the
the
fix
in
private
and
get
that
out
ahead
of
describing
the
vulnerability
they're,
both
a
little
bit
similar
people
finding
ways
to
trick
cortex
into
sending
files
off
the
local
machine.
A
The
the
first
one
by
alert
manager
templates,
which
are
generally
available
feature
the
second
one.
Only
if
you
exposed
cortex
to
let
users
kind
of
forge
this
header,
this
http
header,
and
we
don't
recommend
that
we
always
recommend
you
run
cortex
behind
an
authorization
proxy
that
will
use
some
some
proper
security
credentials
to
figure
out
who's
calling
and
then
and
then
create
this
header
internal
to
your
cluster.
B
A
Alvin
another
proposal
that's
underway,
is
for
down
sampling,
meaning
if
you
originally
recorded
at
one
rate,
say
every
15
seconds.
Then
you
might
drop
that
after
a
while
to
every
five
minutes
to
have
fewer
samples
to
deal
with
makes
the
system
go
faster.
So
that's
underway
now
and
wanted
to
mention
something.
A
Now
question
that
always
gets
asked,
so
I
thought
I'd
put
it
in
the
video.
How
how
would
I
compare
cortex
and
thanos
they're,
both
cncf
projects
that
are
based
on
prometheus
they're,
both
solving,
certainly
one
problem
which
is,
which
is,
if
you
have
a
lot
of
prometheus
data?
How
can
you
handle
that,
and
we
share
a
lot
of
code,
the
blocks
that
I
talked
about
a
lot
of
the
code
for
that
came
from
thanos
and,
for
instance,
the
the
caching
the
query
front.
A
So
the
main
thing
I
would
stress
is
the
cortex
is
designed
to
build
a
large
centralized
system.
It
has,
for
instance,
this
this
automatic
sharding
and
shuffle
sharding
facilities
to
to
distribute
the
work
across
many
many
machines
it
also
built
for
multi-tenancy
cortex
has
has
had
multi-tenancy
added,
but
it's
it's
not
as
mature.
A
So
so,
if
that,
if
that
kind
of
sounds
like
you,
that
you
want
a
a
certainly
a
very
large
system,
because
if
you
don't
want
a
very
large
system,
you
probably
find
prometheus
will
handle
the
load
just
fine.
But
if
you,
if
you're,
going
beyond
what
what
a
large
prometheus
can
do-
and
you
want
to
build
one
of
these
centralized
within
your
organization
or
or
for
external
users-
that's
what
cortex
is
designed
for
thanos.