►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Intro: Cortex - Tom Wilkie, Grafana Labs & Bryan Boreham, Weaveworks
Cortex provides horizontally scalable, highly available, multi-tenant, long term storage for Prometheus metrics, and a horizontally scalable, Prometheus-compatible query API. Cortex allows users to deploy a centralised, globally aggregated view of all their Prometheus instances, storing data indefinitely. In this talk we will discuss a bit of Cortex's history, Cortex's architecture and how to get started with Cortex. Cortex is a CNCF sandbox project.
https://sched.co/MPhX
A
A
A
Hand
up
yeah,
I
I'm
familiar
with
cortex.
Ok,
ok,
so
be
people!
Ok!
So
that's
where
you
are
so
you
so
you're
you're,
very
much
in
the
right
room
for
the
intro
session.
I'm
gonna
walk
through
what
is
cortex.
What's
it
all
about
what
problem
is
it
solving
and
then
I'm
gonna
hand
over
to
tom
is
gonna?
Do
a
demo
try
we
have
time
for
questions,
I.
Think
that
just
to
sort
of
gauge
how
much
time
we
might
need
did
anyone
come
with
a
question
apart
from
what
is
cortex
or
ok?
A
A
B
A
A
A
A
It
is
highly
available,
it
has
long
term
storage
and
it
is
multi-tenant.
You
can
store
data
belonging
to
different
people
and
keep
them
separate
in
the
system.
It
is
a
CNC,
F,
sandbox
project
that
that
means
you
know
it's.
It's
kind
of
neutrally
hosted
the
some
of
the
companies
that
contribute
to
to
cortex
are
in
the
same
market.
So
that's
that's
kind
of
really
useful
that
that
it's
not
owned
by
one
of
those
companies
trying
to
try
to
dominate
one
another
one.
We
all
work
together
on
it.
A
We
all
collaborate
and
it
is
apache
licensed,
so
you
can
pretty
much
do
what
you
like
with
it.
Okay,
what
else
we
got
people
using
cortex,
some
of
the
logos
up
there,
we've
works
and
Gryphon
our
labs.
We
are
representing
some
of
the
some
of
the
well
aspinmaster,
oh
you're,
all
wearing
a
ribbon
with
our
logo
on
the
the
user
as
part
of
their
service
and
EA
the
the
games,
people
they.
They
use
it
internally
for
packing
a
lot
of
their
metrics
and
their
game
servers.
A
Just
you
know,
there's
a
whole
bunch
of
people
that
do
use
it,
but
didn't
want
to
give
us
permission
to
put
their
logo
on
the
screen,
but
anyway
enough
enough
people
using
it
projects
been
around
three
years,
just
over
now
yeah
just
over
three
years,
and
so
you
know
it's
got
a
got.
A
decent
amount
of
battle,
testing
in
production.
A
There's
a
little
thumbnail
on
this
screen,
but
I
can't
read
it
because
I'm
old,
my
eyes,
don't
work
well
enough,
so
I
have
to
I
have
to
turn
round
and
and
look
at
this
way.
So
this
this
is
kind
of
restating.
So
the
the
previous
definition
was
the
technical
definition.
This
is.
This
is
stating
it
from
the.
What
problem
is
it
trying
to
solve
point
of
view,
and
so
it
can
give
you
a
global
view
of
as
many
metrics
as
you
need
that
Prometheus
runs
in
a
single
process,
so
it
will.
A
It
will
top
out
at
some
point
on.
You
know,
however,
big
have
a
bigger
box.
You
can
find
to
run
it
on
and
you
also
you
may
have
more
than
one
place
that
you
want
to
get
metrics
from.
You
may
have
more
than
one
data
center
or
or
more
than
one
cloud
region,
or
you
know
you
may
have
multiple
places
so
bring
the
data
together
into
a
global
view,
no
gaps
in
the
charts.
A
You
know
if
from
a
theist
is
handling
everything
and
if
it
goes
down,
you
know
either
for
maintenance
or
crashes
for
some
reason
or
power
failure.
Anything
like
that
you're
going
to
get
a
gap
in
the
chart.
The
cortex
solves
that
problem.
We'll
talk
about.
Why
durable,
long-term
storage
we'll
talk
about
that
and
again
we'll
the
multiple
tenants,
so
you
can
have
multiple
effectively
multiple
separate
systems
running
in
one.
A
B
A
Yeah,
so
so
in
fact,
the
way
most
people
run
cortex
we
actually
run
Prometheus
as
well
as
cortex,
so
the
the
the
part
of
the
operation,
where
we're
collecting
metrics
that
that's
usually
called
the
scraping
operation
that
actually
that
actually
is
handled
by
a
Prometheus
server.
We
did
not
need
to
change
that
bit
at
all,
so
you
can
have
as
many
of
these
as
you
like,
and
they
will
send
the
data.
This
is
cortex.
This
is
our
amazing
logo,
we're
in
the
middle
of
the
screen.
A
They
will
send
the
data
over
an
API
called
the
remote
write
API.
So
this
is
a
there's,
an
API
where
Prometheus
handles
the
the
client
side
and
a
number
of
different
systems,
but
certainly
cortex
handled
the
server
end.
So
the
metrics
are
pushed
up
this
API.
Everyone
knows
that
Prometheus
is
pulled
only
right.
Aha,
we
fooled
you.
We
added
push
anyway,
the
the
and
then
brought
together
in
some
kind
of
view,
so
you
can
run
queries
and-
and
these
queries
can
cover
all
the
metrics.
A
B
B
So
like
it
to
me,
it
means
you
know:
you've
described
it
as
Prometheus
being
constrained
by
a
single
host.
You
can
go
out
to
Amazon
and
buy
a
host
with
I.
Imagine
probably
a
terabyte
of
RAM
and
run
a
very
large
Prometheus
instance
on
that
host.
But
cortex
could
take
10
machines
with
a
terabyte
of
RAM
and
be
ten
times
the
size
of
that
single
Prometheus.
It
can
scale
across
multiple
machines,
so
it
can
scale
horizontally.
B
You
refer
to
yeah
shouting
yeah,
so
the
technique
we
use
under
the
skin
to
do
this
is
we
use
a
distributed
hash
table
to
automatically
shard
between
you
don't
have
to
manually
decide
but
yeah.
So
the
horizontal
scaling
generally
means
you
can
add.
More
machines
and
the
cortex
cluster
could
get
bigger,
have
higher
throughput,
faster
queries
and
so
on
and.
A
Another
interesting
point
is
that
we
can
scale
because
everything
is
broken
down
to
into
microservices.
The
the
ingest
path
is
running,
one
set
of
processes
and
the
query
path
is
running
another
set
of
processes,
so
we
can
scale
those
independently.
You
know
if
you,
if
you,
depending
on
your
load,
if
you
are
ingesting
much
more
than
you
query,
you
can
run
more
in
jester's
and
if
you
are
querying
much
more
than
you
ingest,
then
you
can
run
a
lot
of
query
servers
and
just
you
know,
pick
your
own
adventure.
Okay,
but.
B
The
final
just
before
we
move
on,
thank
you
Tom,
one
of
the
Oh
one
of
the
nice
things.
One
of
the
limits
we
set
in
the
Prometheus
community
is
like
what
we
think
is
reasonable
to
run
in
a
single
machine,
and
so
normally,
if
so,
I
would
lost
the
slides
well,
I
have
to
normally.
If,
if
you
are
querying
more
than
a
few
hundred
thousand
time
series
in
a
single
query,
we're
going
to
say
like
you're,
your
query
is
bad.
You
need
to
think
about
pre
aggregation
with
recording.
Was
this
kind
of
thing?
B
There
we
go
in
in
cortex.
That
argument
falls
flat
because
you
could
have
mean
Gravano,
Labs,
I,
think
we've
got
fifteen
or
twenty
clusters
around
the
world
cube
Letty's
clusters
we've
got
at
least
fifteen
is
actually
more
like
30
or
40
different
Prometheus's,
sending
to
our
internal
cortex
cluster
and
so
any
argument
about
a
hundred
thousand
being
the
limit
for
Prometheus.
B
You
know
gets
x
times
thirty
when
you
apply
it
to
cortex,
for
instance,
so
we've
had
to
make
the
quiff
had
put
a
huge
amount
of
effort
into
the
query
path
to
make
it
scalable
so
that
you
can
do
the
same
query.
You
would
run
against
Prometheus.
You
can
run
it
against
cortex,
just
over
thirty
forty
times
the
data.
That
was
the
final
point
I
want
to
make,
and
it's
really
fast.
It's
mostly
really
fast.
We've
put
a
lot
of
effort
in
the
last
year
or
so
in
making
query
paths.
You
know
I.
A
A
Let's
move
on
to
the
the
next
kind
of
point.
I
said:
we
achieve
no
gaps
in
the
data,
even
though
machines
can
fail-
and
we
do
this
by
replicating
the
data.
So
the
the
data
comes
in
over
that
remote
write,
API
I
mentioned
earlier
and
it
gets
replicated.
You
can
pick
the
factor,
but
but
I
think
most
of
us
run
with
three,
and
so
what
that
means
is
one
of
these
machines
can
lose
power
or
something
like
that
and
restart,
and
you
know,
we've
still
got
the.
We
still
got
the
date
over
here.
A
In
fact,
two
of
them
can
die
and
we've
still
got
the
data.
The
the
data
does
live
in
memory
for
bit
as
we're
compressing
it
that's
one
of
the
features
of
a
good
time
series
databases
is
you
got
incredible
compression
in
the
data
from
the
the
raw
samples
come
in
like
a
8,
bytes,
para
value
and
plus
the
timestamp,
and
that
gets
crushed
down
to
a
little
over
a
byte
per
sample.
A
So
all
that's
that's
why
we
hold
the
data
in
memory
for
a
while,
and
if
that
machine
crashes
you
will
new,
you
will
lose
the
most
recent
data
and
that's
why
we
replicate
it
into
some
more
machines
and
all
of
that
happens
in
memory.
So
it's
really
fast
the
while
the
next
part
is
the
storage
which
I'll
talk
about
that.
A
So
we
we
did
not
want
to
write
a
storage
system,
the
actually,
when
we
started
writing
cortex
Tom
started
writing.
Cortex
the
the
Prometheus
storage
engine
was
was
quite
a
lot
worse.
They
made
that
engine
quite
a
lot
better
and
in
version
2,
but
either
way
it's
the
Prometheus
storage
engine
is
not
built
for
sharding,
it's
not
built
for
for
massive
scale,
and
we
did
not
want
to
write
one.
That
is
when
there
are
already
things
you
can
take
off
the
shelf,
which
run
which
are
designed
to
run
at
enormous
scale.
A
So
the
most
of
the
cortex
systems,
I,
look
after
run
against
DynamoDB,
most
of
the
ones
that
Tom
looks
after
run
against
BigTable.
It
also
talks
to
Cassandra
if
you
are
skilled
at
administering
Cassandra,
otherwise
don't
go
there.
Just
pay
because
so
much
easier
and
and
the
the
other
one
on
here,
the
bolt
DB.
We
use
that
too.
If
we
want
to
run
a
very
small
instance
like
on
your
laptop
just
writing
to
files
on
disk.
A
So
that's
that's
that
one's
not
built
for
enormous
scale,
but
cortex
has
multiple
backends
coded
into
it
and
you
you
basically
pick
one
when
you
could
figure
it
and
and
that's
where
your
data
goes.
All
right.
I
should
maybe
explain
why
is
s3
and
dynamo
and
GCS
and
and
BigTable.
So
the
the
table
type
of
storage
is
needed
for
the
index.
A
You
can
put
them
in
in
dynamo
as
well
or
BigTable
as
well
or
you
can
put
them
in
the
sort
of
blob
store,
s3
type
of
storage,
so
you
can
either
either
use
the
table
style
only
or
you
can
use
table
plus
blob.
There
is
not
an
option
to
use
blob
only
not
yet
not
yet,
which
is
a
little
bit
of
a
shame,
because
it's
it's
a
lot
cheaper.
That's
what.
B
I'm
working
on
actually
I
want
to
I
want
to
use
dynamodb
to
build
the
index
locally
and
then
flush
that
out
to
a
blob,
store,
yeah.
So
final
thing
you
mind
if
I
just
add
one
thing
here,
sure
just
one
more
thing,
just
one
more
thing
like
a
question
I
commonly
get
is:
if
we
use
DynamoDB
or
BigTable,
which
already
these
needs
really
scalable
key
value
stores
is
what
purpose
is
cortex
serving
right?
B
B
Is
you
know
you
can't
execute
prom
QL
queries
against
BigTable,
you
know,
BigTable
doesn't
have
a
prom
QL
API,
so
you
need
certain
things
to
be
able
to
execute
those
crews.
You
need
to
be
able
to
find
the
series
that
match
your
selectors
that
you've
done
in
Prometheus,
so
you
need
an
index.
You
know.
Bigtable
is
just
a
key
value
store
similar
word
DynamoDB,
so
you,
you
know,
cortex
adds
that
indexing
and
it
adds
that
D
amplification
of
Rights.
C
A
B
B
A
B
Have
you
considered
clique
house,
no,
not
for
any
reason
we
just
haven't
got
around
to
it
yet,
but
as
long
as
you've
got
this
kind
of
row
and
column
access
model
like
like
big
table
like
dynamodb
like
sandra,
they
all
have
a
very
similar
data
model.
As
long
as
you
can
support
that,
it
should
be
really
trivial
to
add
support
for
it,
yes
and
as
a
pure
yeah.
I
don't
know
enough
about
clique
house
to
to
really
give
a
good
answer.
There.
A
So
let's
talk
about
multi-tenant.
This
is
my
amazing
artistic
skill.
Here,
what
am
I
trying
to
show
I'm
trying
to
show
that
that
it
you
can
have
data
starting
from
different
domains
that
you
want
to
keep
separate
so
I
work
for
weave
works.
We
have
a
an
online
sass
product
called
weave
cloud.
You
can
sign
up
for
a
free
trial
on
the
internet
right
now
and
start
sending
your
metrics.
A
You
don't
want
your
metrics
to
be
visible
to
Tom,
or
you
know
the
guy
sign
next
to
you
or
whatever
you
want
ask
to
keep
them
all
separate.
So
that's
that's
multi-tenant
right!
That's
and
you
might.
You
might
have
that
in
another
situation.
You
might
have
different
parts
of
your
business
where
you
want
to
keep
those
metrics
separate
or
you
you
might
have
IT
metrics
versus
business.
Metrics
and
people
are
sensitive
about.
Who
can
see
what?
If
you
don't
have
that
problem,
they
ignore
this
slide.
A
You
know
have
a
little
sleep
to
yourself
for
a
minute,
but
basically,
basically
all
that
we
do
is.
Is
we
we
let
you
assign
each
of
these
different
tenants
or
domains
of
metrics
with
an
ID
and
I've
just
coloured
them
here,
and
then
we
we
hold
onto
that
ID
all
the
way
through
the
system
into
the
database
through
the
queries
and
when
you,
when
you
come
in
to
make
a
query,
you
identify
yourself
as
the
same
tenant
and
you
will
get
only
those
metrics.
A
You
will
have
no
information
whatsoever
about
any
other
tenant
when
you,
when
you
come
in
through
the
query,
API,
so
cortex
does
not
implement
authentication
passwords.
Anything
like
that.
Cortex
assumes
that
you
can
find
something
else
that
does
that
and-
and
that
might
be
some
kind
of
reverse
proxy,
like
an
engine
X
or
it
might
be
something
more
sophisticated.
We
don't
care.
A
A
Is
that
Donna
said
label
on
Prometheus?
No
this,
this
is
I,
think
you
know
if
you,
if
you
did
it
as
a
label,
then
that
would
require
that
all
your
queries
queried
on
that
label
and
what
we
want
to.
We
don't
want
to
make
it
up
to
the
person.
Writing
the
query
as
to
whether
they
get
the
room
they're
entitled
to
see.
B
One
of
the
one
of
the
tricks
we
use
for
our
billing
pipeline
we
use
cortex
for
billing,
is
we
actually
do
what
you've
just
described
and
promote
an
external
label
to
be
the
the
tenant
ID,
and
so
we've
written
a
little
a
little
proxy
that'll,
basically
take
a
label
and
promote
it
to
the
header.
You
know:
that's
like
a
hundred
lines
ago.
I
think
it
might
be
open
source
you.
So
you
can't
do
that
kind
of
thing
if
you
want,
but
it's
outside
of
the
scope
for
cortex.
A
You
know
you
might
need
that,
for
instance,
if
you're
running
a
billing
system,
you
might
need
someone's
credit
card
before
you,
you
create
a
user,
but
that's
out
of
scope
of
cortex.
The
cortex
just
sees
a
new
ID
and
says:
okay,
great
I'm,
storing
data
for
a
new
person,
so
that
is
pretty
much.
My
walk
through
those
are
the
problems
that
cortex
solves
and
a
little
bit
about
how
it
solves
them
and
I'm
gonna
hand
over
to
Tom.
Now
who
is
going
to
show
it
in
action?
When
do
we
finish?
When
do
we
finish?
B
A
A
The
those
things
are
different
varieties
of
open
source
cortex
is
fully
open
source,
for
instance,
influx
DB.
You
don't
get
the
clustering,
the
H
a
is
not
part
of
the
open
source
project.
Stackdriver.
None
of
it
is
open
source.
So
you
can.
You
can
pick
one
of
those
they
work
great.
The
the
unique
thing
as
far
as
we
know
about
cortex,
is
that
it's
open
source
thanassis
is
solving
the
same
problem
with
very
different
engineering
trade-offs,
and
we
probably
don't
have
time
we'll.
B
A
A
A
A
B
We
if,
when
we
run
a
like
a
multi
region
cluster
for
customers,
we
that's
how
we
set
up.
We
do
the
replication
at
the
index.
We
use
a
global
bucket
and
we
just
run
three
independent
cortex
clusters
and
master
elect
between
them
in
the
front
end.
But
a
lot
of
that's
kind
of
out
scope
of
cortex.
For
now
at
least
like
I'd
hoped
we
could
open
source
a
bunch
of
that
yeah.
You
can
run
mean
graph
on
a
horizontal
scales
as
well.
B
But
sorry
the
question
was
like:
can
you
run
in
a
horizontally,
scalable
graph
on
or
effectively
like
failover
graph
on
a
PEZ,
and
you
can
but
it's
kind
of
out
of
scope
for
this
tool
anyway
right.
So
my
main
aim
for
the
next
kind
of
five
minutes
is
to
show
you
how
easy
it
is
to
kind
of
experiment
with
cortex,
on
your
laptop
on
your
laptop
I
kind
of
call,
this
airplane
mode.
Hopefully
you'll
see
why
so
you
can
go
and
grab
cortex
just
go
build
it.
B
It's
broken
down
into
this
kind
of
modular
fashion,
I'm
running
a
single
process
single
binary
here,
but
you
can
take
the
same
binary
and
just
run
it
as
an
ingest
adjust
run
it
as
a
distributor,
just
run
it
as
a
querier
and
get
that
kind
of
independent
scaling
that
Brian
talked
about
anyway.
So
that's
running,
it's
listening
on
port
9000
and
9,
so
I'm
gonna
start
a
Prometheus
here.
I'm
gonna
show
you
examples:
Prometheus
yml
in
the
Prometheus
config
I've
got
this
line
here.
B
Saying
remote
write
to
my
local
host
and
everyone
see
that
I'm
stunningly
remote
write
to
localhost
9001.
It
should
be
nine
thousand
and
nine,
so
so
writing
to
9009
now
and
I'm.
Gonna
start
prometheus
and
I'm
also
just
deleting
the
old
data.
So
now
this
line
here
is
telling
us
that
Prometheus
is
pushing
to
that
end
point,
so
this
Prometheus
is
just
set
up
to
scrape
itself.
So
it's
just
gathering
some
metrics
about
his
own
functions
and
it's
just
pushing
water
cortex
as
we
can
see.
B
Dr.
ranga
fauna,
port
3000
I'm
running
gravano
in
docker
because
it
takes
ages
to
build
Ravana
and
now
I
need
to
figure
out
how
to
get
out
of
that
window.
It
doesn't
work
there.
We
go
alright,
so
that's
kind
of
giving
away
the
the
punchline
there.
But
if
we
go
localhost
3000
we
can
get
to
our
growth,
fauna
and
password
for
all.
The
founders
is
admin
and
admin,
and
no
one
was
changing
it
now.
B
What
I'll
show
is
we're
gonna,
add
a
data
source
to
grow
fauna
and
this
time
we're
just
going
to
add
the
Prometheus
for
now.
I
have
a
cheat
sheet
here.
This
is
in
the
readme
for
cortex
how
to
do
all
of
this
demo.
So
I'm
just
going
to
copy
this,
because
the
graph
on
is
running
in
docker
and
the
Prometheus
isn't
so
you
have
to
use
some
funny
tricks
and
we
save
it
and
it
works,
and
then
I'm
gonna
add,
whilst
we're
here
I'm
going
to
add
a
data
source
for
cortex,
which
is
just
Prometheus.
B
So
we're
going
to
call
this
one
cortex
and
it's
gonna
be
pretty
much.
The
same
thing
I
said:
cortex
was
listening
on
port
9009
and
there's
a
little
preface
on
the
or
a
lot
of
saving
test
networks.
So
in
the
Explorer
mode
we
can
see.
Does
anyone
know
a
good
Prometheus
query?
How
about
we
do
Prometheus,
TS
dB.
B
Series
nope
samples
there
we
go
and
we'll
just
see
roughly
what
the
rate
I
have
any
money
it
for
that
long.
So
we
can
see
that
the
Prometheus
is
scraping
or
walking
twenty
five
seconds
samples.
A
second
and
I've
disc
refiner,
is
talking
directly
to
the
Prometheus
and
sending
that
query
and
showing
you
the
result.
Now
we
can.
This
is
by
the
way.
This
is
a
new
feature
in
grow
fauna,
which
David
will
be
talking
about
like
on
the
last
day,
called
Explorer.
It's
pretty
cool
switch
to
cortex.
B
You
can
see
that
you
get
exactly
the
same
answer,
so
this
is
cortex.
101
I've
run
a
Prometheus
I've.
Had
it
send
data
to
the
cortex
I've
run
a
graph
Anna
I've.
Had
it
query
the
Prometheus
and
query
the
cortex
and
show
you
get
the
same
result.
So
that's
the
basic.
The
next
step.
You've
got
two
minutes
next
step.
I
want
to
show
you
is
I'm,
gonna
run
a
cluster
kill
them.
I
said.
B
That's
just
running
in
the
background
now
so
now,
I'm
gonna
run
a
small
cluster.
Just
to
show
you
that
the
only
thing
running
here
is
a
console.
Node
consoles
are
distribute
key
value
store
from
Hashi
Corp
and
we
use
that
to
store
the
state
for
the
cluster
so
which
in
jesters
are
members
which
tokens
they
own.
We're
going
to
a
lot
more
detail
about
it
in
the
deep
dive
and
you
can
see,
there's
agra
fauna.
You
should
be
able
to
see.
There
was
nothing
else.
B
So
now,
I'm
gonna
dock
a
run,
and
this
is
getting
a
bit
hairy.
But
the
commands
for
this
are
in
the
readme.
I'm
creating
a
container
called
console
one
cortex,
one,
I'm
attaching
it
to
a
special
network
so
that
it
can
find
the
grow
fauna
and
the
console
I'm
exposing
it
on
port
9001.
Please,
and
then
I'm
mapping
a
config
file
through,
so
that
it's
got
some
config.
B
It
can
use
and
then
I'm
running
the
cortex
docker
image,
I'm,
telling
it
to
use
a
config
file
and
I'm,
telling
it
to
use
console
to
storage,
State
and
I'm,
giving
it
log
level
debug,
because
the
demo
didn't
work
when
I
tried
this
morning,
but
it
does
now.
So
that's
one
I'm
going
to
run
a
second
one,
I'm
going
to
call
this
cortex
and
I'm
going
to
map
this
through
to
port
9000
and
to
now.
B
If
we
go
to
those
ports,
cortex
exposes
a
small,
really
really
basic
HTTP
HTML
one
admin
interface
that
can
help
you
debug
it,
and
we
use
this.
This
shows
you
basically
the
membership
of
the
core-tex
cluster.
You
see
those
two
cortex
nodes
have
automatically
joined,
have
automatically
distributed
the
ring
between
themselves
or
one
owns
49.8%
in
the
ring
and
the
other
one
who
owns
50.1%
of
the
ring
and
I'm
just
gonna
run
Prometheus
again
this
time,
I'm
gonna
point
Prometheus
at
port
9001.
B
Two
point
two,
nine
thousand
and
two,
and
it
should
be
sorry,
it
should
be
cortex
to
nine
thousand
and
nine
yes.
So
now
what
I've
done
is
you
can
run
the
query
again
you
can
see,
Prometheus
is
still
working,
prometheus
is
still
working.
It's
only
just
started,
there's
very
little
data
in
there
and
I've
set
cortex
a
Prometheus
is
writing
to
the
first
instance
of
cortex
and
the
graph
Anna
is
talking
to
the
second
instance
of
cortex.
B
Now,
if
these
two
things
haven't
clustered,
my
results
will
be
returning
empty
from
cortex
to
right,
because
they're
being
written
to
cortex
one
and
I'm
querying
cortex,
two
and
really
I'm
just
trying
to
show
you
here
that
clustering
is
real
and
it
really
works.
So
if
I
run
this
command
do
I
get
the
same
answer.
I
do
yeah.
Okay,
now
we
were
out
of
time.
I
really
want
the
third
demo
I
wanted
to
do
was
showing
you
that
we
can
run
three
of
them.