►
From YouTube: Centralized vs Decentralized Prometheus Scraping Architecture... - Rabun Kosar & Ales Koprivnikar
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Centralized vs Decentralized Prometheus Scraping Architecture with DoorDash - Rabun Kosar, Doordash & Ales Koprivnikar, Chronosphere
There are two primary approaches to scrape and collect metrics using Prometheus - using a centralized set of dedicated scrapers or decentralized scrapers that run as an agent. With centralized scraping, Prometheus is deployed as a central scraper to pull metrics from all discoverable endpoints and sometimes can be split across multiple centralized instances using a few different approaches. However, with a decentralized approach, Prometheus runs as an agent and in Kubernetes is deployed as a DaemonSet on each node in a cluster and only collects metrics from the node it runs on. Each model has pros and cons - especially when operating at large scale - which can make it difficult when deciding which model to use. In this session, Rabun and Ales will provide an overview of Prometheus metrics collection at DoorDash, where having highly reliable resources, easy endpoint discovery, and real-time insights is critical. They will share insights and best practices into DoorDash’s decision to implement a decentralized model by offering pros and cons of each approach. Leave with a better understanding of the “right” model for your use case(s).
A
Yeah
like,
as
already
said,
Rabun
and
I,
are
going
to
talk
about
decentralized
and
centralized,
scraping
architectures
and
I'm,
going
to
talk
about
a
little
bit
more
in
general
and
raboon
is
going
to
give
you
some
very
interesting
examples
of
how
they
do
the
doordash
work
for
chronosphere
as
part
of
the
solution.
Engineering
team
and
I've
been
in
the
observability
space
for
the
past
five
years,
really
exciting
how
this
space
is
really
developing
growing
and
what
kind
of
challenges
people
are
facing.
B
A
Excellent
thanks
so
going
forward.
Let's
you
know
starting
in
the
beginning,
if
you
look
at
metric
data
and
data,
and
how
do
you
collect
this
data
there's
if
we
talk
about
the
data
of
the
life
of
the
data
point
as
as
we
kind
of
deal
with
this
I
kind
of
threw
an
example
up
here
that
actually
uses
B's.
A
So
if
somebody
has
AP
phobia,
I
I
apologize,
but
they
actually
lend
themselves
really
well
as
an
example
for
how
are
we
dealing
with
these
things
in
in
real
life
right,
if
you
think
about
metric
data,
what's
what's
actually
important,
well
discovering
where
you're
actually
collecting
collecting
it
transporting
it
and
then
storing
it
now,
each
of
these
areas
has
kind
of
different
problems
that
you
have
to
deal
with
and
kind
of
on
the
on
on
the
side
of
Discovery
and
collection
right.
How
do
we
scale
this?
A
A
A
Now,
that's
why
we're
going
to
today
talk
about
how
does
the
collection
infrastructure
actually
work
and
what
kind
of
approaches
do
we
use
here?
So
obviously,
as
the
title
says,
centralized
versus
decentralized.
The
centralized
aspect
is
essentially
having
one
large
thing
that
runs
discovers
all
the
targets
that
it
needs
to
pick
up,
metrics
from
collect
them,
processes
them
and
then
ships
them
off
into
wherever
you
store
them.
Now
that
massive
strip
mining
piece
of
equipment
is
actually
a
good
example
of
it.
Why?
A
Because
it's
big
it's
fairly
expensive
it,
but
it's
only
one
right,
so
you
can
maintain
it
in
one
spot.
You
can
control
it
in
one
spot,
but
it
has
a
massive
downfall,
and
that
is
if
it
breaks
you're,
not
collecting
anything
which
can
be
really
problematic
right.
If,
since
we
rely
on
this
data
more
and
more
to
make
sure
that
we
reliably
deliver
something
to
our
consumers
in
the
end
now
the
right
hand
side
is,
is
the
different
approach,
right,
decentralized
approach?
A
A
They
do
their
thing,
they
collect
whatever
they
collect
and
they
bring
it
back
to
wherever
we're,
storing
the
the
resources
right
and
if
one
of
them
fails
and
we
need
to
replace
it,
that
process
is
usually
quick
and,
if
you
think
about
it
in
the
in
the
context
of
like
a
collection
agent
right,
if
it's
small,
if
it
crashes,
it's
going
to
come
back
up
in
a
couple
of
seconds,
start
doing
its
job
and
you're
Off
to
the
Races.
That
might
not
be
true
in
a
centralized
environment.
A
Obviously
now
that
obviously
also
presents
a
couple
of
challenges,
mostly
around
command
and
control.
So
how
do
you
actually
make
sure
that
that
whole
Fleet
of
things
is
doing
the
right
thing
and
and
make
sure
that
you
are
that
you're
understanding
something
goes
wrong
and
so
on
and
so
on?
Now,
obviously,
This
is
complicated.
A
Right
and
not
all
problems
are,
are
best
solved
in
the
same
way
and
there's
the
the
obvious
thing
you
think
about
is
well
what,
if
we
kind
of
do
both
well
Morpheus,
never
said
that
and
Neo
couldn't
pick
both
pills,
but
in
this
case
we
actually
can
do
something
along
those
lines
and
Rabun
actually
has
a
really
good
example
of
how
that
was
done
at
doordash,
so
I'll.
Let
him
present
that.
C
Before
we
start
I
know,
some
of
you
don't
know
doordash.
So
what
is
zordash
is
a
company
that
does
Last
Mile
Logistics
and
it
provides
the
infrastructure
to
support
local
Commerce.
Basically,
we
have
Merchants,
we
have
the
consumers
and
then
we
have
Dashers
that
are
connecting
them
so
deliveries
or
like.
D
B
C
Are
Cloud
native?
We
have
been
on
cloud
from
the
get-go
and
we
are
right
now
kubernetes
based.
We
have
multiple
clusters,
the
number
of
clusters
I
think
it's
now
over
20.
and
the
largest
one
it
has
between
1500
to
2
000
nodes,
depending
on
the
scaling
and
depending
on
the
traffic,
and
we
have
more
than
35
000
Parts
at
Peak,
and
that
means
we
have
a
lot
of
scraping
targets.
C
So
we
have
on
soundclouds
multiple
targets,
so
we
have
around
100K
scraping
Targets
in
a
particular
one,
so
we
didn't
start
with
Prometheus,
so
we
were
a
stats
D
shop.
For
too
long,
however,
we
had
some
problems
with
scaling,
the
Primitive,
the
stats,
the
infrastructure,
and
if
you
go
and
check
our
blog
post,
you
will
see
things
that
we
have
tried
to
scale
this
up.
To
a
certain
point,
and
after
that
point
we
said:
okay,
this
is
not
working,
so
we
need
a
new
paradigm
and
then
we
shifted
to
Prometheus.
C
I
should
say
that
was
a
tough
migration.
It
was
a
multi-year
migration.
It
had
a
lot
of
tricks,
but
the
nice
thing
about
that
was
we
learned
a
lot
about
how
to
use
Prometheus,
especially
distributed
versus,
centralized
and
right
now
we
are
over
10
million
metrics
PPS.
By
the
way
we
batch
them
30.
Second
batches.
C
We
feel
like
it's
a
good
number
and
we
have
more
than
I
think
10
000
alerts
in
real
time.
Our
teams,
they
love
metrics,
they
love
alerts.
So
it's
a
challenge
so
with
that.
So
this
is
what
we
have
right
now,
I'm
going
to
talk
about
just
metrics,
so
no
logging
or
tracing
so
most
of
our
infrastructure
lives
on
kubernetes
and
what
we
have
with
this
is
well
anyway.
Our
render,
as
you
can
see,
is
chronosphere
and
we
use
their
collector.
C
It's
an
agent
that
we
deploy
as
a
demon
set
in
our
kubernetes
cluster
so
that
they
can
do
the
cloud
native
Discovery
and
Metric
scraping
in
kubernetes
environment.
It's
pretty
well,
it
is
good,
but
that's
not
enough,
sometimes
because
of
like
whatever
reason.
First
of,
for
example,
our
traffic
team
believes
in
dedicated
traffic
and
machines.
So
we
have
some
ec2
instances
and
you
can
see
we
are
on.
E
C
Shop
again
and
for
them
we
don't
use
the
demon
set
approach,
we
deploy
them
as
Services,
more
or
less
a
similar
concept
here,
but
it's
a
different
concept
because
they
have
one
responsibility,
whereas
the
collector
in
the
Chronos
kubernetes
one
is
responsible
for
multiple
endpoints
because
of
the
Pod
deployment,
and
they
do
Discovery
real
time
for
The
Institute
one.
We
generally
use
a
file
based
Discovery,
it's
simple,
it's
convenient
and
that
works
well
for
a
such
thing.
C
So
what
we
do
is
we
have
a
separate
deployment
of
Chronicle
vector
responsible
for
this
and
we
use
intermediaries
such
as
custom
exporters
or
Stat
C,
for
like
touch
jobs
or
scripts.
In
the
end,
you
can
see
I
mentioned
Chronicle
letter
in
different
places.
They
are
all
different
types
of
deployments,
even
if
the
binary
is
the
same,
it
doesn't
matter.
We
configure
them
to
be
special
to
those
tasks,
so
these
are
the
models
that
we
have.
First
of
all,
we
have
a
demon
set.
C
Obviously
we
keep
the
configuration
for
the
demon
set
as
simple
as
possible.
We
want
this
to
be
generic.
We
want
this
to
cover
multiple
microservices,
and
what
we
do
on
top
is
just
add
some
rewrite
rules.
We
have
a
standard
set
of
labels
that
we
attach
to
all
of
those
metrics.
So
this
is
what
we
do
with
the
config
and
we
have
some
extra
Virat
rules
to
add
some
extra
metrics
and
that's
it
it's
it's
kept
simple
on
purpose
and
there's
also
the
sidecar
approach.
Why
would
we
need
this?
C
Because,
unfortunately,
some
services
are
just
too
noisy,
it
might
be
a
buggy
thing,
it
might
be
the
nature
of
the
service.
So
for
those
we
keep
a
sidecar
more
or
less.
We
have
a
special
container
attached
to
that
one,
the
within
the
same
pod,
just
for
the
purpose
of
scraping
that
particular
service.
C
The
good
thing
about
this
one
is:
you
can
allocate
resources
depending
on
that
particular
service
needs.
This
is
critical
because
I
will
discuss
the
Merit
of
this
one
in
the
next
one
and
also
the
custom
configuration
that
they
can
have.
They
can
say
something
like
hey
I
want
my
metrics
to
be
scraped
every
10
seconds
or
some
say
I.
Don't
need
30.
Second
resolution
I'm
fine
with
five
minutes
so
to
handle
those
instead
of
finding
tricks
in
the
demon
set.
We
use
it
in
the
Sidecar
and
also
there
is
a
Unix
service.
C
Like
I
mentioned
it
doesn't
do
kubernetes-based
Discovery,
it
has
its
own
specific
configurations
and
that's
it
and
the
last
one
is
the
deployment.
So
the
deployment
is.
We
have
a
separate
set
of
called
vectors
aimed
for
a
single
task,
and
in
this
case
the
example
is
the
one
that
I
just
gave.
First
of
all,
for
example,
get
ec2
metrics
from
AWS
and
expose
them
to
send
them
to
chronosphere,
and
these
are
these
have
evolved
over
time.
It
wasn't
like
designed
like
this.
C
We
had
problems,
we
discussed
some
of
the
solutions
and
right
now
it
works
better
than
I
would
expect.
So
when
we
were
making
those
choices,
we
had
some
important
parameters.
First
of
all,
when
you
use
a
demon
set,
the
resource
consumption
is
pretty
critical
because
you
are
deploying
the
same
resource
requirements
across
your
whole
feed.
C
C
C
We
also
do
enrichment.
We
add,
for
example,
a
couple
of
labels.
One
is
the
environment
variable
in
stats,
T
world.
This
was
a
chaos,
so
we
had
n
as
a
label
name
versus
environment.
We
had
prod
versus
production,
uppercase
lowercase,
all
those
different
combinations
and
it
prevented
us
from
building
standard
dashboards.
So
what
we
did
was
we
said:
okay,
no,
more
infra
labels.
C
Let
us
tag
those
for
you
and
then
you
can
use
them
as
if
you
are
creating
those
yourself
and
we
have
a
lot
of
rewrite
rules
in
different
places
for
various
reasons-
and
this
was
another
parameter
and
the
last
one
is
the
recording
group,
our
SRE
team,
that
they
love
this
because
they
keep
their
slos
SLO
tracking
using
the
recording
rules.
So
we
have
a
lot
of
them,
so
we
also
needed
something
that
would
scale
and
keep
up
with
the
ongoing
SLO
SLA
project.
C
So
with
this
we
have
both
of
the
both
of
the
world's
right.
So
we
have
the
distributed
one
versus
the
centralized
one.
So
obviously,
if
you
have
a
decentralized
architecture,
you
don't
have
a
single
point
of
failure,
and
this
saved
us
multiple
times.
We
there
is
always
something
there
is
always
a
bad
neighbor,
there's
always
a
bad
machine.
There
is
always
a
bad
configuration,
because
our
compute
team
also
relies
on
the
collectors
to
scale
the
cluster
and
up
and
down
it's
critical
that
we
should
be
able
to
survive.
So
it's
pretty
resilient.
C
It's
pretty
scalable
I'm,
saying
this
from
our
point
of
view:
I'm
a
client
of
the
chronosphere
and
from
our
point
of
view
we
just
ask
for
more
resources
and
we
get
it
which
is
nice,
it's
flexible,
like
I
mentioned,
we
can
have
different
combinations
of
those
and
because
they
are
local,
the
rewrite
rules
are
local.
There
is
negligible
performance
penalty
because
we
distribute
and
we
scale
them
differently
if
there's
a
need
and
the
last
one,
if
you
have
noisy
neighbors,
they
just
impact
that
particular
node,
not
the
whole
cluster.
C
However,
the
cons
resources
is
the
problem
in
decentralized
architecture,
because
you
have
a
demon
set
right.
So
you
need
to
make
sure
that
you
are
scaling
your
system
to
handle
your
worst
performance.
So
that
is
always
your
limiting
one.
So
you
go
up
up
until
you
feel
fine,
but
then
you
have
most
of
the
cluster,
not
utilizing
your
allocated
resources,
and
then
you
have
to
play
with
the
rules
you
do
over
subscribing.
You
do
all
those
tricks,
but
this
is
the
case.
C
B
C
A
critical
piece,
so
we
want
to
make
sure
that
it's
reliable,
but
it
means
you
have
to
let
it
deploy
really
slowly,
which
can
take
some
time
and
it
can
create
so
create
some
operational
burden
and
the
last
one
is
when
you
have
an
issue
when
it's
some
of
the
code
vectors
or
some
of
the
agents
misbehaving,
then
you
have
a
problem
because
then
you
have
to
find
in
the
dynamic
kubernetes
environment.
What's
going
on,
you
have
a
lot
of
logs
because
every
log
is
like
really
multiplied
by
thousands.
C
So
you
have
to
pinpoint
what's
going
on,
you
have
to
isolate
or
coordinate
some
of
the
nodes
to
see
what's
going
on
and
then
what
you
do
is
like
you're
looking
for
a
needle
in
a
haystack.
So
that's
that
was
the
original
problem
that
we
had
to
improvise
all
those
different
types
of
deployments
and
that's
what
we
ended
up
with
so.
C
These
are
my
conclusions
by
the
way,
so
our
experience
told
us
that
you
need
a
distributed
deployments
deployment
after
a
certain
scale
for
for
us
that
was
when
we
passed
this
1
million
metrics
per
second
mark,
because
at
that
point
we
started
seeing
some
loss.
Maybe
we
didn't
do
a
good
job
of
optimizing,
but
it
felt
like
we
needed
something
and
it
also
plays
nicely
with
kubernetes.
So
we
wanted
to
scale
kubernetes.
So
observability
is
really
critical
for
the
compute
team,
so
we
had
to
scale
better
than
what
they
asked
for.
C
It
can
have
its
own
issues,
so
you
need
to
improvise.
You
need
to
take
some
of
them
out,
put
in
a
single
deployment,
do
sidecar
model
or
whatever,
but
once
you
have
this
once
you
understand
and
reconfigure
your
system,
it's
pretty
stable.
You
don't
have
to
do
anything
you
just
let
deployments
go
through,
but
you
will
need
to
like
work
with
a
lot
of
exceptions
you
have
to
like
experience
and
see
by
the
way.
C
This
is
doordash
experience
right,
so
your
experience
can
be
a
little
different,
but
you
know
that
there
will
be
some
and
you
will
be
creating
a
lot
of
Helm
charts,
a
lot
of
changes
and
with
this
is
this
the
last
one
yeah
so
with
this.
That
concludes
our
presentation
and
thank
you
and
any
questions.
C
E
C
A
D
C
Yeah,
that's
a
good
question
and
I
think
on
Friday.
We
will
have
a
presentation
explaining
this,
but
I'm
gonna,
give
you
a
short
summary
on
on
Friday,
yeah
I.
Think
it's
Friday
yeah
Friday
morning,
but
the
short
summary
is
I.
Think
you
need
to
take
it
very
slowly.
You
need
to
First
understand
that
you're
Shifting,
the
Paradigm.
It's
not
just
pull
versus
push.
It's
also
like
you're
moving
around
from
percentiles
to
histograms
you're.
Moving
away
from
your
I
can
do
whatever
I
want
I'm
going
it's
going
to
push
the
metrics
and
it's
easy
to.
C
You
need
to
have
a
libraries
you
need
to
expose
them.
You
need
to
understand
your
observability,
so
what
we
did
was
we
started
with
a
well-known
set
of
services
that
had
strong,
leads
and
strong
engineering
and
we
use
them
as
our
better
customers.
We
worked
with
them
to
see
what
was
failing.
What
we
need
to
do
like.
Is
it
just
the
library
change?
C
Is
it
a
rewrite
because
some
teams
use
micrometer
and
they
just
switched
from
strategy
to
Primitives,
and
it
was
a
disaster
for
us.
The
metrics
lost
their
reliability,
so
we
had
to
re-improvise
and
say:
okay,
let's
use
the
native
libraries
instead
of
using
a
facade
like
micrometer,
and
then
you
need
to
understand
how
you
do
things
with
your
storage
systems.
Is
your
database
exposing
Prometheus?
If
not,
how
do
you
do
this?
So
all
those
stories
you
lay
down
one
by
one
and
you
attack
them.
The
the
biggest
problem
is
not
that
one.
C
Those
were
all
based
on
our
experiences
with
other
teams
and
we
had
to
improvise
so
a
lot
of
improvisation
and
in
the
end,
even
with
the
best
intention,
we
have
some
stats
team
metrics
and
we
do
a
lot
of
conversion.
Unfortunately,
that
means
that
metrics
are
not
reliable,
but
at
least
they're
better
than
having
no
visibility,
but
the
details
I
think
my
friends
will
be
talking
about
them
on
Friday.
B
C
They
didn't
like
the
metrics
at
all.
They
were
seeing
a
discrepancy
and
it
was
consistent.
So
we
knew
that
we
were
losing
metrics
and
then,
when
we
switched
over
to
the
distributed
model
and
the
Prometheus,
we
saw
that
we
were
losing
like
25
percent,
a
massive
thing.
So
this
is
the
this
is
the
reason
we
are
using
the
distributed
model.
So
we
want
to
make
sure
that
we
are
not
having
any
issues.
So
the
collector
itself
has
a
lot
of
metrics.
C
It's
exposing
its
own
errors,
Etc,
which
you
didn't
have
with
the
stats
D1,
because
it
was
already
too
late
when
the
metric
didn't
arrive
and
the
Agents
like
that's
the
didn't,
have
any
idea.
So
we
that
was
the
big
shift
and
that's
how
I
think
we
are
in
a
good
position.
We
don't
have
a
lot
of
issues
right
now.
It's
stable,
but
yeah,
which
starts
the
we
tried.
All
of
it.
We
tried
scaling
the
machines,
we
tried
increasing
the
cluster
size.