►
From YouTube: How Not to Scale Your Prometheus in Production? - Kush Trivedi & Nikola Collins, DevRev
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
How Not to Scale Your Prometheus in Production? - Kush Trivedi & Nikola Collins, DevRev
DevRev, The World’s First Developer CRM, has seen a 10X increase in the number of microservices during the past year. DevRev, since its inception, has adapted the cloud-native way of running its microservices; with recent migration to the service mesh architecture, DevRev has a strong appreciation for the cloud-native community. After migrating to service mesh and onboarding the customers to the public beta offering, we had seen a massive increase in the metrics ingestion, our observability costs, and new dashboards on top of the custom metrics. It was a matter of a few weeks when we realized that we needed more flexibility, customization, and control of our observability setup. This talk will guide you through the journey of our learnings while adapting and scaling the Prometheus, our learnings on various sort of Prometheus management strategies we went through, and how we efficiently controlled our costs without compromising the flexibility service-owners needed to manage and visualize their custom metrics.
A
B
C
So
nice,
well,
we
do
have
phase
healing
issues
and
Prometheus
going
into
crash
loopback
off
a
lot
of
times.
So
worry
not
so
we
hope
that
after
the
stockings,
you
would
have
some
fruitful
insights
that
how
not
to
scale
your
Prometheus
and
go
on
the
right
way.
So,
hey
everyone
good
afternoon,
I
amedi
and
he's
my
colleague
Nicola
Collins.
We
are
platform
engineers
at
devrip
and
today
we
plan
to
talk
you
through
some
of
the
mistakes
we
made
while
taking
shortcuts
with
her
Prometheus
deployment
so
that
you
don't
have
to
go
through
them.
B
Yep
so
yeah.
So
as
Kush
mentioned,
we
work
at
devrev
and
we're
on
the
platform
side
of
things
and
at
Dev
rev.
We
really
own
the
infrastructure
to
support
our
developer,
oriented
API
first
CRM
or
devcrm,
and
with
that
I
can
kind
of
talk
you
through
a
little
bit
of
at
least
what
our
topology
looks
like
right.
So
we
are
kubernetes.
We
are
primarily
eks
and
then
we
initially
started
out
with
some
particularly
small
nodes.
B
On
top
of
that,
we
use
Argo
CD
heavily
istio
for
service
mesh.
We
use
Kali
to
visualize
our
mesh,
and
then
we
use
grafana
initially
early
on
to
to
do
some
visualizations
of
some
internal
metrics.
So
on
top
of
that,
when
cush
and
I
started
this,
we
had
around
20
microservices
that
we
were
supporting
today
we're
up
to
approximately
70
and
then
on
top
of
that,
when
we
started
all
of
this
out,
we
were
primarily
data
dog
shop.
C
C
Up
until
this
point,
we
had
the
luxury
of
dumping
everything
to
the
dog
without
worrying
about
the
cost,
but
now
we
had
to
become
more
Discerning,
so
we
decided
to
go
ahead
with
the
hybrid
observity
model,
where
we
would
have
a
cloud
provider
as
well
as
n
cluster
absolute
setup.
So
that's
when
we
decided
to
move
some
of
our
opportunity
setup
concerned
around
our
platform
and
infrastructure
to
Prometheus,
and
since
we
were
very
new
to
Prometheus,
we
didn't
know
how
to
manage
it
very
well.
B
Yeah,
so
our
first
attempts
are
really
our
first
interaction
with
Prometheus
was
indirect.
It
was.
It
was
a
necessity
to
visualize
our
service
mesh
metrics
and
just
to
get
that
rolling.
We
simply
installed
the
example
Prometheus
yaml
that
came
with
our
istio
install,
but
we
are.
We
try
our
best
to
deploy
and
control
everything
out
of
Argo,
so
we
pretty
quickly
moved
to
the
Prometheus.
The
the
most
basic
Prometheus
chart,
Helm
chart
yeah.
B
So
one
of
the
things
that
we
immediately
noticed
and
were
informed
of
was
our
data
costs.
Our
data
dog
cost
essentially
doubled
overnight
as
soon
as
we
turned
on
these
mesh
metrics.
So
that
was
a
pretty
Stark
realization.
So
the
the
first
quick
answer
was
all
right:
let's
filter
those
out,
we
don't
really
our
app
teams,
our
developers,
don't
really
need
those
metrics
they're
more
for
us
to
internally
debug.
B
So
we
we
filtered
those
out
and
we
started
dumping
them
directly
into
our
local
Prometheus
instance
that
led
to
the
first
real
instability
actually
in
our
cluster.
Is
we
noticed
that
this
Prometheus
pod
just
kept
going
down
it
just
kept
going
down
so
initially
our
first.
My
first
question
or
I
asked
that
kush
was
just
turn
on
the
horizontal
pod,
Auto
scaler
and
everything
should
be
fine,
but
he
quickly
informed
me.
B
That's
not
really
an
answer,
so
the
next
thing
we
did
was
obviously
just
try
to
throw
more
memory
at
it
and
with
our
tiny
nodes.
We
capped
that
out
pretty
quickly
and
then
in
the
last
ditch
effort.
We
just
we
tried
the
vertical
pod
Auto
scaler,
Without,
Really,
investigating
or
diving
too
deeply,
but
that
actually
led
to
us
having
node
instability.
So
we
we
backed
that
out.
We
backed
that
out
almost
immediately
but
yeah.
C
But
then
things
started
changing
pretty
soon.
When
we
started
to
explore
candy
deployments
for
our
microservices,
we
were
already
using
istio
and
ago
CD
setup
and
guess
what
that's?
What
all
we
needed
to
start
our
migration
to
Candy
rollout
strategy?
We
quickly
installed
our
rollouts
and
started
exploring
subset-based
routing
via
Stu,
and
now
we
needed
an
n-cluster
obserability
solution
to
monitor
the
Matrix.
There
were
two
reasons
why
we
needed
an
interest
solution.
C
First
thing
for
latency
reasons:
we
didn't
want
to
send
all
those
kenry
Matrix
to
some
cloud
provider
and
fetch
them
every
five
seconds,
just
to
make
sure
that
how
can
repot
is
working
well
and
to
make
sure
that
we
don't
slow
down
our
deployment
pipelines.
We
needed
again
in-house
solution
which
could
provide
us
with
apis
and
we
can
query
our
Matrix
every
five
seconds
six
seconds.
Whatever
is
our
frequency.
B
B
We
realized
that
okay,
we
have
two
very
stable
Prometheus
installs,
but
we
have
again
a
very
unreliable
mesh
instance,
so
it
in
some
ways
it
worked
well.
At
least
we
minimized
the
blast
radius,
but
at
the
same
time
we
essentially
ended
up
with
the
same
problem,
and
that
is,
we
have
one
Prometheus
instance
that
every
day
we
look
at
it,
it
ran
into
another
of
memory
error
and
that
outer
memory
error
leads
to
a
crash
loop
back
off
and
then
that
crash
loop
back
off.
Well,
our
probes
are
timing
out.
C
How
do
you
fix
one
of
the
parameters
instead,
which
is
always
crashing
as
Nick
said,
are
too
thorough
from
each
is
what
stable
we
are
getting
cluster
Matrix?
We
are
getting
custom
Matrix,
but
again
the
mission
stands
for
super
unstable.
So
what
we
did
back
then,
was
we
just
deleted.
The
PVC
restarted
the
pause
and
boom
a
Prometheus
was
back
again
for
two
to
three
hours
until
it's
reached
the
limit
of
the
Matrix
and
started
going
to
crash
to
back
off
again
we
needed
to
fix
this
yeah.
C
C
Very
soon
we
started
using
vertical
Auto
scaler,
but
then
the
vertical
Auto
scalar
met
Prometheus
increased
its
limits,
which
in
turn
led
to
cubelet
going
to
crashing
State,
ultimately
bringing
a
nodes
down,
and
when
the
nodes
were
down,
some
of
our
critical
service
started
going
on,
because
Prometheus
was
scheduled
with
one
of
those
services.
Then
we
tried
isolation
to
limit
the
blast
radius.
We
started
scheduling
Prometheus
on
a
dedicated
node,
but
again
Prometheus
soon
breached
those
limits,
and
one
of
our
node
was
always
down
and
trust
me.
C
Nobody
likes
a
kubernetes
cluster
with
one
of
its
node
always
down.
We
started
pushing
more
resource
at
the
problem,
so
we
bumped
the
node
size,
and
now
we
were
at
16
gigs
of
memory
and
8
core
CPU.
During
that
time
we
might
say
that
Prometheus
was
pretty
stable,
but
up
until
that
we
started
querying
the
Matrix
from
past
48
hours,
past
36
hours
and
that's
when
the
query
started:
Prometheus
breezed
the
limits
again
and
things
started
going
to
om.
So
we
were
back
to
zero.
B
Yeah
so
now
we
start
exploring
really
what's
our
last
option
and
Kush
mentioned
this
operator
model,
so
this
Prometheus
operator
and
I
think
the
thought
was
at
least
on
my
side
is
well.
We
have
this
problem
with
instability,
but
if
we
can
recover
gracefully
without
manual
intervention,
we
can
worry
about
it
later,
and
so
we
just
blindly
again
a
blind
attempt
to
just
install
some
more
technology
in
the
hope
that
it
fixed
the
problem
and
we
actually
resulted
up
with
the
exact
same
thing.
B
So
it's
really
at
this
point
that
I
felt
we
tried
everything
and
I
also
very
much
appreciate
that
Kush
to
kind
of
took
it
upon
himself
so
yeah
and
before
we
get
into
that,
just
really
quick
again,
so
more
resources
doesn't
work.
We
can't
go
horizontal
isolating
it
stops
impacting
other
services
federating.
It
got
us
stable
where
we
didn't
have
issues
and
then
operator
didn't
help
us
particularly
much.
So
it's
at
this
point
that
kush
I
think
took
it
much
deeper
and
really
decided
to
own
the
problem.
C
So
what
do
you
do
when
nothing
works?
We
were
done
with
hit
and
trials.
We
were
done
with
bumping
from
random
GitHub
issues
about
Prometheus
going
to
crash
from
it
is
going
to
owe
him.
So
that's
when
we
started
to
holistically
dive
into
the
promises
documentation.
It's
working.
We
started
reading
a
lot
of
blog
posts
articles
and
we
came
to
know
that
there
were
few
things
which
are
very
fundamentally
Wrong
by
how
we
were
installing
and
managing
a
Prometheus
deployment.
C
So
we
first
went
to
the
brand
Brazil's
Prometheus
memory
calculator
where
we
gave
all
the
inputs
we
had
and
it
came
out
that
we
needed
33
gigs
of
memory
to
make
sure
our
Prometheus
runs
in
stable
state
well
coming
from
T3
micro
nodes
to
a
node
with
33
gigs
is
not
something
which
we
can
easily
afford
for
at
startup.
So
that's
when
we
started
looking
at
the
problem
more
briefly
and
we
found
out
that
our
scrape
interval
was
too
small.
C
We
were
scraping
everything
at
five
seconds,
which
was
too
frequent,
so
we
bummed
up
our
scale.
Scrape
interval
second
thing
was
that
we
had
three
million
Time
series.
Honestly,
I
don't
know
if
we
actually
needed
the
this
much
time
series.
That's
when
we
stumble
upon
the
blog
by
Thomas
degracinto
around
how
to
investigate
the
high
memory
usage
by
Prometheus.
C
We
use
the
tsdb
analyze
tool
to
identify
whatever
the
high
card
MRT
Matrix.
What
was
The
Matrix
what's
highest
number
of
labels,
and
then
we
use
the
grafana
Mimi
tool
where
we
came
to
know
that
what
all
Matrix
do?
We
only
need
to
make
sure
that
our
grafana
dashboards
are
working
correctly
and
we
are
getting
all
the
absolute
data
we
need.
C
So
we
started
dropping
all
the
news
Matrix
we
started
reducing
the
labels
and
things
started
becoming
more
and
more
stable,
and
then
we
stumble
upon
these
two
documentation
around
where
it
was
mentioned
that
if
any
of
your
call
is
coming
from
out
of
the
mesh
or
if
any
of
your
call
is
going
out
of
the
mesh
istio
by
default,
marks
the
destination
Service
as
the
host
header.
What
it
can
do
is
it
can
result
in
very
high
Cardinal
destination
service
Matrix.
So
then
we
disable
the
hosted
a
fallback
in
our
clusters
and
boom.
C
D
B
So
this
is
the
one
area
really
in
our
entire
infrastructure
that
we
can't
rely
on
auto
scaling,
I
would
say,
and
it's
not
that
we
exclusively
rely
on
it.
It's
just
this
is
a
much
more
manual
tuning
than
I
would
have
hoped
or
expected.
So
I
think
it's
on
cushinai
now
to
number
one.
We
have
to
really
understand
the
data
that
we're
ingesting
and
we
have
to
be
a
little
bit
more
proactive
with
our
strategy
as
we're
number
one.
A
one
day.
B
Retention
is
not
an
answer,
so
we
know
we're
going
to
have
to
bump
that
up
pretty
significantly
here,
but
at
least
we
have
a
Baseline
and
from
that
Baseline
we
can
have
some
type
of
you
know,
deterministic
scale
model.
So
on
top
of
that
we
have
a
Dev
team
that
is
pretty
spoiled
by
their
dashboards
in
a
single
place
and
all
of
their
metrics
and
everything.
B
But
what
we
really
can't
have
is
three
or
four
or
five
different
places
to
get
metrics,
so
that
opens
up
a
new,
a
new
issue
for
us
or
something
that
we're
going
to
definitely
have
to
tackle,
which
is
this
idea,
of
instance,
aggregation
and,
let's
say
more
long-term
retention
and
from
there
we've
done
very,
very
brief
research,
but
things
like
Thanos
cortex
and
three
yeah.
There
are
some
other
other
options
out
there,
but
that's
definitely
next
on
our
list
and
then
another
big
thing
for
us
is
this.
You
know
these.
B
These
metrics
are
particularly
important
to
our
developers
and
developers
are
our
customers
at
devrev,
so
we've
been
trying
to
figure
out.
What's
the
best
way
to
number
one
use
some
of
these
metrics
internally,
to
let
let's
say
kick
off
internal
tickets
or
issues
programmatically
feedback
into
the
platform,
and
then
another
thing
that
we
do
is
we
kind
of
have
the
opportunity
to
visualize
some
of
your
infrastructure?
B
B
E
Hi
guys
so
when
you
said
that
you
guys
were
aggregating,
all
the
data
in
the
Prometheus
was
your
Prometheus,
the
only
source
for
metrics
or
you
guys
were
aggregating
and
pushing
it
somewhere
else
like
when
I
said,
talked
about
the
retention
So
when
you
say
24
hours,
36
hours
or
like
whatever
that
time
frame
is
so
you
get.
Your
Prometheus
was
the
only
store
or
you
guys
were
still
pushing
it
elsewhere,
to
have
like
a
week
store
or
a
month
store
or
something
like.
B
That
purely
one
day
at
this
point,
so
we
have
not
yet
looked
past
that
and
that's
primarily
because
the
metrics
that
we
have
today
are
used
for
our
internal
debugging.
We
don't
really
have
that
use
case
and
at
the
moment
our
app
Business
metrics
are
Developer
important,
metrics
they're
in
datadog
and
and
that's
where
the
retention
becomes
at
the
moment
and
more
important
but
long
term.
We
have
to
be
able
to
answer
this.
D
C
Of
now,
our
data
is
still
Federation,
based
and
for
long
term.
We
are
planning
to
have
each
microservice
as
a
Federated
Prometheus,
which
will
again
be
aggregated
by
global
Prometheus,
which
by
then
we
could
have
a
Thanos
for
different
cluster
primitive,
which
can
integrate
around
multi-cluster
setup.
So
we
can
have
long
term
attention.