►
From YouTube: Stateless Collectors For Stateful Data: Scaling Prometheus As a Node Agent - Danny Clark, Google
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Stateless Collectors For Stateful Data: Scaling Prometheus As a Node Agent - Danny Clark, Google
Speakers: Danny Clark
prometheus-operator is the de facto standard for running Prometheus on Kubernetes. Yet, its configuration can be complicated and baroque, making it hard to know what is being scraped, or to properly enforce RBAC. Scaling also requires careful thought. However, there are an increasing number of ways to run Prometheus as “stateless”. How can we adopt this to solve these problems? This talk introduces an alternative, operator-based approach for running stateless Prometheus instances on Kubernetes by leveraging Prometheus as a node agent. This prompted rethinking how Prometheus configuration is done today, and led to new, simpler, and more opinionated CRDs. We will discuss trade-offs in the new configuration model and the challenges of running a fleet of node-agent Prometheuses at scale. The hope is this lowers the barrier to entry of managing Prometheus infrastructure, while still supporting features and access controls for enterprise users.
A
Name
is
Danny
Clark
I'm,
a
software
engineer
at
Google
working
on
cloud
monitoring,
specifically
helping
to
build
out
Google
Cloud
managed
service
for
Prometheus
Google
Cloud's
managed
Prometheus,
offering
in
general
I'm
an
avid
geek
for
observability
tools,
distributed
systems
and
all
things
Cloud
native,
and
you
can
reach
me
on
Twitter
and
GitHub
at
Pinto.
Hutch.
A
Here's
an
outline
of
what
we'll
be
going
over
today.
First
I'll
give
a
background
into
conventional
ways
to
scale
Prometheus.
Today,
then
I'll
go
into
the
trade-offs
that
we
were
making
when
we
were
analyzing.
Those
scenarios
when
setting
out
to
build
a
managed
service
I'll
then
go
into
the
finer
details
of
our
operator-based
approach
and
custom
resources
and
conclude
with
some
future
direction
of
the
project.
A
So
what
is
Prometheus
specifically
the
Prometheus
server?
It's
a
metrics-based
monitoring
and
alerting
Tool,
and
what
it's
really
good
at
what
it
was
built
to
do
was
to
be
deployed
as
a
single
process
running
as
a
single
server
running
alongside
your
workloads,
specifically
tailored
to
monitor,
live
numeric
metric
data
through
polling
or
scraping
and
by
live
I
mean
persistence
on
the
order
of
weeks,
as
opposed
to
months
or
years
of
data.
A
Larger
instances
can
be
scaled
pretty
well
via
fine-tuning,
but
what
Prometheus
is
not
at
least
out
of
the
box
is
scalable
long-term
storage,
I
like
to
think
of
Prometheus
more
like
a
traditional
postgres
database,
running
on
a
single
node
in
your
cluster,
as
opposed
to
something
like
Cassandra
running
on
a
fleet
of
nodes,
doing
talks
over
the
network,
and
at
this
point
it's
really
become
a
ubiquitous
for
metrics
infrastructure
in
kubernetes.
In
fact,
kubernetes
components
themselves
use
Prometheus
to
metrics
format,
for
their
monitoring.
A
So
how
do
we
deploy
Prometheus
on
kubernetes?
Today?
We
use
Prometheus
operators
that
de
facto
standard
it's
battle
tested.
Foundational
was
one
of
the
first
kubernetes
operators
and
how
most
people
tend
to
deploy
Prometheus
operators
using
the
cube
Prometheus
stack,
which
gives
you
a
Soup
To,
Nuts
monitoring
infrastructure
in
your
cluster,
specifically
Prometheus
operator,
AJ,
Prometheus
alert
manager,
some
grafana
dashboarding
common
metrics
exporters
like
Cube
State,
metrics,
node,
exporter
Etc,
as
well
as
some
recommended,
Prometheus,
recording
and
alerting
rules,
and
all
that
data.
A
A
A
A
A
How
do
I
do
this
with
Prometheus
operator
instead
of
one
custom
resource
I
now
have
two
Prometheus
a
using
the
service
monitor,
selector
I'm,
scraping
workloads,
one
and
two
and
Prometheus
B,
three
and
four
I'm,
also
giving
it
a
replica
count
of
two
to
demonstrate
some
I
applied
these
to
my
cluster.
The
operator
sees
those
and
reconciles
that
into
two
instances
of
Prometheus
each
configured
to
scrape
their
respective
workloads.
A
But
there's
a
problem
here
in
order
to
query
metric
data
that
I
care
about
I
need
to
know
which
Prometheus
server
a
given
metric
resides
on
now.
This
might
not
be
a
big
problem
if
we're
talking
about
two
Prometheus
servers,
but
when
it
grows
beyond
a
handful
of
servers,
this
quickly
becomes
cumbersome.
A
What
are
the
fixes
for
this?
Well,
we
could
Federate
and
a
federation
setup.
We
have
parent
Prometheus
servers
that
are
continuously
scraping.
Child
servers
in
a
tree-like
structure
against
their
Federate
endpoint
as
they're,
scraping
that
data
they're
collecting
it
and
aggregating
it
and
moving
it
up
to
the
next
level
of
the
tree.
A
A
A
A
The
downside
here
is
there's
currently
no
support
for
query
push
down,
so
larger
queries
could
be
potentially
pulling
in
gigs
of
data
over
the
network
where
you're
extra
susceptible
to
things
like
Network
failures,
maybe
some
Ingress
policies
that
prevent
you
from
doing
that.
So
again,
we
find
ourselves
in
a
situation
where
we
need
to
dedicate
some
sres
to
the
task
of
maintaining
this
infrastructure,
maybe
introducing
a
caching
layer
for
for
client-side
aggregation
or
something
like
that.
A
A
A
A
A
So
there's
a
fundamental
problem
with
all
these
approaches
and
the
need
to
rebalance
shards
and
what
I
mean
by
that
is
specifically
with
Federation
and
remote.
Read
where
each
Prometheus
server
is
maintaining
State
Once
A
Shard
contains
data,
it
stays
there
scaling
or
rebalancing.
Shards
is
a
manual
process
and
would
require
some
manual
orchestration,
maybe
some
back
filling
with
Thanos
and
remote
right,
where
we're
shipping
our
data
off
to
a
back
end,
whether
it
be
a
blob
store
or
a
database,
we're
still
susceptible
to
being
becoming
overwhelmed
at
scrape
time,
particularly
in
large
scale.
A
A
Additionally
at
Google
we
were
able
to
leverage
our
planet
scale,
time
series
database
and
Monarch,
which
had
the
capacity
we
needed
to
offer
the
service
specifically
being
able
to
serve
over
2
trillion
active
time
series
and
offer
long-term
retention
of
metrics.
So
we
could
stand
up
a
prom
ql
compatible
API
on
top
of
Monarch
and
solve
our
query
concerns,
but
we
still
needed
a
stable,
scalable,
metrics
ingestion
approach
to
offer
a
seamless
product
experience.
A
A
A
A
We
can
then
Mount
that
the
node
name
as
an
environment
variable,
for
example,
to
The
Container
manifest
which
is
what's
on
the
left,
and
then
we
could
use
that
in
our
Prometheus
configuration
now
in
Prometheus
to
do
down
selection
or
filtering
of
targets.
We
use
what
are
called
relabel
configs.
So
in
this
case
we
have
any
incoming
candidate
Target
that
has
the
node
name
that
matches
my
environment.
Variable
will
keep
everything
else,
will
implicitly
drop.
A
Okay,
so
that's
a
functioning
solution,
I'd
like
to
briefly
interlude
into
Prometheus
service
discovery
on
kubernetes
in
general,
so
in
order
to
discover
targets
to
scrape
Prometheus,
opens
up
watch
connections
against
the
kubernetes
API
server
for
various
resources,
whether
it
be
nodes,
pods,
endpoints
Etc,
so
whether
you've
known
it
or
not,
whenever
you've
installed
Prometheus
on
your
cluster
you've,
always
given
it
a
cluster
role.
That
looks
something
like
this
to
allow
it
to
do
that.
A
But
remember
we're
running
Prometheus
as
a
Daemon
set,
which
means
we
have
a
Prometheus
server,
opening
up
these
watch
connections
on
every
node
in
the
cluster,
the
number
of
watches
compounds
by
n
the
number
of
nodes
and
on
larger
clusters,
where
we
have
on
the
order
of
hundreds
or
thousands
of
nodes.
This
puts
considerable
strain
on
the
kubernetes
API
server.
A
A
So
this
deployment
model
is
fundamentally
different
than
anything
offered
by
Prometheus
operator
today.
I
do
want
to
point
out
that
there
are
discussions
on
GitHub
about
this
idea,
though,
when
we're
trying
to
enforce
the
Pod
role
and
node
field
selectors
in
our
custom
resources
to
handle
these
exact
scaling
challenges,
as
well
as
to
address
some
other
things,
specifically,
our
back
and
force
tenancy
and
just
having
a
simpler
configuration
service.
A
It's
closely
modeled
after
the
Prometheus
operator,
service,
monitor
or
pod
monitor,
with
some
differences
pod
monitorings
on
the
left.
You
can
see
it's
a
namespaced
resource
running
in
the
namespace
back
end,
configure
to
scrape
our
prom
example
workload.
The
analogous
service
monitor
on
the
right.
It
can
be
in
any
namespace
you
want,
so
we
can
stick
it
in
monitoring,
for
example,
and
then
use
a
namespace
selector
to
tell
it
go.
A
Look
in
that
namespace
for
targets
to
scrape
in
this
case
back
end
in
pod
monitoring
we
have
strict
namespace
tenancy,
so
a
custom
resource
in
one
namespace
cannot
scrape
workloads
in
another,
and
this
is
in
contrast
to
the
namespace
fields,
the
namespace
selector.
We
just
talked
about
this
aligns
well
with
kubernetes
our
back
enforcement
around
namespaces
as
a
cluster
administrator
I
have
full
control
over
not
only
who
can
create
and
configure
a
pod
monitoring
but
I'm
effectively,
controlling
which
service
accounts
can
scrape
which
workloads
this
has
some
nice
properties.
A
A
This
also
works
for
certain
exporters,
where
we
want
to
preserve
the
namespace
label
on
our
time
series
so,
for
example,
Cube
State,
metrics
and
again
this
is
a
dedicated
custom
resource.
So,
as
a
cluster
administrator,
I
can
allocate
only
certain
service
accounts
have
this
cluster-wide
scope
for
scraping.
A
A
A
Additionally,
we
found
a
service
level.
Spec
adds
a
layer
of
indirection
for
Target
Discovery.
So,
for
example,
if
I
have
a
service
monitor
selecting
service
workloads,
one
two
that
has
these
pods
behind
it,
and
then
someone
throws
workload
7
into
the
mix.
That
happens
to
be
selected
using
some
bad
luck
with
label
selectors.
We
can
have
a
hard
time
debugging,
what's
going
on
and
in
the
worst
case,
have
unintentional
metric
blow
up.
A
Okay,
so
that's
scraping,
Prometheus
also
offers
recording
and
alerting
rules.
Now
our
Prometheus
collectors
are
running
effectively
stateless
on
every
node
and
it
didn't
really
make
sense
to
put
the
burden
of
analyzing,
recording
and
alerting
rules
on
them.
So
we
wanted
to
deploy
a
separate
workload
whose
sole
concern
it
was
to
enforce,
recording
and
alerting
rules.
A
A
So
again,
rules
have
strict
namespace
tendency,
so
a
rules,
resource
and
namespace
a
cannot
query
or
write
to
time
series
in
namespace
b.
This
has
some
nice
properties.
So,
for
example,
we
prevent
expensive
queries,
matching
time
series
and
other
namespaces
as
well
as
preventing
collisions
when
writing
recording
rules.
A
Again,
the
tenancy
is
enforce
that
persistence
time
with
the
namespace
and
the
cluster
of
where
that
rules
and
resource
exists
and
mirroring
pod
monitoring.
We
recognize
there
are
cases
where
it
makes
sense
to
allow
users
to
scrape
Beyond
a
single
to
query
Beyond,
a
single
namespace,
so
we
offer
cluster
rules
as
a
way
to
do
that.
A
Finally,
it's
totally
possible
that
you
have
multiple
kubernetes
clusters
that
are
all
writing
metric
data
to
this
back
end.
If
you
want
to
be
able
to
query
against
all
those
time
series,
we
offer
yet
another
scope
in
global
rules
that
has
no
boundaries
on
what
it
can
query.
Your
right
to
again
the
common
theme
Here
is
giving
the
cluster
administrator
the
tools
they
need
to
control
which
service
accounts
have
what
level
of
scope
with
regard
to
metrics
in
your
cluster.
A
So
how
do
we
control
all
these
components?
What
we've
been
calling
managed
collection?
We
have
yet
another
custom
resource
called
the
operator
config.
Now
the
operator
config
runs
as
a
Singleton
in
your
cluster
and
controls
everything
from
collection.
So,
for
example,
if
you
want
to
filter
Data
before
sending
it
over
the
wire
or
compress
it,
you
could
do
that.
You
can
configure
Prometheus
alert
manager
endpoints
to
Route
alerts,
to
we
even
offer
a
managed
alert
manager
that
you
can
configure
through
this
resource.
A
A
A
A
Throughout
development,
we've
really
strive
to
minimize
our
configuration
surface,
I
guess,
as
developers
we're
always
trying
to
do
that,
but
with
with
pod
monitoring
and
cluster
pod
monitoring,
specifically,
we
provide
label
selectors
for
Target
Target
selection,
basic
scrape
configuration
and
very
limited
relabeling
capabilities.
Prometheus
for
labeling
is
very
powerful,
but
it's
really
easy
to
shoot
yourself
in
the
foot
and
cause
a
metrics
explosion
through
a
bad
relabeling
rule
or
even
have
some
collisions
and
drop
a
bunch
of
metrics.
A
We've
also
brought
broken
the
operator
out
from
the
reconcile
Loop
in
a
lot
of
respects
where,
instead
of
reconciling
the
entire
resource,
the
entire
Daemon
set
or
the
entire
deployment,
the
operator
is
just
concerned
with
reconciling
configuration,
we
let
kubernetes
reconcile
the
managed
resource
at
the
infrastructure
level.
So
as
a
cluster
administrator,
if
I
have
some
Custom
Security
profiles
or
volume
mounts
or
certain
quotas,
around
CPU
requests
and
limits,
I'm
free
to
alter
the
damage
that
myself
and
the
operator
will
not
overwrite
me.
A
A
A
It's
tempting
to
just
say:
hey
collectors
you
can
have
you
can
open
a
watch
connection
against
every
secret
in
the
cluster
and
then
retrieve
it
then,
but
then
we
find
ourselves
back
in
the
same
scenario
where
we're
overwhelming
the
API
server
with
all
these
watch
connections.
So
we're
working
on
ways
around
that
now.
A
Additionally,
Prometheus
provides
a
collection
side
status
apis,
basically
any
HTTP
API,
that's
not
prom
ql
falls
into
this
category
and
they're
really
helpful
for
debugging
right.
So,
if
you
need
to
know
like
what's
the
status
of
some
Targets
in
my
cluster
or
what
are
the
configured
rules
or
build
info
or
things
like
that,
these
apis
are
really
useful
for
users,
but
remember
since
Prometheus
is
deployed
as
a
Daemon
set.
This
status
is
effectively
sharded
throughout
the
cluster.
A
A
A
This
infrastructure
straddles
two
namespace
as
a
system
and
a
public
namespace
that
allow
us
to
constrict
the
rbac
necessary
for
our
components
to
run.
Users
interact
with
the
dedicated
public
namespace
for
configuration,
and
we
really
strive
to
keep
our
configuration
surfaced
as
simple
as
possible.
Oftentimes
trading
Simplicity
over
configurability
and
the
operator
itself
is
really
just
concerned
with
reconciling
metrics
configuration,
not
monitoring
infrastructure.
A
Yeah,
so
this
this
General,
so
the
question
was:
are
there
any
limitations
to
the
back
ends
that
we
support
so
the
code
that
it
runs
as
it
is
on
that
repository
just
goes
to
Google's
backend
to
Monarch.
However,
this
this
approach
would
work
in
theory
with
any
back
end.
It's
really
just
a
way
to
deploy
Prometheus
and
kubernetes
in
a
different
way.
So
that's
what
I
wanted
to
focus
on.
D
D
A
The
cost
is
not
zero
right,
I
mean
it's.
You
know
it's
kind
of
like
that.
Meme
like
side
cars,
sidecars
everywhere
this
you
know
it
will.
It
really
highly
depends
on
your
your
environment
and
the
workloads
and
the
metrics
that
you're
producing
so
Prometheus
will
in
kind
use
more
resources
if
you
have
more
rapid
metrics
right.
So
it's
it's
definitely
I
wouldn't
say
it's
using
more
resources
overall
than
a
standard
Prometheus
server,
because
that's
effectively
sharded
throughout
your
cluster
right,
so
you're
kind
of
dividing
up
those
resources
on
every
single
node
yeah.
E
A
E
A
Probably
the
cost
yeah
totally
yeah,
so
I
would
say,
I
think
yeah.
So
the
what
the
question
was,
if
you
have
a
small
cluster
or
a
small
workload,
is
deploying
this
really
necessary.
Does
it
kind
of
Overkill
in
a
lot
of
ways
and
I
would
say,
yeah
I
would
say
this?
Is
you
know
again
we
were
trying
to
offer
a
managed
service
where
we
didn't
have
to
think
about
the
size
of
the
cluster
it'll
just
work
and
the
trade-off
there
is,
you
know
it's
all
trade-offs
and
Engineering
the
trade-off.
F
F
A
Asking
if
you
can
use
grafana
in
this
in
this
configuration,
yeah
yeah?
Yes,
you
can
so
the
question
was:
how
can
you
use
grafana
with
this
setup
again,
since
all
of
our
metrics
are
persisted
centrally
in
a
global
back
end,
you
would
essentially
Point
your
grafana
at
the
URL
of
the
back
end
and
then
grafana
interacts
with
that.
The
the
infrastructure
I've
described
here
is
purely
for
collecting
data
and
and
also
doing
recording
and
learning
rules.
So
it's
grafana
is
sort
of
concerned
with
the
query
side
and.
A
So
yeah,
so
the
idea
here
is
that
you
can
write
to
any
any
back
end.
You
would
like
the
the
code
that,
as
it
sits
on
GitHub,
that
we've
been
writing
rights
to
Google
Cloud,
but
you
could
stand
up
the
same
exact
sort
of
infrastructure
pattern
to
write
to
any
back
end.
It's
really
just
leveraging
Prometheus
as
stateless
in
the
cluster.
G
A
Yeah
I
mean
so
the
question
was
like:
is
it
possible
for
a
single
Prometheus
server
on
a
given
node
to
essentially
become
overwhelmed,
because
there
could
be
a
ton
of
targets
on
that
node
right
and
that's
totally
possible?
I
mean
it's.
You
know
it's
again
about
like
mitigating
the
need
to
manually
scale.
H
A
A
F
C
It
it's
interesting
to
hear
a
lot
of
different
use
cases
with
Thanos
and
Prometheus,
and
you
know
your
new
service
You're
Building,
because
we
actually
use
Thanos
for
our
use
cases.
So
it's
cool
to
hear
about
that.
But
in
relation
to
that
actually
I
know
with
Thanos
with
the
rules
deployment,
it
can
have
performance
issues
when
you're
actually
trying
to
evaluate
the
alerting
rules
right.
Have
you
guys
run
to
that
problem
when
you're
not
evaluing
them
on
the
actual
Prometheus
themselves,
but
have
a
separate?
You
know,
obviously,
that
you're
running.