►
From YouTube: Deep Dive: Prometheus: Shaping Metric Monitoring in 2020 - Bartlomiej Plotka & Goutham
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Deep Dive: Prometheus: Shaping Metric Monitoring in 2020 - Bartlomiej Plotka, Red Hat & Goutham Veeramachaneni, Grafana Labs
With the 2.x version Prometheus fixed previous scalability limitations and greatly improved the performance. With the growing community the development velocity of the Prometheus increased, allowing us to improve things further. While the number of external integrations grows, Prometheus still pushes the boundaries of what is achievable with vanilla capabilities of out-of-box solution. During this talk Goutham and Bartek, Prometheus Maintainers, will walk through those advanced usage patterns of the Prometheus and new, useful features available in newest versions.
https://sched.co/cpB8
A
Hello,
everyone
welcome
to
the
prometheus
deep
dive.
We're
super
excited
to
show
you
how
to
use
prometheus
and
how
to
take
prometheus
usage
to
the
next
level
before
we
start
a
quick
intro
hi,
I'm
gautham,
I'm
a
software
engineer
at
grifana
labs
and
I
am
a
prometheus
and
cortex
maintainer.
I've
actually
started
contributing
to
prometheus
about
three
and
a
half
years.
Four
years
ago
and
later
I
started
working
on
cortex
to
provide
a
hosted
prometheus
service
between
prometheus
and
cortex.
B
Amazing,
okay,
my
name
is
bartok
vodka
and
I'm
an
engineer
working
in
the
monitoring
team
at
red
hat.
I
love
open
source
and
solving
problems.
I
am
part
of
the
promotions
properties
team
and
I'm
also
a
cauter
of
thanos
and
on
the
top
of
on
top
of
that,
you
might
know
me
from
the
newly
created
cnc
special
interest
group
observability,
where
we
focus
on
the
cloud
native
observability
topics
like
and
projects
like
you
know:
cortex
plantanos,
prometheus,
open,
telemetry
and
so
on.
A
So,
let's
start
with
a
typical
team:
in
this
case
the
team
in
which
kate
and
tom
are
a
part
of
so
they
run
an
api
service,
they've
successfully
built
and
launched
that
api
service
and
it
works
really
well,
but
a
couple
of
days
later,
they
noticed
that
the
users
are
looking
at
500
issues
or
500
errors
and
they're
trying
hard
to
debug.
What
is
throwing
these
500?
Is
it
the
load?
Balancer?
Is
it
their
own
application?
Is
the
database
locking
up?
Is
the
database
being
slow?
What
was
the
cause
of
it
to
debug
this?
A
A
In
this
case,
kate
actually
chose
prometheus
and
grafana
the
golden
stack
and
she
added
an
exporter
in
front
of
the
load
balancer
in
the
database,
exposing
all
the
metrics
of
the
load,
balancer
and
database,
and
she
also
instrumented
the
application
itself
to
expose
prometheus
metrics
prometheus
now
collects
all
this.
Data
and
kate
can
now
alert
on
500
errors
or
other
issues.
She
also
added
dashboarding
on
top
of
it.
So
whenever
there's
an
alert
or
an
issue,
they
can
directly
look
at
all
this
dashboard
with
all
the
information
to
directly
figure
out.
A
Okay,
the
database
is
having
issues,
which
is
why
we
are
throwing
500
address,
so
their
outages
didn't
stop,
but
whenever
there
was
an
outage,
they
could
very
quickly
figure
out
what
was
happening.
What
was
going
wrong
and
what
to
debug-
and
they
were
very
quick
at
fixing
all
these
issues
looking
at
their
success,
the
other
teams
also
started
using
prometheus,
as
is
the
prometheus
best
practice.
Each
team
was
maintaining
their
own
prometheus,
they
were
metering
their
own
grafana,
and
all
of
this
was
working
really
well.
A
Now
all
the
teams
in
the
organization
were
using
prometheus
and
as
the
usage
of
the
organization
and
the
application
grew,
they
started
deploying
to
several
data
centers
again,
as
is
the
prometheus
best
practice.
You
deploy
a
prometheus
poor
data
center,
the
next
slide
yeah.
This
is
because
prometheus
needs
to
be
close
to
the
application.
It's
monitoring.
A
So
in
this
case
they
were
deploying
to
us
central
one
and
eu
s2
and
whenever
there's
an
alert,
the
alert
actually
contains
which
prometheus
generated
this.
So,
if
they're
looking
at
an
alert
from
u.s
central
one,
they
directly
go
to
the
grafana
of
us
central
one.
Look
at
look
at
the
system
and
see
what
was
going
wrong
and
it
was
all
working
really
well
until.
B
Well,
you
cannot
use
query
api,
which
is
which
is
kind
of
the
the
first
thing
you
would.
You
would
try
to
use,
and
this
is
because
chromeql
elevation
is
made
on
the
leaf
node.
So
once
you
have
the
data
from
two
sources,
there
is
already
pro
chrome,
ql
evaluation
made.
So
you
would
need
to
have
another
layer
of
query
evaluation
to
be
made
to
adjust
and
to
add
additional
aggregate
like,
for
example,
sum
to
summarize
those
results
together
to
tell
you
the
overall
error
rate,
for
example.
B
Now
this
is
not
trivial
work,
because
there
are
lots
of
catchy
things
and
cabinets
like
additional
load
on
doing
from
ql
and
and
resolutions
and
steps.
So
all
of
this
is
not
easy
to
solve
with
using
using
query
api.
However,
the
prompt
use
allows
other
apis
to
solve
this
problem
and,
first
of
those
apis
is
the
federation
api.
Let's
call
it
in
in
this
kind
of
scenario.
B
And
so
even
you
create
a
scrape
configuration
for
that
and
this
kind
of
works
great
until
you
have
huge
or,
like
you
know,
bigger
amount
of
data
in
those
leaf
produces
because,
as
you
can
imagine,
if
you
are
scraping
and
replicating
all
of
the
data
into
the
global
promoters
or
like
yeah
into
the
global
chromato
server,
it
grows
in
the
same
with
the
same
kind
of
pace
as
the
lift
from
it
uses.
Because
you
replicate
all
of
this
data
and
suddenly
you
do
that.
B
B
You
should
only
use
federation
for
only
only
for
subset
of
your
data
and
the
easiest
way
to
do
that
is
to
essentially
create
recording
rules
for
the
things
that
you
want
to
have
on
the
global
level,
and
also
you
can.
You
know
things
of
that
reduce
the
cardinality
of
the
recorded
data,
because
you
can
sum
aggregate
across
those
and
you
configure
federation
to
only
federate
to
only
scrape
the
rules
by
the
parameter
of
matcher
and
well.
B
B
The
third
option
is
really
similar
to
the
query
api,
where
you
just
provide
a
query
that
that
has
to
be
evaluated
for
your
answer.
Instead
of
that,
you
are
going
deeper
and
trying
to
access
the
actual
samples
stored
in
the
database
of
properties,
and
this
is
the
api
that
allows
that
is
called
remote
read.
So
during
that
api
you
are
using
a
slight
different
payload,
it's
not
a
json.
It's
protobuf
is
actually
should
be
familiar
to
you.
B
That
is
tunnels
and
essentially
thanos
allows
you
to
add
a
side
card
which
probably
to
use
that
using
that's
just
using
this
reit
protocol
and
then
expose
it
into
the
grpc
that
prompted
thanos
is
using,
and
then
you
essentially
allow
yourself
to
create
and
to
deploy
and
global
query
component,
which
does
the
pronquial
evaluation
on
the
global
level
having
the
data
from
each
promote
users
separately.
So,
and
this
is
how
you
can
essentially
transparently
have
the
global
view
without
recording
rules
and
without
replicating
all
of
your
data
multiple
times.
B
However,
you
know
there
are
other
when
we
are
thinking
about.
You
know
with
bigger
adoption
and
more
users.
So
at
some
point,
kate
was
kind
of
annoyed
that
she
has
very
short
metric
retention.
Only
you
know
a
couple
of
weeks,
so
she
thought
about
hey
what
about
maybe
longer
one,
maybe
years
of
data,
so
I
can
analyze
my
my
data,
maybe
also
some
teams
have
the
policy
to
store
the
metrics
for
whatever
was
like
life
period
of
your
service.
B
There
is
a
misconception
that
promoters
is
not
suitable
for
long-term
storage
and
you
have
to
deploy
some
external
systems
and
and
have
some
integration.
That's
not
always
the
case,
because,
especially
from
prometus
2
and
the
recent
versions
of
promote
use,
we
made
sure
that
the
older
data
that
you
store
actually
adds
marginal
resources
when
they
are
not
used.
So
whenever
you
are
just
querying
short
period
of
times
and
and
mostly
from
from
our
experience,
people
are
queries.
You
know
the
very
fresh
data,
those
all
their
data
even
for
years.
B
B
However,
yes,
there
are
some
trade-offs.
After
all,
prometheus
was
mainly
focused
on
the
monitoring
for
resin
and
data
and
alerting.
So
there
are
some
caveats.
One
of
it
is
that
it's
super
hard
to
efficiently
or
like
effectively
plan
capacity
of
your
disks,
because
there
are
lots
of
unpredictable
spikes.
It's
hard
to
sometimes
control
cardinality
of
the
data
you're
ingesting.
So
it's
not
the
easy
task.
After
all,
the
second
problem
might
be
backup.
B
You
know
if
you
have,
those
disk
hardware
can
fail,
and
you
have
to
have
some
backup
plan
and
operational
kind
of
you
know,
structure
of
it,
some
scripts,
automation
and
it's
not
always
the
easiest
way,
especially
on
bare
metal
and
last,
but
not
the
least.
Well.
There
is
no
native
down
sampling
on
prometheus
side,
which
means
that
the
large
range
queries
like,
for
example,
for
two
years
for
two
years,
will
fetch
all
those
samples
into
the
prom
ql
engine
and
fromql
has
to
run
for
all
those
samples
to
calculate
your
response.
B
A
Cool
so
now
in
the
organization
every
team
is
using
prometheus
and
now
more
and
more
users
want
to
use
prometheus
and
there
are
more
use
cases
coming
up.
For
example,
let's
say
the
marketing
team
comes
to
kate
and
us
asks
if
they
could
use
the
data
in
their
data
lake
or
on
another
database,
or
maybe
some
someone
wants
to
use
the
prometheus
data
and
store
it
in
a
replicated
or
distributed
database.
That
was
that
is
much
better
at
storing
long-term
long-term
storage.
A
How
do
you
send
all
of
this
data
to
a
different
database?
How
do
you
do
this
replication
so
prometheus?
Has
a
protocol
called
remote
right?
It's
an
extremely
simple
protocol.
So
in
your
prometheus
configuration
you
just
add
of
stamps
are
called
remote
right.
You
name
the
remote
or
remote
endpoint
you.
If
you
have
any
authentication
on
it,
you
specify
the
authentication
and
then
prometheus
will
start
sending
every
single
sample
its
create
to
this
remote
endpoint.
A
Now,
typically,
your
prometheus
will
scrape
millions
of
samples
a
second,
and
sometimes
you
only
want
to
send
a
subset
of
the
data
to
the
remote
endpoint
and
doing
that
is
also
extremely
use.
Easy
you
can
use
the
right
relabel
configs
to
kind
of
specify
what
data
you
want
to
send
over
and
what
data
you
want
to
keep
or
like
what
data
you
want
to
drop
and
it's
a
very
powerful
config.
You
can
also
manipulate
the
data
before
sending
it
over.
A
So
with
this
you
can
send
as
much
or
as
little
as
you
want.
If
marketing
just
wants
to
look
at
the
ui
level
visits,
you
can
only
send
the
metrics
for
the
ui
level
visits
to
a
remote
store,
and
now
this
is
extremely
popular
and
we
already
have
almost
every
single
popular
time
series
database
out
there,
integrating
with
the
prometheus
remote
right
and
read
natively.
A
So
we
have
a
non
exhaustive
list
of
projects
like
about
25
of
them,
which
kind
of
support
prometheus,
read
and
prometheus
right,
and
you
will
most
likely
see
all
your
favorite
dhdbs
in
there,
but
in
case
you
want
to
send
a
prometheus,
remote
right
data
to
a
different
database.
That's
not
in
the
list
that
doesn't
have
a
existing
integration.
It's
extremely
easy
to
build
an
integration.
A
We
use
the
same
protobufs
if
you're
familiar
with
grpc.
You
will
be
familiar
with
this,
so
in
this
case
we
are
basically
sending
a
prototype
of
message
called
write
request,
which
is
a
list
of
time
series.
Now.
What
is
the
time
series
a
time
series
is
basically
the
labels
for
the
time
series
and
the
samples
in
that
time
series.
A
So,
with
this
extremely
simple
protobuf,
you
just
create
the
protobufs
and
send
it
across
like
prometheus,
creates
this
protobuf
and
sends
it
across,
and
if
you
can
actually
accept
this
protocol,
you
can
store
all
the
data
that
prometheus
sends
to
you
now.
One
of
the
typical
use
cases
for
this
is
actually
long-term
storage.
A
This
is
also
another
solution
for
the
global
view
problem
that
bartek
was
talking
before.
If
you
want
to
combine
the
data
between
u.s
central
one
and
eu
s2
prometheus,
you
just
basically
remote
write
data
from
both
the
prometheus
to
a
central
cluster
and
because
all
the
data
is
in
a
central
place,
you
can
query
the
single
single
central
cluster
and
you
get
all
your
data
that
you
want.
One
of
the
examples
of
the
projects
that
you
can
use
for.
This
is
cortex,
which
I
help
maintain.
It's
also
a
cmcf
project.
A
You
can
also
use
thanos
or
m3
or
several
different
time
series
databases
out
there,
so
this
is
about
remote
right.
I
just
also
want
to
talk
to
you
about
metadata.
This
is
extremely
useful,
as
you
scale
your
teams.
So
one
of
the
things
that
we
notice
is
new
people
come
to
the
teams.
They
look
at
dashboards
and
sometimes
it's
hard
to
understand
what
a
particular
prom
ql
query
is
doing.
A
Proncul
itself
is
simple,
but
if
you
don't
understand
what
the
metric
means,
it's
extremely
hard
to
figure
out
what
the
query
means
and
in
most
cases
you
will
be
able
to
figure
out
just
by
looking
at
the
metrics,
for
example,
node
cpu
seconds
total.
So
this
you
can
kind
of
guess.
Okay,
this
is
the
cpu
usage
per
node,
and
the
reason
of
behind
this,
like
behind
the
obviousness,
is
because
we
have
an
exhaustive
and
really
nice
guide
on
how
you
should
label
your
metrics
and
the
labels
always
follow
the
naming
best
practices.
A
But
sometimes
when
you
deploy
an
exporter,
it
might
not
follow
the
best
practices
or
even
after
following
the
best
practices,
sometimes
the
name.
The
metric
or
the
query
is
not
clear,
for
example,
cube
node
status
capacity.
You
kind
of
don't
understand
right
away
what
it
is
to
help
with
that.
We
actually
expose
the
help
and
type
information
for
every
single
metric
as
part
of
the
exposition
format.
So
if
you
hit
the
slash
metrics
page,
you
will
already
see,
for
example,
in
this
case
you
we
can
see
that
go.
Gc.
A
A
So
in
previous
versions,
prometheus
was
not
able
to
store
all
of
this
data,
but
in
the
recent
version
starting
2.17,
I
think
we
actually
store
and
expose
all
this
metadata
inside
prometheus
itself.
So
we
have
now.
We
now
have
the
metadata
api,
where
you
can
query
for
the
metadata
of
a
particular
metric
next
slide.
A
A
You
can
already
use
this
like
you,
don't
need
to
hit
this
api
you.
It
is
part
of
the
ui
in
grafana
7.0.
So
if
you're,
using
rafana
70
or
the
explore
tab
or
even
the
dashboards,
whenever
you
type
in
the
metric
name,
it
kind
of
helps
you
figure
out
what
the
metric
is.
In
this
case,
you
can
see
this
cube.
Node
status
capacity
cpu
course
is
a
gauge
and
it's
a
total
cpu
course
of
the
node.
So
this
is
super
useful,
especially
for
newcomers
to
understand
what
is
happening
now.
A
Just
as
an
aside,
we
are
having
a
new
react
ui
as
part
of
prometheus.
So
if
you
are
running
prometheus,
we
have
a
tiny
new
shiny,
try,
experimental,
ui
button,
just
click
on
it
and
you
will
be
taken
to
the
new
react.
Ui
we're
actively
developing
this
ui
and
the
same
feature.
The
metadata
expansion
will
be
part
of
this
ui
as
well,
and
we
would
like
to.
We
would
like
all
of
you
to
press
this
ui.
A
If
you
have
bugs
please
file
issues
and
if
you're
a
front-end
developer,
please
contribute
to
all
the
like
all
the
issues
around
the
new
ui.
This
is
going
to
be
the
future
ui
of
prometheus
and
don't
worry
about
breaking
things.
If
something
breaks,
you
can
basically
click
on
the
classic
ui
and
go
back
to
the
old
ui,
all
right
so
going
to
the
future
of
my
data.
We
are
just
not
done
with
metadata
yet
so
we
currently
store
metadata
in
memory,
but
now
in
the
future,
we
want
to
persist
this
metadata.
A
B
Thanks
god,
so
in
the
same
dev
summit
we
actually
discussed
more
more
items
and
one
of
the
further
discussion
points
were
backfilling,
and
this
is
like
very
wanted
feature
of
prometheus.
So
imagine
that
you
know
our
team
and
and
katie
really
wants
to
import
the
metrics
into
the
prometheus
from
other
systems.
Maybe
she's
using
you
know
on
some
other
promoters
or
some
other
system.
She
want
to
migrate
the
data
into
her
existing
instance.
So
how
do
I
do
it
with
prometheus
right?
B
So
it
was
always
available
this
kind
of
way
of
migrating
and
and
and
allowing
to
query
your
data
from
the
other
systems
using
prometheus.
It's
called
remote
read,
so
you
can
configure
your
pro
materials
to
read
from
the
external
system,
for
example
influx
db.
So
every
time
you
query
something
it
will
go
to
that
system,
but
this
is
not
really
backfilling
right,
because
backfilling
means
that
you
want
to
import
of
data
into
your
promotion
storage
directly
to
persist
that
and
use
it
for
future.
B
So
how
do
I
do
this?
Maybe
it
would
be
super
amazing.
If
you
know,
katie
could
just
put
and
write
some
csv
file,
which
is
easy
to
you
know,
play
with
and
just
generate
and
put
it
imported
into
the
promote
storage
now
well.
This
is,
or
will
be
possible
very
very
soon,
so
we
started
like
the
actual
work
on
adding
support
for
importing
from
two
file
formats
csv
and
open
metrics,
and
it
will
look
like
this.
B
You
will
essentially
have
an
a
tool
called
tsdb
import
and
you
just
pass
the
file
that
will
stream
the
data
into
the
rows.
Let's
say
into
the
into
this
tool,
which
will
generate
the
blocks,
which
is
you
know,
kind
of
parsed
and
understandable
by
pro
materials
tsdb
database.
So
once
you
generate
such
blocks
like
such
such
files,
you
can
just
copy
them
onto
the
promoters.
Working
directory
and
prometheus
will
immediately
reload
that
and
and
allow
you
to
access
everything.
B
Okay.
Now,
let's
try
to
show
you
how
it
can
look
like
once
everything
is
merged
and
available.
So
let's
go
to
my
terminal
and
let's
play
with
it
a
little
bit.
So,
let's
imagine
we
have
some
data
to
import
and
well
I
don't
have
any
csv
file
anywhere
handy.
So
maybe
let's
do
something
quick,
so
I
will
go
to
demo
robust
perception
server.
B
B
So
let's
quickly
run
that
in
the
csv
file
I
can
specify
and
different
fields
by
headers
and
essentially
let's,
let's
show
it
and
you
can
see
that
I
kind
of
defined
the
type
of
the
field
by
by
the
field
name
and
by
the
head
in
the
header.
So
you
know
all
those
libel
name.
Little
values
are
defining.
Essentially
what
part
of
the
csv
file
field
is
is
for
label
values
or
names
or
what
are
the
actual,
where
you
put
timestamp
and
where
output
values
and
things
like
that.
B
So
now
we
generated
like
1000
rows
of
of
data.
So
let's
install
our
tsdb
tool.
You
can
install
this
via
one
such
comment
where
I
just
go
get
and
essentially
I'm
pulling
a
certain
comment
that,
as
is
essentially
part
of
my
pull
request,
so
ongoing
work,
but
should
be
available
soon.
Once
this
is
installed,
I
can
hopefully
run
tsdb
tool
to
generate
my
my
blog.
B
B
Let's
actually
do
that
right,
so
we
will
cut
our
file
into
the
tsdb
import
tool
and
output
that
in
some
directory
this
is
actually
kind
of
fast,
because
we
don't
have
much
data,
it's
only
four
series
and
it
generated
a
block
of
one
hour
block.
Essentially
that
has
following
id.
You
can
see
that
block
is
written
and
have
all
the
necessary
files
for
our
tsdb
to
understand
that
so,
let's
just
let's
just
run
from
it,
use
right.
So
I
would
just
create
empty
configuration.
B
Now,
let's
see
if
our
data
is
available,
so,
as
you
can
imagine,
app
is
not
there
because
we
actually
uploaded
gogo
routines
and
it
is
available.
So
when
we
execute
for
our
needed
time
range,
we
will
see
the
data
available
for
us
and
you
can
see
and
that
this
data
is
exactly
the
same
as
on
our
robust
perception
server,
but
actually
robust
perception
server
has
more
of
it
versus
you
know.
Our
server
has
only
imported
only
a
bit
of
it.
B
A
How
you
can
use
just
prometheus
what
the
caveats
are,
how
you
can
use
remote
right
to
write
to
a
different
server
and
how
you
can
you
do
global
view
through
remote
right,
we've
talked
about
metadata,
one
of
the
new
features
that
I'm
super
excited
about,
and
finally,
we've
also
talked
about
backfilling
and
the
future
features
that
are
coming
into
backfilling.
So
this
is
something
I'm
super
excited
about
again,
because
it
will
help
people
migrate
from
older
systems
to
prometheus,
with
all
their
data.
A
B
B
We
want
to
make
sure
that
the
project
is
focused
and,
as
you
can
imagine,
if
you
focus
on
few
functionalities
versus
everything
to
solve
every
problem,
you
will
you
you
will
end
up
with
more
solid
and
concise
product
that
actually
will
excellent
will
excel
on
those
elements
that
matters
so
pick
only
a
few
goals,
and
this
will
kind
of
reduce
the
risk
of
failure.
So
this
is
what
we
did
with.
Probably,
to
use
we
focus
on
scrape,
scraping
metal
methodologies
so
make
sure
there
is
very
consistent.
You
know,
scraping
mechanism
very
reliable.
B
That
prompted
us
did
not
address,
and
that's
totally
okay,
because
those
projects
are
not
competing
with
with
each
other
supreme
use,
it's
not
competing
with
tunnels
and
cortex.
In
fact,
we
are
collaborating
and-
and
we
are
extensions
of
each
other-
let's
say
so
yeah.
This
is
why
chrome
to
use
probably
will
never
kind
of
address
kind
of
you
know
those
scalability
issues
that
tunnels
and
and
and
cortez
cortex
allows.
B
A
So
there's
the
second
question
which
is
cortex
or
thanos,
seem
to
be
more
suitable
for
end
user
consumption.
What
do
you
think
so?
The
idea
around
prometheus
is
you
should
not
rely
like
it's
not
a
distributed
system.
It's
a
single
node
system
that
has
everything
and
it
focuses
on
a
subset
of
the
features
that
as
martech
said
and
one
thing
is
it's
not
a
distributed
system.
So
it's
very
robust
and
even
when
your
data
center
is
under
fire,
you
can
go
to
your
prometheus
and
query
it
and
it's
super
stable.
A
That
way,
and
even
when
you
use
cortex
or
thanos,
you
still
need
prometheus
to
do
the
scraping
and
collection
of
matrix,
and
then
you
send
these
metrics
to
cortex
or
thanos
for
long
term
studies.
So,
even
if
you
run
cortex
or
thanos,
you
still
need
prometheus
and
for
most
organizations,
as
you
start
out,
you
like
you
can
do
a
lot
with
just
prometheus
and
only
once
you
reach
a
certain
scale.
Once
you
go
beyond
few
data
centers,
that's
when
cortex
and
thanos
come
into
picture
so
always
start
with
prometheus
and
then
extend
it.
B
We
have
another
question,
one
yeah
operator
development
plan.
So
that's
good
question.
So
actually
I
was
just
yesterday
was
preparing
some
tutorials
for
prometheus
operator,
because
premature
computer
operators
supports
stateful
parts
of
thanos,
let's
say
so.
You
can
enable
prometheus
operator
to
schedule
or
like
to
you
know
you
can
define
promote
users,
but
people.
You
can
also
define
tunnels,
rulers
and
you
can
add,
by
single
switch
sidecar
to
the
promoters
as
well.
B
So
in
some
some
way
from
two
separators
supports
tunnels,
then
there
are
like
two
other
operators
that
focus
on
the
full
kind
of
tunnel
support.
So
definitely
there
is
work
on
that.
You
can
check
the
banzai
cloud
thunder
separator.
I
think
that's
that's
one
thing.
However,
I
would
recommend-
and
this
is
my
personal
suggestion-
to
use
operators
with
care
and
use
them
only
for
stateful
services
and
stateful
resources
on
kubernetes,
because
that's
where
operators
are
extremely
useful
so
and
and
other
tennis
components
are
just
stateless
but
yeah.