►
From YouTube: How to Be 10x SRE? A Deep Dive to Prometheus Operator - Jayapriya Pai & Haoyu Sun, Red Hat
Description
How to Be 10x SRE? A Deep Dive to Prometheus Operator - Jayapriya Pai & Haoyu Sun, Red Hat
Prometheus Operator is a fairly known solution for monitoring Kubernetes workloads using Prometheus. Many Cloud Native users benefit from Prometheus Operator CRD-based components like ServiceMonitors, PodMonitors, PrometheusRules, Probes which allows better configuration management, self-service or even multi-tenancy. Many things were said about Prometheus Operator in the past, but we believe there is room for a dedicated talk about the designed way of utilizing Prometheus Operator on production Kubernetes clusters. In this talk, Jayapriya a Prometheus Operator contributor from the Red Hat Monitoring team and her teammate Haoyu will explain all you need to know about the common usage patterns. The audience will see practical examples and learn advanced features like securing Prometheus with TLS, enabling robust remote write and operating AlertManager via Prometheus Operator. The talk will also summarize the monitoring and operating aspects of the Prometheus Operator itself, sharing first hand experience of maintaining Prometheus Operator in thousands of OpenShift clusters.
A
Hello,
everyone.
I
welcome
you
all
to
this
talk
on
how
to
be
10x
sre,
a
deep
dive
to
prometheus
operator
as
a
sre.
We
would
like
to
set
up
efficient
and
reliable
monitoring
and
alerting
for
infrastructure
and
also
help
the
application
development
team
setup
monitoring
for
their
own
application
as
well.
But
how
long
does
it
take
to
set
up
the
monitoring
in
kubernetes
and
maintain
it
in
this
presentation?
A
A
The
key
takeaways
from
this
presentation
are,
you
will
get
to
know
how
to
configure
monitoring
in
kubernetes,
with
the
help
of
prometheus
operator
details
on
how
prometheus
operator
works
and
its
available
features,
examples
on
tls,
config,
remote,
write
and
alert
manager
configs
towards
the
end.
We
will
also
discuss
some
of
the
limitation
and
also
the
future
aspects
of
prometheus
operator
project.
A
A
This
can
be
very
overwhelming
and
very
time
consuming
task.
This
is
where
prometheus
operator
comes
into
picture.
The
prometheus
operator
provides
humanity's
native
deployment
and
management
of
parameters
and
its
related
components.
The
purpose
of
this
project
is
to
simplify
and
automate
the
configuration
of
prometheus
based
monitoring
stack
for
the
kubernetes
cluster.
A
A
A
One
is
prometheus
operator
itself,
which
automatically
manages
promises
and
related
monitoring
components
and,
second,
is
queue
prometheus,
which
is
used
to
set
up
the
monitoring
stack
in
kubernetes
out
of
the
box.
We
have
a
detailed
section
on
this
later
now.
Let's
see
how
the
prometheus
operator
works,
we
will
talk
about
operator
pattern,
custom,
resource
definition
and
custom
controllers.
A
Operator
pattern
was
introduced
by
coreos
as
a
class
of
software
that
operates
other
software.
The
function
of
the
operator
pattern
is
to
capture
the
intention
of
how
a
normal
human
operator
would
manage
a
service.
A
human
operator
needs
to
have
a
complete
understanding
of
how
an
app
or
services
should
work,
how
to
deploy
it
and
how
to
fix
any
problems
that
may
occur.
In
short,
we
can
term
operators
as
software
sras
operators
are
built
on
two
key
principles
of
kubernetes
custom
resources,
which
is
implemented
by
the
way
of
custom,
resource
definition
and
custom
controllers.
A
A
Custom
controllers
are
used
to
observe
this
custom
resource
and,
with
the
help
of
information,
it
takes
action
to
adjust
the
community's
cluster
to
the
decide
state.
Let's
look
at
the
prometheus
operator
architecture,
it's
made
of
eight
custom
resources
as
seen
the
figure
and
controllers
which
watches
these
resources
and
manages
their
deployment.
A
A
As
mentioned
earlier,
there
are
eight
custom
resource
types
available
as
part
of
prometheus
operator.
The
first
three
crds,
which
are
prometheus
alert
manager,
thanos
ruler,
forms
a
stateful
set
configuration
of
these
applications,
whereas
the
other
five
crds
service,
monitor
port,
monitor,
probe,
prometheus
rule
alert
manager,
config
from
the
configuration
objects.
They
are
used
to
define,
scrape
conflicts,
monitoring,
different
types
of
targets,
defining
the
alerting
rules
and
alerting
conflicts.
A
Let's
look
into
the
details
on
the
prometheus
resource,
the
prometheus
custom
resource
definition
declaratively
defines
a
desired
prometheus
setup
to
run
in
a
kubernetes
cluster.
It
provides
options
to
configure
replication,
persistent
storage
retention
and
alert
managers
to
which
the
deployed
prometheus
instances
sent
alerts
to
for
each
prometheus
resource.
The
operator
deploys
a
properly
configured
stateful
set
in
the
same
namespace.
A
A
A
A
What
it
means
is
that,
if
you
have
services
which
has
labels
defined
on
them,
you
can
specify,
in
the
selector
field
of
the
service,
monitor
to
select
all
the
services
which
has
sets
a
particular
labels,
and
port
monitor
is
used
to
select
pods
directly
based
on
labels
again
similar
to
the
service
monitor.
Based
on
what
labels
have
been
set
on
pods,
you
can
specify,
in
the
selector
field
of
pod,
monitor
to
select
all
those
powers
which
has
those
labels
set
and
the
last
one
prob.
It
is
used
to
monitor
static
targets
or
increase
objects.
A
A
They
should
be
already
deployed
in
your
cluster
and
the
next
field
is
the
targets
field
inside
that
you
can
specify
the
static
config
under
that
static
field,
you
will
be
specifying
all
the
static
targets
you
wish.
You
want
to
monitor
once
you
create
this
object
and
there
are
no
issues
in
the
config
you
created.
The
prometheus
operator
should
pick
this
up
and
once
it
sinks
all
the
configuration
it
will
create
the
targets
for
this
in
the
promises.
A
On
the
left
side,
I
have
an
ingress
resource
which
has
the
label
set
as
apps
dot,
kubernetes
dot,
io
slash
name
example
app,
and
on
the
right
side
I
have
a
probe
resource
in
the
selector
field.
I
have
specified
the
same
labels
which
has
set
on
the
increased
resource,
so
this
is
how
the
probe
is
able
to
select
the
particular
ingress
resource
is
which
each
one
to
monitor.
A
A
A
And
on
the
right
side,
you
can
see
here
how
the
prometheus
pod
is
been
actually
designed.
So
inside
the
prometheus
board,
you
will
have
the
prometheus
container
as
well
as
a
config
reloader
sidecar,
the
config
reloader
sidecar
is
responsible
for
watching
the
changes
that
has
been
happened
in
the
prometheus
config
and
it
will
reload
those
changes
in
the
parameters.
A
B
B
Here.
In
this
example,
there
are
one
alerting
rule
and
one
recording
rule
more.
Details
can
be
saved
into
annotation
messages
so
that
the
user
once
received
this
alert
can
better
understand
the
situation
for
better
management
of
the
alerts.
Prometheus
operator
is
also
capable
of
operating
alert
manager.
Cluster
alongside
prometheus
alert
manager
receives
alerts
from
prometheus
and
thanos
for
further
processing.
B
B
B
If
there
are
more
than
one
replica
evil
setups
as
a
cluster,
the
global
configuration
is
either
stored
in
a
secret
using
the
name.
Let
manager
dash
the
name
of
this
manual
crd
or
we
can
directly
put
our
configurations
into
the
spec
section
of
the
crd
to
complete
the
global
configurations.
There
is
another
crd
alert
manager
config.
B
B
B
There
are
metric
collection
when
prometheus
scripts,
metric
endpoints
from
service
polled
or
probes
the
web
services.
The
api
server
of
prometheus
other
manager
and
thanos
can
secure
themselves
by
using
tls
the
remote
read
and
write
the
prometheus
can
use
tls
to
secure
their
remote
endpoints,
both
when
sending
and
receiving
the
samples.
B
B
The
other
part
is
the
server
name
and
the
value
we
should
skip
the
validation
of
server
certificates.
The
operator
can
also
help
us
in
setting
up
remote
right
for
prometheus.
Remote
variety
is
often
used
in
two
purposes.
The
first
one
is
to
save
metrics
in
the
long
term
storage
for
future
reference,
then
we
can
export
matrix
to
some
external
system
for
data
analysis.
B
B
The
second
is
the
queue
config
where
we
can
tune
up
the
performance
of
the
remote
right.
Then
we
have
the
authentication
config
section
here.
In
this
example,
it
is
an
oauth2
setting,
but
many
more
are
available.
Such
as
the
amazon,
c4,
the
basic
password
and
sata
one
last
thing
here,
we
can
also
specify
the
tls
config
we
had
just
covered
in
the
last
section
till
now.
B
B
B
B
B
B
B
B
There
are
many
new
features
planned
for
prometheus
operator.
Here
are
the
four
improvements
coming
in
near
future.
The
first
one
is
script
config.
It
is
a
crd
to
provide
generic
script
config
that
is
not
limited
to
the
existing
monitoring
crds
then
the
other
manager
config
will
become
available
in
v1
and
beta1.