►
Description
How do you know if something bad is happening in your cluster? How do you know that a node is down, an application isn’t responding, or the storage backing a PVC has “disappeared”? If your answer to any of those is “when the users tell us there’s an error”, then it may be time to reevaluate your monitoring and alerting strategy.
Fortunately, OpenShift has built-in tools for doing just this. With only a small amount of work you can ensure that you’re receiving the proper alerts and warnings so that you can, hopefully, avoid any sticky situations. Join us as we focus on Alertmanager, how to configure it and how to customize the settings to avoid both too many alerts and not enough.
Learn more:
https://www.openshift.com/blog/openshift-4-3-alertmanager-configuration
About Ask an OpenShift Admin:
Do you have questions? We want to answer them! Every Wednesday at 11am ET, hosts Andrew Sullivan and Chris Short sit down to answer your questions about Red Hat OpenShift.
Follow us:
Andrew Sullivan https://twitter.com/practicalandrew
Chris Short https://twitter.com/chrisshort
Subscribe to Red Hat's YouTube channel: https://www.youtube.com/redhat/?sub_confirmation=1
About OpenShift:
Red Hat OpenShift is an open source container application platform based on the Kubernetes container orchestrator for enterprise application development and deployment.
Openshift.com https://red.ht/33Fh6uV
OpenShift on GitHub https://github.com/openshift/
OpenShift on redhat.com https://red.ht/2o4ccDk
#RedHat #OpenShift #ContainerPlatform #Containers
A
A
A
A
A
A
Good
morning,
good
afternoon,
good
evening,
wherever
you're
hailing
from
welcome
to
another
edition
of
ask
an
open
shift
admin
office
hours-
we
are
here
myself
chris
short
executive
producer
of
this
thing.
We
call
openshift
tv
and
I
am
joined
by
the
one
and
only
andrew
sullivan
andrew.
How
are
you
today.
B
B
Well,
my
wife
she's
an
exercise
instructor
and
exercise.
People
are
a
little
crazy,
so
her
first
class
starts
at
4
30
a.m,
so
she
was
up
at
3
45
this
morning,
lovely
yeah.
I
was
not
but
yeah,
there's
some
kind
of
mental
thing
about
like
there's
a
huge
difference
mentally
between
5
59
and
earlier
and
6
a.m.
B
I
don't
know
what
it
is,
but
it's
just
like.
Why
does
5
59
a.m
feel
so
much
earlier
and
so
much
harder
to
get
up
than
6
a.m:
anyways
yeah!
Yes,
thank
you.
Chris
appreciate
the
the
introduction
and
welcome
to
the
ask
and
open
shift
admin
office
hours.
So
this
is
episode
31,
where
we'll
be
talking
about
alert
manager
and
kind
of
the
entire
open
shift.
Monitoring
stack
right,
the
things
that
are
going
on
inside
of
there.
B
So
this
is
one
of
our
office
hours
series
of
shows
here
on
openshift
tv.
What
that
means
is
that
we
are
here
for
you
for
our
audience,
and
we
want
you
to
ask
us
questions
through
whatever
client
you
happen
to
be
watching
us
on
twitch
or
youtube,
whether
it's
the
openshift
or
the
red
hat
youtube
channel
feel
free
to
ask
us
questions
in
chat
at
any
point
in
time
about
whatever
topic
it
is.
That
is
top
of
mind
for
you,
and
I
see
that
there's
a
couple
of
people
chatting
already
so
yeah.
C
B
B
Up
with
the
the
following
week's
stream,
if
we
need
to
so
and
that's,
I
also
am
not
afraid
to
to
post.
You
know:
retractions
corrections,
etcetera
because
andrew
frequently
gets
things
things
not
not
quite
right.
How
shall
we
say
so?
As
I
said
today,
our
topic
is
alerting
monitoring
right
to
sort
of
observability
stack,
and
I
am
extremely
happy
to
welcome
our
guest
brian
gottfried.
C
Absolutely
and
you
nailed
the
name
well
done
gottfried
about
two
tts
yeah.
So,
yes,
my
name
is
brian
gottfried.
I've
been
with
red
hat
for
about
a
year
and
a
half.
Now
I
am
a
devops
guy
in
general
pipelines,
app
development
that
sort
of
thing,
but
also
lately
what
I've
been
working
on
is
a
lot
of
like
logging,
monitoring
and
metering.
How
do
you
make
sure
that
when
your
cluster
and
applications
up
and
running
they're
actually
staying
healthy?
B
Yeah-
and
this
is
a
topic
I'm
I'm
excited
about,
and
actually
I've
been,
as
I
said,
I've
been
up
for
a
while
this
morning,
so
I've
already
been
doing
a
lot
of
research
and
and
what
I
thought
I
knew
about
the
openshift
observability
stack
turns
out.
There's
been
a
lot
of
changes
yeah,
so
I
actually
for
any
red
hatter
is
watching.
I
actually
did
a
presentation
at
tech
exchange
back
in
2019.
You
know
in
the
before
times
around
the
open
shift.
B
You
know
monitoring
stack
and
since
then
there
has
been
a
dramatic
change
in
things.
So
I
know
importantly,
we
want
to
cover
some
of
the
architecture,
what
it
looks
like
what
are
the
different
components,
and
then
you
know
brian.
I
definitely
want
to
want
to
see
you
know
how
all
of
that
comes
together
to
enable
user
workload
monitoring.
That
was
a
huge
one.
I
know
people
were
asking
about
it
for
a
long
time,
and
that
was
ga
and
four
six.
If
I
remember
correctly
so
it's
it's
been
there
for
a
little
bit.
B
A
Yes,
so
our
hope
nine
is
asking
some
very
future
looking
questions
and
I
will
ask
them
to
email
me
short
at
redhat.com
as
well
as
andrew.sullivan
redhat.com.
I
forget
your.
Is
that
accurate,
sorry,
andrew.sullivan
yeah
send
us
an
email.
We
will
get
you
the
answers,
you're
looking
for,
because
what
you're
asking
for
is
very
much
a
product
manager
kind
of
question
that
I
don't
have
the
answer
to
you
in
my
head.
Nor
do
I
know
where
to
go.
B
A
B
A
B
So
yeah,
that's
that's
one
way
to
find
out,
so
we
have
two
types
of
presentations
when
it
comes
to
that.
So
what's
new
is
what
is
in
the
usually
the
upcoming
release,
so
in
this
case
4.8
so
you'll
see
product
management
present
on
all
of
the
cooling
stuff,
that's
been
added,
and
then
we
have
what's
next,
which
is
also
product
management,
doing
a
presentation
on
what's
coming
in
future
versions
and
always
keep
in
mind
that
you
know
roadmap.
B
B
A
B
Yeah
so
daniel,
I
see
your
question
any
suggestions
for
long-term
storage
of
prometheus
data
looking
to
keep
historical
data
for
a
product
cluster
for
a
year.
So
I
I,
let's
I'll,
acknowledge
that
and
we'll
push
that
one
off
until
we
let
brian
take
over
and
we'll
make
sure
that
we
address
that
in
that
segment.
A
That
the
the
links
I
dropped
or
restream
bots
spammed
are
for
use
matesh,
but
the
there's
some
breakage
in
the
way
the
links
came
out.
So
I'm
going
to
fix
that
real,
quick,
but
basically
to
answer
your
question:
how
does
openshift
make
container
orchestration
easier
than
others?
What's
the
key
point,
the
key
point
is:
it
is
a
full
like
cloud
native
experience
openshift,
whereas
you
know
some
other
distros
you
kind
of
have
to
okay.
Now
I
have
kubernetes.
B
Yeah
yeah,
I
usually
compare
it
to
rel
right
red
hat
for
25
years,
has
had
this
relationship
with
the
linux
community.
Where
we
work
upstream,
we
do
a
lot
of
things
upstream,
but
for
our
customers
right
the
enterprise
customers.
We
bring
a
lot
of
additional
testing
and
validation,
and
you
know,
ultimately,
all
of
those
things
that
we
do
to
provide
a
supported
linux
right,
one
that
we
can
trust,
one
that
we
know,
one
that
you
can
trust.
B
You
know
to
red
hat
enterprise,
linux
and
openshift
is
the
same
way
right.
We
do.
We
have
that
same
relationship
with
the
upstream
kubernetes,
as
well
as
all
of
the
other
projects.
Products
that
make
up
open
shifts
so
openshift
at
its
core
is
kubernetes,
and
then
we
add
that
you
know
red
hat
supportability
work
and
then
we
add,
on
a
whole
bunch
of
other
things,
to
make
it
a
fully
complete,
robust
production,
ready
platform
for
for
your
containers.
B
B
A
B
A
B
There
was
somebody
who
had
tweeted
at
chris
who
looped
me
into
the
tweet
about
disconnected
installs,
are
hard
right.
They're
they're
not
fun,
to
manage
primarily
because
of
the
image
content,
source
policies
so
icsp
and
what
that
does
for
a
disconnected
install.
Is
you
say
I'm
going
to
and
you
use
the
oh
now:
what's
the
command?
It's
not
scopio!
It's.
B
B
I
think
four
different
registries
there's
quay,
there's
registry.redhat.io,
there's
catalog.redhat.com,
there's
like
four
different
places
that
that
spread
across
it
pulls
down
all
of
that
content
and
it
creates
an
image
content
source
policy
which
maps
this
source
image
is
now
stored
in
this
destination
or
this
new
place
so
that
for
the
cluster
that
icsp
is
added
to
it.
Just
it
doesn't
know
that
it's
when
you
you
know
say
I
need
this
image.
It
doesn't
know
that
it's
actually
coming
from
somewhere
else.
It
automatically
gets
remapped
to
the
new
location.
B
So
I
I
saw
that
tweet
I
reached
out
to
product
management.
I
talked
with
both
the
installer
pm,
as
well
as
one
of
the
pms
who
works
heavily
with
the
disconnected
stuff
and
effectively
for
the
the
answer
was
yeah.
We
know
and
we're
trying
to
make
it
better.
So
I
know
that
there's
some
road
map
stuff
again
pay
attention
for
when
the
the
what's
new
comes
along.
I
know
they're
doing
a
lot
of
work
to
help
make
that
process
much
simpler.
B
A
B
B
Let's
turn
on
do
not
disturb
so
that
way,
I
don't
get
things
popping
up.
So
if
I
go
to
and
I'm
now
what
I
was,
what
was
I
going
to
search
for?
It
is
the
I
want
to
search
blog.openshift.com
or
openshift.com
blog.
I
think,
and
I
want
the
it
is-
the
update
service.
I
think
so
there
is
a
thing:
yeah
openshift
update
service,
so
we
we
talked
about
this
back
when
we
did
the
disconnected
install
stream.
B
We
we
talked
about
it
briefly
back
in
the
four
or
five
days
when
this
was
initially
announced.
You
can
see
four
five
four.
Unfortunately,
it
is
still
not
ga
yet
so
this
is
one
thing
that
when
it
does
go
ga
this
will
make
that
process
much
easier
effectively.
It
connects
into
cincinnati.
It
takes
all
of
the
graph
data,
which
is
what
maps
versions.
B
You
know
edges
to
edges
and
what's
an
eligible
update,
and
what's
not
all
that,
so
it
takes
that
whole
service
and
brings
it
offline
so
that
from
your
deployed
openshift
instance,
you
point
at
that
update
service,
and
then
you
get
that
same.
You
know
off
the
same
over
the
air
update
experience
of
just
going
in,
and
click
update
with
a
disconnected
cluster
that
you
get
with
a
connected
cluster.
B
So
that's
one
thing
that
will
make
it
easier,
but
I
also
know
that
there
is
a
rather
significant
efforts.
I
think
this
slideshow
that
I
had
the
proposal
was
something
like
20.
Slides
of
here
are
a
number
of
pain
points
and
here's
how
we
can
address
those
pain
points.
So
I
know
that
they're
aware
of
those
pain
points
and-
and
I
I
know
that
that
will
that
process
this
process
will
be
getting
better
over
time.
I
don't
know
if
you
shared
this
link
or
not
chris,
I'm
going
to
oh.
B
A
B
B
The
official,
like
red
hat
labs,
those
guys
created
this
thing
as
a
part
of
access.redhead.com,
where
you
can
see
all
of
the
well
there's
a
stable
4.8
already
or
you
can
see
all
those
edges.
So
this
graph
that
data
comes
from
cincinnati
and
that's
where
it's
generated
from
so
effectively.
The
update
service
is
what
replicates
all
of
that
in
an
offline
environment.
B
Anyways
stay
tuned,
we'll
have
more
information
to
share
on
that.
As
time
rolls
forwards,
I
expect
to
revisit
disconnected
in
the
not
too
distant
future,
so
and
again,
much
like
the
last
couple
of
weeks.
If
you
have
topic
ideas,
suggestions,
requests,
let
me
know
I'm
doing
planning
for
the
next
quarter.
So
anything
you
are
interested
in
I'm
more
than
happy
to
accommodate
that
we've
got
some
exciting
and
interesting
shows
coming
up
we'll
share
those
we'll
add
them
to
the
calendar,
as
we
know
what
they
are
or,
as
the
dates
get
solidified.
C
B
So
somebody
had
asked
about
load,
balancer
options,
and
this
was
the
other
one.
I
don't
know
if
this
came
from
twitter
or
if
it
came
from
an
internal
or
if
it
came
from
an
email,
but
it
was
more
or
less
like
hey.
I've
got
an
openshift
cluster
and
openshift
requires
a
load
balancer.
So
what
are
the
options
I
need?
Or
what
are
the
options?
I
have
for
load
balancing
access
to
that
yeah.
B
And
you
can
see
that
there
are
two
sets
of
load
balancers,
the
api
load
balancer
and
the
application
ingress
load,
balancer
or
star.apps.
So
this
one
is
api.cluster
name
this
one
is
star.app
stack
cluster
name,
so
it
doesn't
have
to
be
too
physically
or
or
virtually
logically
separate
to
load
balancers.
It
can
be
one
single
instance
that
you
use
for
both
of
these
functions,
that
that
absolutely
works
just
fine,
but
there's
a
number
of
different
ways
that
you
can
do
this.
B
So
the
first
one
is
not
really
a
load
balancer,
and
that
is
quite
literally
round
robin
dns
right.
There
is
nothing
that
says
that
it
won't
work
to
take
a
round
robin
dns
for
api
dot
cluster
name
and
make
sure
that
it
is.
You
know
the
three
entries
there
are
your
control,
plane,
nodes
and
same
thing
for
your
start-up
apps
right.
Make
sure
that
it
is
the
ingress
or
excuse
me,
the
infrastructure,
nodes
that
have
a
ingress
controller
instance.
So
you
will
encounter
errors
with
star.apps.
B
If
you
have
nodes
that
don't
have
an
ingress
controller
as
a
part
of
that
round,
dns
round
robin
because
it's
not
doing
any
checking
right,
it
doesn't
know
whether
or
not
there's
something
on
the
other
end
of
that.
It
just
says:
hey
go
to
this
host
and
if
there's
no
ingress
controller.
Well,
it's
going
to
respond
back
with
an
error,
so
you
would
want
to
have
a
you
know:
infrastructure
nodes
that
are
hosting
those
those
ingress
controllers
in
that
instance.
So
you
know
where
they
are
but
yeah
it's.
It
works.
B
It's
not
a
load
balancer,
but
it
would
work.
You
know,
keep
in
mind
christian.
I
can't
see
the
chat,
christian's,
probably
giggling.
If
he's
listening
about
you
know
using
dns
as
a
quote:
unquote,
load
balancer,
but
so
the
next
option,
if
you
are
using
on-prem
ipi,
is
the
integrated
right
and
I'm
going
to
use
air
quotes,
load
balancer.
B
So
we've
talked
about
this
a
number
of
times.
I
won't
belabor
the
points
but
effectively
this
is
you
assign
a
virtual
ip
address
right,
so
just
an
ip
address
that
has
a
that
is
pointed
to
the
dns
name
for
startup,
apps
or
api,
and
then
keep
a
live
d
on
the
cluster
manages
which
node
that
that
virtual
ip
address
is
hosted
on.
B
So
this
isn't
really
a
load
balancer,
because
all
of
the
traffic
goes
to
just
one
node
in
the
cluster,
so
the
vip
lands
on
a
node
that
has
an
ingress
controller.
So
start
at
apps
and
all
of
the
ingress
traffic
is
going
through
that
one
node.
So
if
you,
you
know
shard,
if
you
scale
up
the
ingress
controller
from
two
to
three
to
five
to
ten
nodes,
you're,
adding
more
instances
that
are
basically
available
for
keepel
id
to
move
that
vip
to
that
virtual
ip
address
too,
it
doesn't
direct
traffic
to
those.
B
So
with
upi,
and
in
some
scenarios
we've
talked
about
that
before
I'll,
try
and
dig
up
the
the
appropriate
episodes
you.
Basically,
you
want
to
use
an
external
load,
balancer
and,
of
course,
with
non-integrated.
It's
required
so
that
external
load
balancer
can
really
be
anything
that
you
want
or
that
you
are
comfortable
with
right
that
is
capable
of
providing
the
services
that
you
need.
So
what
do
I
mean
by
that?
B
Usually
there
are
two
big
things
that
you
need
to
take
into
account
with
a
load
balancer,
so
the
first
one
is
of
course,
literally
throughput
right.
I
need
my
applications
need
74
terabits
of
throughput
right
to
be
able
to
access
whatever
they're
doing
okay.
Well,
that
means
that
you
need
to
choose
a
load
balancer,
that's
capable
of
doing
that
or
load
balance
or
farm
or
whatever
it
happens
to
be.
I
hope
it's
a
farm
at
that
that
day.
A
B
The
second
one
is
high
availability,
you
know,
so
there
are
a
number
of
options
out
there.
You
know.
Certainly
partners
like
f5,
citrix,
nginx,
etc
are
all
possibilities.
You
know
red
hats.
A
lot
of
us,
you
know
on
on
the
red
hat
side,
will
deploy
aha
proxy
to
like
a
rel
vm
and
just
use
that
you
can
make
that
highly
available
by
deploying
two
of
those
and
using
keep
a
live
d
so
that
you
have
that
that
virtual
ip
address
between
the
two
load
balancers.
B
B
B
C
C
A
B
B
You
know,
there's
others
out
there
that
absolutely
can
work,
but
just
work
with
whoever
your
vendor
is
to
understand
if
they
have
a
kubernetes
ingress
controller,
whether
or
not
that
works
with
openshift
and
whether
or
not
you
want
to
use
their
ingress
controller
or
if
you
just
want
to
use
their
load
balancer
and
the
default
open,
shifting,
crest
controller,
and
I'm
sure
somebody
is
probably
thinking
you
know,
hey
well.
Why
would
I
choose
their
ingress
controller
over
the
openshift
one?
B
B
B
Exactly
and
you
don't
want
it
to
drift
for
a
number
of
different
reasons,
not
the
least
of
which
is,
you
know,
in
line
with
today's
topic
being
able
to
do
things
like
troubleshoot
and
logging.
You
know
correlation
and
all
that
other
stuff
time
is
important,
so
you
definitely
want
to
go
in
and
you
want
to
configure
crony
to
synchronize,
but
we've
had
a
number
of
folks
ask
I
don't
like
rate
I.
B
I
live
in
eastern
time
or
pacific
time
or
whatever
time
zone,
and
everything
in
my
organization
is
on
that
time
zone,
but
the
openshift
logs
are
in
utc.
Can
I
can
I
configure
the
the
time
zone
right
so,
unfortunately,
not
yet,
so
there
is
an
rfe
for
that,
there's
actually
two
of
them.
So
they
are.
I
know
that
they're
looking
into
how
to
do
that,
etc.
I
don't
have
a
time
frame
I'll
see
I'll
dig
up
the
rfe.
So
that
way
you
all
can
look
at
it.
Yeah.
A
B
B
Well,
we're
not
talking
about
logging
necessarily
today
right
yeah,
although
it
is
a
part
of
the
overall
observability
story,
but
yeah
so
just
be
aware.
Unfortunately,
today
you
cannot
and
I'll
I'll
take
a
when
I
shut
my
mouth
here
in
a
moment.
I'll
take
a
moment
see
if
I
can
figure
or
find
the
rfe
I'll
post
that
into
the
chats.
If
you
want
to
take
a
look
at
it,
hopefully
it's
a
public
one
but
yeah
it's
not
there.
B
Okay,
that's
all
I
got,
I'm
I'm
done
with
the
the
top
of
mind
stuff.
So,
let's
move
on
to
today's
topic,
so
chris
I
haven't
been
paying
attention
to
chat.
Really
I
don't
know
okay,
so
just
a
reminder
for
any
any
of
y'all
who
are
listening
to
us.
Please
feel
free.
Any
point
in
time.
Ask
us
questions
and
we'll
we'll
address
those
as
they
come
up
so
tz
file
tz
data
file
for
detroit.
I
know.
B
B
B
B
Chin,
you
know
they
don't
think
of
an
an
alerting
or
a
monitoring
set
of
tools.
So
the
first
thing
that
I
wanted
to
talk
about
is
what
are
the
components
of
the
openshift
monitoring
stack,
so
brian?
If
you
want
to
describe
those
and
I'll
kind
of
you
can
take
over
the
screen
or
I'll
bring
up
some
of
those
components
as
as
you're
talking
about
them.
Yeah.
C
C
Is
it's
going
to
be
trouble
like
you
need
to
know
when
specific
things
happen
that
are
critical
to
the
health
of
your
cluster
and
that's
where
alert
manager
and
the
thanos
ruler
come
in.
So
it's
really
this
idea
of
gather
the
data
and
then
to
do
things
with
the
data.
You
either
want
to
be
able
to
view
it
at
a
high
glance
and
over
time
and
that's
through
your
fauna,
or
you
want
to
get
quick.
Oh
no
something's
going
wrong
alerts
and
that's
where
alert
manager
and
the
thanos
ruler
come
in.
C
The
introduction
of
these
thanos
components
has
been
a
little
bit
more
recent
in
the
later
half
of
the
the
four
versions
of
openshift
thanos
is
a,
I
want
to
say,
like
a
sister
or
a
a
compatibility
project
with
prometheus.
It
implements
a
lot
of
the
same
prometheus
apis,
but
it
has
a
lot
of
functionality
on
top
of
it.
That's
designed
for
distributed
large-scale
systems,
and
I
know
there
was
a
question
earlier
about
long-term
storage.
That's
one
of
the
things
that
thanos
does.
C
I
think
that
when
we
dive
into
it
a
little
bit
more,
we
can
talk
about
the
different
components
of
it.
Really.
What
the
thanos
querier
is
doing
in
this
specific
case
is
it's
it's
acting
as
a
central
point
for
any
of
those
components
that
need
to
go
back
and
access
data,
whether
it's
data,
that's
shipping
in
from
the
cluster
prometheus
or
data,
that's
being
gathered
by
the
user
workload
prometheus.
C
The
querier
acts
as
a
central
point
to
be
able
to
query
that
data,
and
we
don't
have
it
right
now,
but
if
you
were
to
also
deploy
some
of
the
thanos
back-end
stuff
like
long-term
storage,
the
query
allows
you
to
pull
data
from
both
in-memory
storage,
persistent
storage
within
the
cluster
and
that
long-term
storage
kind
of
simultaneously
and
seamlessly.
So
you
can
get
a
query
across
a
real
long
scale
of
data
without
having
to
do
any
sort
of
like
stitching
or
combining
across
multiple
queries.
C
So
it's
as
you
mentioned,
the
diagram
is
large
and
complex,
but
thankfully
most
of
it
really
works
behind
the
scenes
for
you.
When
you're,
looking
at
a
cluster
you're
going
to
be
querying
stuff
via
prometheus
running
prom
queries
to
get
sort
of
ad
hoc
ad
hoc
stuff,
you
can
then
take
more
complex
queries
or
queries
that
you
know
you're
going
to
be
looking
at
a
lot
and
you
can
make
them
into
grafana
dashboards
and
then
you're
also
going
to
use
those
queries
to
say.
C
C
I
want
to
get
an
alert
about
that
and
that's
how
you
end
up
setting
up
alerts
for
alert
manager
or
really,
when
you're
setting
your
own
alerts
to
set
them
up
to
go
through
the
thanos
ruler,
you're
still
using
prom
kill
to
do
it
underneath
the
hood.
It's
just.
It's
sending
you
out
an
alert,
that's
more
descriptive,
rather
than
showing
you
a
graph
that
has
a
sudden
drop
or
something
spike,
or
something
like
that.
Yeah.
B
So
I'm
going
to
switch
over
to
my
my
cluster
here
and
what
I
want
to
look
at
is
I'm
going
to
go
to
projects
I'm
going
to
go
to
monitoring,
so
we
have
two
projects
that
are
inside
of
here:
openshift
monitoring,
openshift
user
workload,
monitoring.
So
this
block
here,
which
is
the
openshift
monitoring
components.
This
is
for
monitoring
all
of
the
core
openshift
things
right,
basically,
anything
in
a
namespace
that
starts
with
openshift.
B
B
Let's
get
that
out
of
the
way,
so
it's
a
little
easier
to
read.
So
here's
all
of
our
alert,
my
alert
manager,
pods
right
we've
got
our
node
exporters.
This
is
just
a
simple
machine
set
that
goes
right,
deploys
all
of
those
prometheus
exporters
to
each
one
of
the
nodes,
the
prometheus
pods
so
on
and
so
forth,
and
here's
our
thanos
queer.
B
B
C
Yep
similar
manager
handles
how
often
alerts
are
fired,
where
you're
actually
sending
them
to
if
they're
going
outside
of
alert
manager
when
like
when
you
repeat
them,
whether
you
send
resolved
solutions
for
them
sort
of
managing
how
multiple
alerts
are
firing.
So
there's
a
there's
the
possibility
to,
if
you
know,
there's
some
alert
that
says
like
the
entire
cluster
is
down.
You
probably
don't
want
to
be
firing.
The
15
other
alerts
that
are
saying
these
individual
components
are
also
down
right.
C
So
you
have
the
ability
to
set
up
a
hierarchy
of
rules
so
that
if
a
certain
alert
fired
you're
not
getting
bombarded
with
a
bunch
of
other
alerts
that
aren't
relevant
at
the
time.
It
offers
a
lot
of
control
over
the
alerts
that
come
out
of
prometheus,
prometheus,
you're
kind
of
defining
alerts
on
an
individual
level
and
they're
relatively
sandboxed
within
each
alert
alert
manager.
Lets
you
control
the
relationship
between
them.
B
A
B
B
Say
yeah
yeah
alerts
are
the
same
way
if
you've
got
a
bunch
of
noisy
alerts.
That
don't
mean
anything.
It
increases
the
likelihood
that
you're
going
to
miss
something
that
is
important.
I
should
also
point
out.
I
didn't
start
at
the
beginning.
When
I
was
talking
about
the
operators.
You
know
I
started
with
prometheus,
but
actually
it's
the
cluster
monitoring
operator
is
is
kind
of
the
first.
B
B
So
if
we
look
at
this
guy,
very
very
simple,
literally,
all
the
data
is
enable
user
workload
is
true
in
typical
fashion.
The
operator
then
takes
action
and
it
deploys
all
of
those
components
or
redeploys
them
as
necessary
for
doing
user
workload.
Monitoring
so
come
back
up
to
our
projects
and
go
to
monitoring
so
by
default.
Without
that
configured,
this
would
be
an
empty
namespace.
B
C
That
service
is
the
same
thing
as
alert
manager
functions,
the
same
way
so
it'll
handle
rules
any
alerting
rules
you
set
up
for
the
new
monitoring
that
you
set
up
via
user
workload.
The
thanos
ruler
receives
those
alerts
and
handles
sending
them
out
to
various
places,
so
it
gets
a
same
style
of
configuration,
but
it's
just
a
different
component
for
it,
and
then
everything
still
does
get
routed
back
to
the
original
alert
manager
got
it.
B
And
so
I
want
to
take
a
moment
to
talk
about
thanos
in
a
little
bit
more
detail
and
in
particular
where
why
and
and
where
thanos
is,
is
important.
So
prometheus
is
really
good
at
what
it
does
scraping.
Metrics,
you
know
aggregating
them.
You
know
being
able
to
display
back
all
of
those
charts
graphs
right
all
of
the
data.
That's
there,
although
there's
arguably
better
tools
for
displaying
back
the
data.
A
B
C
Yeah,
so
actually,
I
think,
do
you
have
the
the
thanos
I
o
page
up
right
now,
yeah.
So
if
you
want
to
go
to
that
and
then
I
think,
if
you
scroll
down
a
little
bit
and
we
look
for
the
components,
thanos
has
long
term
storage
components.
It's
really
just
it's
a
combination
of
several.
Let's
try
the
docks
and
then
let's
see
here,
design
yeah,
maybe
inside
the
design,
okay,
yeah.
So
it's
it's!
A
combination
of
a
bunch
of
different
possible
components,
and
one
of
those
is
the
thanos
storage
thing.
C
So
what
it
does
is
it
takes
component?
It
takes
the
the
time
series
database
data
that
is
being
stored,
either
directly
in
memory
by
a
prometheus
or
a
thanos
instance,
or
in
persistent
storage
and
every
so
often
it
grabs
that
and
like
you
were
mentioning
it
goes
and
it
down
samples
it.
So
it
takes
it
from
the
exact
timeframe.
It
was
in
it
down
samples
it.
I
think
the
defaults
might
be
60
seconds
five
minutes
and
an
hour
and
it
has
sort
of
different.
C
C
And
compacts
it
down,
and
then
you
can
push
it
off
to
the
thanos
store,
which
is
an
object,
storage.
Anything
s3
compatible
and
it'll
store
it
into
there,
and
then
your
thanos
receiver
can
sit
in
front
of
both
that
thanos
store
and
the
object
store,
as
well
as
your
in-memory
data.
That's
inside
the
receiver
itself.
Your
query
excuse
me:
you
can
sit
in
front
of
both
of
those
and
query
across
them
at
the
same
time.
C
Origin
of
itself
is
just
a
lot
cheaper
than
persistent
storage
and
storing
that
way,
but
the
compaction
really
gives
you
that
multiplier
effect
and
being
able
to
store
for
long
periods
really
indefinitely,
possibly
so
it's
very
powerful
in
that
way,
and
then,
with
like
you
mentioned
that
multi-tenancy
aspect
of
it,
it
makes
it
a
lot
easier
to
seamlessly
support
all
those
and
give
whoever
is
querying
that
data.
A
single
point
of
entry
for
still
getting
back
to
it.
They
don't
need
to
worry
about.
C
B
So
I
I
don't
want
to
put
you
on
the
spot.
I
want
to
spend
about
two
minutes
talking
about
grafana
and
observatorium,
and
then,
if
you
have
the
I
want
to
address
daniel's
question.
If
you
have
like
an
example
that
you
can
show
of
how
to
configure
that
longer,
term
retention
or
and
whether
that's
you
want
me
to
drive
and
show
or
if
you
have
something
that
you
can
share
either
way.
B
Okay,
so
I'm
gonna
talk
just
real
briefly
about
grafana
and
observatorium,
so
brian,
if
you
need
to,
if
you
need
to
stage
anything
I'll,
give
you
that
that
time,
so
so
I
do
want
to
quickly
highlight
if
you've
been
using
openshift
for
any
period
of
time.
When
you
look
at
the
dashboards
in
the
recent
couple
of
versions,
these
have
changed.
B
It
used
to
be
that
this
would
link
directly
out
to
grafana,
which
we
have
here
and
you
would
be
able
to
see
all
of
the
dashboards
as
soon
as
I
move
this
out
of
my
way,
and
I
can
go
in
here
and
I
can
look,
and
I
can
see
all
of
the
things
that
are
happening
in
my
cluster
right
through
grafana,
and
my
understanding
is
that
we
are.
We
are
moving
away
from
this
being
the
primary
interface
for
reviewing
metrics,
with
the
primary
reason
being
our
back
effectively.
B
We
don't
have
or
grafana
does
not
have
a
way,
and
I
may
be
misunderstanding
the
way
that
it
was
explained
to
me.
But
it's
hard
to
do
our
back.
That
is
integrated
with
openshift
and
namespaces,
and
specific
only
allowing
specific
people
to
see
their
specific
things
like
it
is
or,
unlike
it
is
with
when
it's
natively
integrated
into
the
dashboards.
B
B
C
B
So
if
you're
using
you
know,
openshift
direct
or
rosa
red
hat
open
shifts,
aws
on
aws,
aro,
azure,
red
hat
open
shifts
right
rocks
the
ibm
offering
basically
all
of
those
feed
data
into
an
observatorium
instance
that
is
run
by
red
hat,
and
that
is
how
we
provide
those
long
term
analytics
and
other
information
out
of
that.
So
I
don't
have
a
lot
more
information
on
that.
This
is
one
that
I'd
like.
I
said
I
just
learned
about
it,
so
I'm
I'm
interested
in
learning
more.
You
know.
B
C
That's
actually
how
acm
manages
its
multi-cluster
metrics,
it
deploys
the
observatorium
operator
and
then
the
operator
goes
and
sets
up
a
thanos
instance
with
all
the
necessary
components
that
actually
ties
into.
I
think
there
was
a
question
around
seeing
an
example
of
setting
up
a
thanos
store.
That
is,
I
think,
probably
the
the
greatest
weakness
that
thanos
right
now
is
that
it's
relatively
new
and
it's
relatively
complex.
C
C
It
really
makes
it
incredibly
simple
to
get
multicultural
reservability
set
up,
but
also
get
that
long-term
storage
set
up.
I
think
there
would
probably
be
some
growing
pains
with
getting
thanos
set
up
within
a
single
cluster
prometheus.
So
if
you're,
trying
to
if
you're,
trying
to
work
out
long-term
storage,
you're
already
operating
multiple
clusters,
it
may
be
from
a
a
time
perspective.
C
Certainly,
if
you
only
really
have
one
or
two
clusters-
and
you
just
want
to
be
able
to
set
up
a
thanos
instance
for
that
it
may
be
worth
the
time
to
do
it
individually,
but
you
are
going
to
find
that
there's
going
to
be
probably
a
lack
of
examples
out
there
so
far,
thanos
is
still
a
pretty
new
project
and
it's
still
growing
and
evolving.
So
I
personally
prefer
I
love
having
to
be
you
developing
behind
me
and
supporting
me
in
that
way,
but
whatever
whatever
floats.
A
C
B
Okay
and
a
reminder
for
our
audience,
please
submit
any
questions
that
you
have
so
I
I
see
I'm
gonna
while
you're
shifting
stuff
around
brian.
So
I
see
a
couple
of
things
in
here,
so
our
hope,
nine
yeah,
I
I
know
grafana
has
arb
our
back.
Arbok
are
back.
So
that's
the
fancy.
I
need
to
find
some
details
I'll
see
if
I
can
find
details
and
include
them
in
the
blog
post
or
maybe
we'll
talk
about
it
next
week
as
to
the
precise
rationale
there.
I
know
it's
changing.
B
So,
let's
see-
and
I
want
to
talk
about
ex
exer
exec
rr
is
user
workload,
monitoring
with
istio
injected
apps
going
to
be
supported
with
I
don't
know
what
mtls
is
mutual
tls.
Okay,
at
the
moment
there
seems
to
be
no
way
to
scrape
metrics
on
service
mesh
with
mtls
enabled
the
only
solution
we've
seen
is
probably
federation
brian.
Unless
you
know
the
answer
to
that.
That's
not
one
of
those.
C
So
I
haven't
done
much
with
istio.
I
don't
know
whether
the
limitation
there
is
something
to
do
with
the
mtls
portion
of
it
or
the
or,
if
it's
something
about
where
service
mesh
is
being
deployed.
I
do
know
that
user
workload
monitoring
right
now,
doesn't
let
you
create
alerts
inside
any
of
the
default
openshift
namespaces,
I'm
guessing
that
istio
is
being
deployed
inside
something
like
openshift
service
mesh,
and
that
may
be
the
limitation
there
and
that's
kind
of
just.
C
You
get
two
different
teams
working
on
different
priorities,
so
you
gotta
get
the
communication
board
in
there.
An
rfe
may
be
the
best
option.
I
I
my
only
concern
with
federated
prometheus
is.
I
know
that
performance
is
a
real,
real,
big
issue.
That's
kind
of
the
main
reason
they
created
thanos
in
the
first
place
is
trying
to
set
up
a
prometheus
that
scrapes
from
other
prometheuses,
because
it's
a
pull
method
as
opposed
to
a
push
as
you
get
more
data
into
there.
C
It's
really
really
difficult
to
scale
that
top
level
prometheus
so
that
it
can
actually
handle
it
in
memory,
so
you
can
certainly
give
it
a
shot.
I
personally
haven't
had
a
lot
of
success
with
trying
to
do
a
production
ready,
federated
prometheus.
It
requires
a
lot
of
tweaking
and
very
careful
measuring
of
what
you
think
your
load
is
going
to
be
doing
sure
that
you
don't
get
kind
of
a
cascading
crash
effect.
So
I'm
sorry
I
don't
have
a
better
answer
in
regards
to
that.
That
is
a
a
complex
situation.
C
I
think
maybe
go
back
to
your
service
point
for
it
and
see
there.
May
you
may
be
ahead
of
the
path
and
you
may
be
the
one
opening
at
the
rfe.
Otherwise
there
may
be
somebody
who's
already
sort
of
noticed
that
some
of
these
components
may
need
extra
measuring,
but
don't
have
the
ability
to
do
so.
Yeah.
B
C
So
that's
a
good
question.
I
think
what
you
could
do
is
so
you're
setting
your
pod
up.
You
have
multiple
containers
in
there.
You
could
have
each
one
of
them
pushing
out
metrics
on
a
different
endpoint
and
then
set
up
a
service
monitor
for
each
port,
essentially
on
that
pod,
and
that
way
you
can
measure
from
multiple
ones.
That
way,
I
haven't
actually
tried
that
before.
C
Usually
I
I
try
to
as
much
as
possible
stick
with
these
sort
of
one
pod,
one
behavior
type
thing,
so
I
don't
normally
have
a
need
to
measure
metrics
from
multiple
containers
within
a
pod.
Usually
if
I've
got
a
pod
of
multiple
containers
in
there,
you've
got
one
main
container
and
then
some
supporting
ones
in
there
that
their
behavior
usually
stays
internal.
C
So
I
think
it
would
be
possible,
but
I
haven't
tried
it
personally,
so
I
don't
want
to
commit
to
that
necessarily.
But
my
guess
would
be
try
and
expose
multiple
ports
on
that
pod
and
have
the
containers
output
their
metrics
to
each
one
of
those
ports
individually,
and
then
you
would
set
up
a
service
monitor
or
pod
monitor
to
measure
each
one
of
those
separately,
and
then
you
would
you'd
actually
be
able
to
capture
the
instances
that
way.
C
So
I
think
I
just
I
threw
a
bunch
of
terms
out
there
that
we
maybe
didn't
cover,
but
within
the
documentation
for
managing
metrics
inside
the
openshift
one.
So
there's
a
thing
for
setting
up
metrics
for
user-defined
projects
and
that's
where
service
monitors
and
pod
monitors
come
in
really
what
they're
doing
is
you're
just
pointing
user
workload
monitoring
to
a
specific
service.
B
A
B
New
ibm
acquisition,
so
I
want
to
make
sure
that
we
that
we
don't
have
to
do
an
abrupt
close.
So
there's
three
questions
that
I'm
aware
of
so
one
is
the
configuring
long-term
storage
so
I'll
be
sure
to
include
that
in
the
blog
post.
We'll
also
briefly
talk
about
it
next
week,
daniel,
if
you're
able
to
turn
it
tune
in
so
for
the
matter
most
example.
B
I
will
dig
up
where
that's
at
there
is
one
I'll
put
that
in
the
blog
post
as
well
as
we
can
talk
about
that
next
week.
So
brian,
maybe
maybe
you
might
be
back
on
a
little
bit-
sounds
good
awesome
and
then
yeah
and
then
daniel
can
alert.
Managers
send
alerts
to
separate
email
addresses
based
on
namespace
labels,
in
which
the
alert
occurs.
C
You
know
that's
a
good
question.
I
think
it's
going
to
depend
on
how
you
define
your
alert.
You
can
definitely
apply
individual
labels
well,
actually
so
alert.
Any
alert
that
comes
out
of
alert
manager
is
going
to
come
with
the
namespace,
I
think,
included
on
it
already
and
then,
within
your
alert
configuration
do.
I
have
an
example.
Let's
see
here
of
here
we
go
so
this
is
an
example
alert.
This
is
actually
sort
of
related
to
the
the
mana
rust
one
or
no
hold
on.
C
I
don't
have
the
alert
in
question,
but
at
any
rate,
there
is
a
possibility
to
sort
of
set
up
dynamic
endpoints
for
alerts
within
your
alert
manager
configuration
so
within
this
router.
Here
you
can
see
for
routing
to
a
slack
channel
in
this
case,
setting
up
the
receiver.
It
has
this
web
hook
url
here
you
could
set
it
up
to
pull
from
a
label
there.
I
know
I
think,
there's
another
robust
perception
one
about
and
by
the
way
this
is
an
incredible
resource.
C
If
you're,
trying
to
figure
out
a
specific
use
case
for
alert
manager,
prometheus
robust
perception
is
a
great
blog
for
that
and
then
actually
you're
using
labels
to
direct
email
notifications.
So
we
can
link
this.
I
can
send
this
blog
post
to
you
guys
and
have
you
push
it
out
via
there
via
the
chat,
as
well
as
later
on
in
the
blog
post,
sent
that
via
google
chat?
C
But
you
can
see
here,
you
can
pull
out
a
labels
from
a
alert
that
is
reaching
alert
manager
and
then
specify
that
in
the
receiver
configuration.
So
you
would
probably
want
to
do
the
same
sort
of
thing
here
and
then
I
think
it
would
probably.
I
think
it
would
just
be
dot
labels
and
then
namespace
I'd
have
to
go
and
look
and
see
where
the
namespace
label
actually
comes
up,
whether
it's
in
the
group
labels
or
in
the
individual
alert
one.
C
But
you
should
be
able
to
do
the
same
sort
of
thing
as
this
in
order
to
specify
them
in
there.
The
only
thing
that
might
be
difficult
is
that
if
you're
not
mapping
the
field
of
the
namespace
label
to
directly
to
the
email
that
it's
sending,
you
might
need
to
do
some
sort
of
translation
inside
this
receiver,
right
so
you'd
say
for
namespace
ns1.
I
need
you
to
send
the
email
this
for
namespace
ns2.
C
I
need
to
send
the
email
list
you
might
have
to
do
that
logic
inside
the
receiver
here
or
if
you
can
name
your
name
spaces
based
on
whatever
the
email
is
that
would
also,
or
vice
versa.
If
you
set
up
an
email
account
that
just
means
for
the
namespace,
then
you
don't
need
to
do
any
sort
of
that
translation.
You
can
just
directly
put
the
value
of
the
namespace
label
into
the
to
field
right
there
and
send
it
that
way.
C
B
Right
so
I
know
that
we
had
some
outstanding
questions
that
we
didn't
get
to.
If
you
saw
me
looking
down,
I
was
taking
notes
on
on
what
to
follow
up
on
so,
like
I
said
I
will
follow
up
in
the
blog
post
on
friday.
We'll
also
follow
up
in
the
opening
segment
of
the
show
next
week,
which
is
I
understand
chris,
you
will
not
be
with
us
next
week.
I.
B
Thank
you,
everyone!
Thank
you,
especially
brian.
I
really
appreciate
you.
You
joining
you've
been
tremendously
helpful
and
I
have
learned
a
huge
amount
in
this
particular
episode.
More
than
usual,
so
yeah,
it's
as
I
like
to
say,
openshift
is,
is
like
a
a
fractal
right.
Every
time
you
learn
something
it
turns
into.
You
know
a
whole
bunch
of
other
things
that
every
time
you
learn
something
about
one
of
those
it
turns
into
a
whole
bunch
of
new
things.