►
Description
From one cluster to hundreds, there’s always maintenance to be done, upgrades to be installed, and optimizations to be implemented - wouldn’t it be nice if the cluster could just guide you on what to do next , or better yet - warn you before something goes wrong. Join co-hosts Gurney and Joydeep Banerjee along with their guests from the Red Hat Insights team as they talk about all things Insights. From cluster health and upgrades insights to cost efficiency, stability concerns to configuration changes - insights seeks to bring only the most relevant, actionable insights to your fingertips at the cluster - and fleet - scale. Learn about the service and operational challenges they faced and help shape the future of insights in today’s show!
A
A
Hello
and
welcome
to
the
cloud
multiplier.
This
is
episode,
seven
and
your
show
for
all
things
distributed
cloud.
I'm
joined,
as
always
by
my
co-host
joy,
deep
and
today
we
have
three
guests
to
talk
about
red
hat
insights,
one
of
the
cool
one
of
one
of
the
most
interesting
and,
in
my
opinion,
most
useful
tools.
I've
worked
with
kind
of
just
integrated
into
my
openshift
dashboard
and
in
my
acm
dashboard.
A
So
thanks
for
ever
thanks
to
everyone
for
joining
we'll
go
ahead
and
kick
off
with
an
around
the
room
of
our
guests,
we'll
go
in
alphabetical
order.
I
think
this
time
so
kachi
you
get
to
go
first,
do
you
want
to
introduce
yourself
tell
us
about
what
you
work
on
and
then
we'll
go
around
the
room
and
dive
in
on
some
insights.
B
C
D
And
finally,
it's
happening
so
hey
everyone.
My
name
is
raleigh
vocal,
I'm
running
a
small
team
of
product
managers
that
are
focused
on
all
things:
insights
around
openshift,
so
today
we're
gonna
be
touching
on
some
of
these
services.
The
engineering
work
behind
that
what
is
happening
behind
the
scenes
with
the
insights
and
as
tomas
mentioned,
we'll
also
show
you
some
of
the
interesting
integrations
where
we
are
using
insights
data
and
we're
making
data-driven
decisions,
because
that's
what
the
talk
is
about
today,
amazing.
I.
A
D
So
there's
there's
three
services
right
now,
but
there's
a
lot
of
magic
happening
behind
the
scenes.
So
you
as
a
customer,
you
see
a
portion
of
that
and
we're
actually
adding
new
services.
So
I'm
not
gonna
touch
on
it,
because
this
should
be
another
session
at
some
point.
But
we're
gonna
introduce
a
new
service
in
about
a
month,
and
I
would
love
to
present
it
here
as
well.
A
A
E
I,
before
you
go
there
guarding,
I
actually
have
something.
Let
me
send
send
you
the
link
control
the
chat.
This
is
a
very
interesting
github
repo,
which
I
came
across,
which
allows
us
to
test
kubernetes
applications.
You
know
test
whether
they're
working
well
test
their
slos,
etc,
etc.
This
is
very
fascinating.
I
want
to
explore
this
a
little
bit
more,
and
I
think
this
might
be
interesting
for
us
as
well.
I
just
happen
to
chance
on
this
repo.
E
D
It
looks
interesting
and
we
should
probably
touch
on
that
as
well,
because
when
we
start
talking
about
insights
there's
there
are
some
open
source
tools
that
we're
using
behind
the
scenes
as
well
to
look
at
the
deployments
that
you
have
on
your
cluster
to
look
at
some
consistency
and
compliance
even
over
your
cluster.
So
we'll
touch
on
that
too.
I
haven't
heard
about
this
one.
So
thanks
jody
for
the
link,
we'll
take
a
look
at
that
for
sure.
A
That
is
amazing.
Okay,
so
I
onto
my
my
morning,
one
is
anecdotal.
I
warned
everyone
here
that
I
was
gonna
tell
my
story
about
insights,
so
we
were
running
upgrade
just
kind
of
routine
well
slightly
slightly
more
than
routine
upgrade
of
one
of
our
clusters
from.
I
think
it
was
four
eight
to
four
nine,
so
we
were
going
ocp
4849..
A
I
kick
off
the
upgrade.
Nothing
happens,
we're
sitting
there,
the
upgrade
isn't
occurring.
I'm
like
what
is
what
is
wrong
and
then
I
get
an
alert.
It
says:
hey
you
have
here's
a
warning.
You
need
to
acknowledge
that
you
need
to
acknowledge
this
warning
before
you
proceed,
because
something
bad
is
gonna
happen,
and
I'm
now
immediately
over
talking
to
someone
in
support
and
I'm
like,
I
was
warned
that
I'm
about
to
do
something
very
bad
to
my
cluster
and
like
yeah,
you,
it
turns
out,
you
are,
and
you
should
have.
D
A
Yeah-
and
it
wasn't
easy-
it
even
said
here
is
what,
if
you,
if
you
acknowledge
this,
if
you
know
what
you're
doing
go
ahead
and
make
this
command
everything's
good
to
go,
it's
very
visible.
I
knew
immediately
what
to
do
what
doc
page
to
go
to
one
of
our
colleagues
trevor
immediately
said:
oh
yeah,
you've
hit
this
thing
interesting.
God.
Insights
worked
on
the
other
side
of
this
support
journey
if
you
hadn't,
so
that
said,.
D
And
I
want
to
say
garnet
this.
This
is
what
I
like
about
insights
as
well
right,
I'm.
I
have
another
anecdotal
thing
to
your
thing.
I
recently
purchased
a
new
drone
and
the
first
thing
when
I,
when
I
started
that
was
it
was
asking
me:
do
you
want
to
share
the
usage
and
analytics
data
with
us
all
right
guess
what
I
checked
right
like
hell?
No,
I
don't
know
what
you're
doing
with
the
data.
D
I'm
not
going
to
send
you
anything,
but
with
insights,
we're
actually
showing
our
customers
and
we're
showing,
even
here
today,
what
we're
doing
internally
with
the
data
and
how
it's
useful
for
the
product
quality
for
the
support,
efficiency
and
also
for
some
edge
cases
like
you
had
your
issue
with
your
upgrade.
So
that's
this
is
the
cool
thing
about
insights
that
we're
very
open
about
what
we're
doing
with
that
information
and
how
we're
bringing
it
back
to
you.
A
Yeah,
it's
those
little
nudges,
those
little
edge
cases
where
hey.
If
we
can
catch
a
customer
before
this
easy
this
easy
to
make
misconfiguration
this
change.
We
know
these
things.
How
can
we
make
sure
that
a
customer
knows
these
without
having
to
go
pull?
Our
release?
Notes
turns
out
if
their
cluster
says
hey,
I
need
help.
Here's
here's
the
problem,
here's!
What's
going
on!
I
it's!
I
I
talked
to
a
customer
recently
and
they
said
we're
currently
doing
reactive
operations.
A
How
can
we?
How
can
we
change
that
reactive
operations
into
proactive?
We
want
to
be
ahead
of
the
issues
we
would
anticipate.
Then
we
want
to
step
in.
We
want
to
do
a
good
job
and
know
know
something's
going
to
go
wrong
before
it
goes
wrong.
The
usual
dream
of
I
want
to
know.
I
want
to
know
about
an
outage
about
10
minutes
before
it
starts.
D
D
Elsewhere
and
chances
are
that
we
also
have
a
solution
for
you.
So
that's
that's
another
thing
where
we
basically
match
the
information
with
our
own
internal
knowledge
base.
With
the
experience
of
our
support
people
solution,
architects,
even
our
own
sre
team,
that
runs
openshift
for
internal
services
or
as
a
managed
offering
for
our
customers.
We
take
that
experience
and
bake
it
into
inside,
so
you
can
use
it
as
well.
E
That's
amazing,
so
you
are
actually
you
guys
are
actually
consuming
the
data
that
we
customers
would.
I
okay
customers
would
send
to
redhat
cloud,
and-
and
that
is
the
data
across
thousands
of
clusters-
that's
deployed
and
you
are
being
totally
transparent
on
what
data
you're
collecting.
In
fact,
you
know
it's
it's
out
there
in
the
get
up,
of
course,
and
then
you're
mining
that
data
to
find
out
patterns.
D
Exactly
but
with
one
thing
that
we
always
start
with
the
product
first,
if
the
problem
can
be
fixed
in
the
product
we
go
to
the
operator,
we
go
to
the
company
and
we
fix
it
there,
then,
if
we
cannot
fix
it
or
there's
again,
some
sort
of
recommendation
best
practices
that
you're
not
following.
As
a
customer,
you.
D
D
E
D
And
yeah
this
one
is
interesting,
but
there
are
some
more
serious
ones
and
I'll
show
some
examples
here,
of
course
about
things
that
you
really
have
to
follow
and
keep
in
mind
that
a
lot
of
these
things
are
that
insights
advisor
because
we're
talking
about
this
one
component
right
now,
insights
advisor
one.
The
thing
that
insights
advisor
is
doing.
Is
it's
proactive
right,
so
a
lot
of
these
things
are
basically
telling
you
if
you
follow
this
path.
D
D
Things
will
get
worse
if
you
continue
with
your
setup
and
that's
why
you
should
absolutely
watch
this
and
look
at
the
problems,
the
one
that
he
garnered
brought
up
with
an
upgrade.
That
was
a
great
example
right,
so
the
cluster
was
operational.
It
was
working,
but
something
was
blocking
the
upgrade
look
at
the
recommendation,
the
recommendations
majority
of
them
are
actionable.
So
it's
going
to
tell
you
the
exact
command
line
that
you
should
run
on
your
cluster.
A
Yep,
exactly
and-
and
I
know
I
may
have
been
able
to
find
the
right
people
and
get
support,
engaged
and
fix
it.
If
I
just
waved
my
badge
and
said,
I
don't
care
about
your
insights,
I'm
going
to
go
ahead
and
upgrade
my
cluster,
but
I
know
for
a
fact
if
I
had
pulled
our
entire
fleet
up
to
that
new
version
at
the
same
time
that
that
is
the
the
multiplier,
the
scalar
factor
that
make
takes
insights
from
well.
It
helps
me
get
out
of
a
bad
case
and
a
support
ticket
wants
to
it.
A
Gets
me
out
of
our
entire
operations.
Is
we're
completely
down
we're
dead
in
the
water?
All
hundred
clusters
are
gone
because
we
we
did
this
thing
that
we
we
weren't
supposed
to.
Do
we
misconfigured
it.
We
had
something
wrong
and
it's
those
cases
where
you
don't
need
to
fix.
One
thing
you
need
to
fix:
a
hundred
and
insights
can
save
that
entire
fleet
or
the.
D
D
All
these
operators,
all
these
components,
being
degraded
firing
an
alert
because
the
alerts
are
usually
component
focused
within
science,
we're
looking
at
the
whole
cluster
and
we're
combining
these
alerts
with
additional
conditions,
with
the
additional
things
that
we
see
on
the
cluster
and
only
after
that,
we're
giving
you
the
recommendation.
So,
but
what
happens
with
these
recommendations
that
again
they're
more
spot-on,
they're
looking
at
the
root
cause?
And
if
you
fix
these
things,
all
these
20
alerts
will
suddenly
disappear.
D
E
D
So
we
use
two
sources
of
information
here.
We
use
the
telemetry
data
that
are
continuous
and
we
use
that
a
lot
for
the
internal
analyzes
katya
will
be
talking
about
it
again,
but
we
use
insights
operator,
which
is
a
snapshot
of
additional
piece
of
information
that
we
collect
every
two
hours
and
every
two
hours
you
immediately
get
new
recommendations,
new
results
of
the
analysis,
the
reason
for
that.
B
D
D
C
D
It's
what
you
ask
about
garnier
at
the
beginning,
insight
is
a
set
of
services
instead
of
functionality,
it's
basically
our
connection
to
console.
and
to
redhead
services
and
and
we're
using
insights
as
a
broader
term
for
multiple
things,
including
additional
features
that
you
get
from
red
hat
and
a
reminder
here
as
well.
All
the
services
that
we're
talking
about
they
are
free
of
charge.
And
again
these
are
value-added
services
that
we
offer
to
every
single
openshift
user.
A
A
I'm
I'm
really
interested
in
the
huge
data
problem
that
you've
solved,
because
I've
only
been
a
bit
of
a
a
window
shopper
when
it
comes
to
looking
at
all
the
work
you're
doing
in
cloud
dot
and
all
of
the
services
that
live
there
to
make
this
happen
and
all
those
data-driven
services.
I
think
you
had
saw
some
dashboards
early
earlier.
So
I
think
you
have
some
data.
Do
you
you
guys
want
to
take
it
away.
D
D
All
right,
so
I'm
sharing
my
screen
right.
Can
you
actually
see
it
and
there
we
go
yep,
perfect,
perfect,
so
just
that
we
know
what
you're
talking
about.
So
insights
is
a
set
of
services
again
that
you,
as
a
customer
you'll,
find
them
on
console.com
openshift.
Once
you
log
in
here
you'll
see
all
your
clusters
and
you've
probably
used
the
hybrid
console
already,
because
you
wanted
to
install
your
cluster.
You
probably
obtained
the
pulse
secret
from
this,
this
ui.
D
What
is
interesting
here
so
this
is
my
demo
account.
So
don't
get
too
hot
to
what
you
see
here,
because
some
of
the
clusters
are
a
little
bit
fake.
But
what
you
see
here
is
my
list
of
clusters.
I
have
some
clusters
that
are
still
connected.
There
are
still
active
some
there
are
older
ones
and
for
each
of
the
cluster
I
actually
have
a
lot
of
information
that
you'll
see
here
and
what
we're
actually
interested
in
are
these
things
like
advisor
recommendations,
cause
brain
ground.
These
are
the
inside
services.
D
So
again,
these
value
add
free
of
charge,
additional
services
that
you'll
get
with
your
openshift
installations.
You
also
find
them
here
on
the
under
the
red
head
insights
menu.
I
already
mentioned
that
there
are
three
main
services
that
we're
looking
at
right
now
advisor.
These
are
the
proactive
recommendations
that
we've
just
talked
about
subscription
management.
This
is
how
you
manage
your
subscription
of
not
only
your
openshift
clusters,
but
also
what
you're
running
on
your
cluster
and
can
do
some
additional
magic
and
cost
management.
D
This
is
a
huge
component
of
insights
and
we
should
probably
have
a
dedicated
session
for
this
one
as
well,
because
it
has
a
lot
of
it.
It's
completely
different
view
on
your
cluster.
It
is
looking
at
the
cost
of
your
infrastructure,
the
cost
of
your
different
projects.
You
can
do
a
lot
of
magic
around
labeling
projects,
different
cost
models,
exploring
what
you're
spending
for
the
interesting
thing
is
that
here
that
the
all
the
data
that
openshift
offers
you
we
can
use
it
in
different
ways.
D
Proactive
support
would
be
one
of
them,
but
then
you
can
even
use
it
for
additional
things,
like
cost
management,
to
understand
how
much
you're
actually
spending
on
your
cluster.
I'm
gonna
click
real,
quick
here
on
the
overview.
This
is
an
important
thing
for
me.
Advisor
is
telling
me
that
I
have
a
set
of
recommendations
here
of
different
criticality
and
different
categories,
and
I'll
might
notice
that
okay,
so
security
ones
are
those
that
I
really
care
about.
So,
let's
quickly
look
at
the
security
recommendations
that
are
available
for
my
clusters.
D
In
this
example,
I
filtered
across
all
the
clusters
all
the
recommendations
that
are
labeled
as
success
as
security,
and
I
can
see
that
this
thing,
the
authentication
operator
degraded
because
there's
a
wrong
cluster
like
proxy
settings.
This
sounds
like
a
serious
one,
so
I'm
gonna
look
at
more
details
here.
D
I
have
a
lot
of
information
about
the
impact
and
the
likelihood
of
this
issue
actually
impacting
my
upgrade
the
performance
of
the
cluster,
but
I
can
also
go
here
and
see
the
additional
details,
including
all
the
clusters
that
are
impacted
by
this
one,
and
when
I
look
at
this
cluster,
this
thing
actually
gives
gives
me
detailed
information
about
what
is
actually
happening
on
the
cluster.
Why
is
the
issue
present
there
or
why
do
we
think
that
this
is
an
issue
steps
to
resolve?
D
In
this
case,
it's
linked
to
knowledgebase
article,
sometimes
you'll
find
here
additional
information
and
openshift
documentation
like
actually
we
do
have
an
additional
information
here,
an
openshift
documentation,
but
quite
often
again,
you'll
find
here
specific
steps
how
to
resolve
a
problem.
Let's
look.
D
D
So
again,
if
it's
possible,
we
always
try
to
give
you
these
specific
steps,
how
to
resolve
the
problem,
and
I
believe
this
is
gonna.
Maybe
the
issue
that
you
hit
on
your
cluster,
that
you
might
have
an
issue
with
one
of
the
workers
and
because
of
that,
you
are
not
able
to
upgrade
right.
So
this
this
would
be
the
case
where
the
upgrade
won't
be
allowed,
because
something
like
this
happens
here.
D
I
can
look
at
all
the
recommendations
here.
I
can
do
additional
magic,
additional
filtering
again
something
I
I
totally
recommend
to
watch,
but
here's
the
thing
you
probably
don't
go
to
consolered.com
openshift.
Every
single
day-
and
you
don't
have
to
the
insights
recommendations-
are
integrated
into
openshift
web
console.
If
you
open
the
openshift
web
console
dashboard
or
the
landing
page
you'll
find
their
information
about
the
number
of
insights
recommendations
available
for
your
cluster.
D
Similarly,
in
advanced
cluster
manager,
for
all
the
clusters
that
you
manage
through
acm
you'll
find
information
about
all
the
insights
recommendations
for
each
individual
cluster
and
acm.
Integration
is
actually
much
more
advanced.
You'll
find
all
the
details
about
every
single
recommendations
right
with
an
acm
and
similarly
to
this
ui
you'll
find
also
the
the
remediation
steps
specific
to
the
cluster.
A
A
D
E
C
A
customer
of
the
azure
red
hat
open
shift
right.
You
will
get
specific
guidance
for
that
particular
flavor
of
openshift
and
all
these
recommendations
associated
if
you're
a
customer
of
different
platform
that
is
ready
to
support
it.
You
will
get
it
as
well
and
you
will
get
it
in
all
of
the
uis
that
that
you
are
potentially
utilizing
not
only
the
console.trender.com.
E
Yeah-
and
I
was
just
thinking
about
that-
you
know
this
when
we
get
alerts
and
I'm
not
I'm
not
bad
mounting
a
lot.
We
all
know
a
lots
are
critically
important
are
very
much
needed,
but
here
I
am
being
flagged.
There
is
a
problem,
the
person,
the
body
who
is
flagging.
The
problem
knows
why
I
have
the
problem
they're,
giving
the
reason
they're,
educating
me
and
telling
me
how
to
solve
it.
What
more.
E
D
There's
has
been
a
talk,
but
no
work
done
so
far,
and
the
thing
is
that
it's,
it's
probably
for
a
longer
discussion,
but
the
the
real
thing
is
that
we
need
to
be
sure
that
if
we
provide
some
recommendation
and
we
want
to
deploy
it
with
ansible
or
some
other
way,
we
want
to
make
sure
that
the
dynamic
environment
of
kubernetes
and
openshift
won't
change.
This
one
interrupt
this
or
want
somehow
impact
this,
so
we
haven't
really
figured
out
a
good
way
how
to
do
this.
D
D
All
right
and
again,
additional
services,
including
cost
management,
including
subscription-
I
just
gonna,
really
quickly
show
cost
management.
This
is
where
I
was
telling
you
that
we
have
a
different
view
on
the
same
data.
I
can
look
at
each
project,
each
name
space
and
my
clusters.
How
much
it's
costing
me.
I
can
look
across
my
clusters
on
which
of
these
clusters,
on
which
platform
are
the
most
expensive.
D
The
cost
models
are
directly
picked
up
from
the
public
cloud
apis
or
I
can
create
my
own
cost
model
if
I'm
running
openshift
on-prem
and
I
can
do
additional
magic
around
that-
I
think
the
most
cool
thing
is
that
I
can
look
at
the
projects
that
are
running
on
my
clusters
and
I
can
see
some
information
about.
How
much
does
the
project
cost
is.
How
much
is
it's
requesting
cpus
and
memory?
How
much
what's
the
utilization
there
and
I
can
actually
do
this
across
the
project?
B
Now
yeah,
so
I
wanted
to
talk
about
what
we
actually
do
with
that
data
internally,
as
radice
was
mentioning
like
his
example
with
the
drone
a
lot
of
applications.
When
you
install
them,
they
ask
you:
do
you
want
to
send
the
data
to
some
nameless
company
and
now,
after
work
with
that
data?
For
some
years,
I
see
the
value
and
how
it
can
be
used
for
actually
betterment
of
the
product?
And
what
I
really
like
is
when
the
company
is
really
transparent,
with
what
they're
going
to
do
with
the
data.
B
B
We
can
see
if
certain
insights
rules
are
hitting
or
not,
and
we
know
what
versions
those
clusters
are
running,
so
we
can
actually
try
and
check
if
we
release
a
newer
version.
If
a
certain
alert
starts
heating
more
often
than
it
was
before,
so
we
actually
do
anomaly
detection
on
each
of
those
signals
that
we're
getting
so
for
alerts,
operator
conditions
when
they're,
failing
like
when
they're
progressing
for
too
long,
when
they're,
not
available,
degraded
and
for
insights
rules,
hits
we're
calling
them
symptom.
B
So,
let's
just
so,
we
create
a
data
set
out
of
it
and
after
that
we
aggregate
the
data
we
dice
and
slice
it,
and
we
do
anomaly
detection
and
it
runs
every
day
on
the
latest
data
and
what
happens
afterward.
Whenever
an
anomaly
is
detected,
it
would
send
us
a
slack
message
with
information
about
an
issue
detected,
so
we've
been
running
it
for,
I
think,
a
little
bit
over
a
year,
so
we
were
able
to
get
rid
of
false
positive.
B
D
B
B
So
if
we
see
that
20
clusters
of
the
same
customers
are
hitting
that's-
probably
not
the
products
issue,
so
it's
we
receive
a
message.
We
start
talking
and
pulling
the
owners
of
that
component.
So
we
start
the
discussion.
We
file
a
new
bug
or
we
improve
the
detection
or
we
improve
the
data.
So
this
is
where
the
discussion
is
started.
D
B
Yeah,
so
this,
whenever
some
notification
has
some
information,
but
usually
we
need
to
pull
more
like
if
a
specific
platform
is
affected.
If
a
specific
installer
is
a
problem,
what
are
the
other
symptoms,
alerts,
operator,
conditions
and
so
on
are
happening
on
the
same
cluster,
so
we
have
like
an
rca.
B
Notebook
in
this
case,
so
for
the
same
alert
that
you
saw
a
notification
there
in
slack,
we
see
that
it
was
hitting
on
4.11
and
we
can
see
so
this
is
if
it
was
if
the
cluster
is
affected
were
installed
by
upi
api
hive
assisted
installer
and
which
platforms
are
affected.
So
here
we
see
that
aws
clusters
installed
by
hive
are
affected
disproportionately
affected.
B
So
it's
on
the
these
it
map
is
relative,
so
something
red
means
that
it's
a
bigger
problem.
So
in
this
case
it
was
we
have
managed
fleet.
So
that
was
a
specific
issue
with
the
managed
we
can
see
how
many
our
clusters
are
hitting
the
problem.
How
many
are
upgraded
versus
just
installed?
What
are
the
other
issues
we're
seeing
in
the
same
clusters?
So
usually,
when
you
see
a
human
set
stack,
that
means
it
cannot
be
rolled
out
properly.
B
We
also
see
that
miss
the
scheduled,
so
now
that
graph,
I
would
say,
is
a
bit
boring.
So
quite
common
alert
is
not
clock,
not
synchronizing.
So
that's
when,
on
the
nodes,
crony
is
not
configured,
so
there
is
a
time
drift
and
you
can
see
so
hit
rate
here
is
in
percentage.
So
that's
how
many
of
the
clusters,
on
a
specific
version
are
hitting
firing
that
alert.
You
can
see
that
it's
kind
of
all
over
the
place,
because
that
means
there
is
no
real
dependency
per
open
shift
version.
B
That's
the
so
whenever
we
detect
a
spike,
this
is
the
green
line.
That's
when
we
send
the
notification,
so
here
the
blue
one
is
the
signal
that
we're
getting
from
the
clusters
based
on
that,
so
the
like
most
of
the
data
science
model
that
get
successful
in
production,
it's
10,
science,
90
engineering,
so
the
modeling
behind
that
is
quite
simple.
So,
based
on
the
data,
we
calculate
the
rolling
mean
of
the
last
10
data
points,
and
we
calculate
this
is
the
standard
deviation
from
that
mean.
B
E
B
B
So
that's
basically
how
the
detection
look
from
the
inside.
So
whenever
something
is
detected,
we
get
a
message
in
slack
we
can
fire
up
this
more
detailed
investigation
check
which,
if
there
is
some
pattern
on
which
clusters
are
affected,
we
also
had
an
notification
for
4.11
the
latest
version
that
was
actually
happening.
It
was
ingress
related
and
it
was
hitting
only
on
cloud
platforms,
mostly
aws,
nutanix
and
so
on,
so
that
was
basically
caused
by
ingress,
not
being
configured
correctly
and
cloud
credentials
were
also
failing.
B
And
you
can
see
the
issues
happening
at
the
same
time,
the
another
quite
common
one
is
issues
problems
with
etcd.
Basically,
cd
is
the
database
that
gets
accessed
through
cube
api.
So
if
there
are
issues
with
the
cd,
you
will
definitely
see
that
and
again
we
see
a
choppy
graph
because
there
are
no
bugs
in
the
product.
That
depends
on
the
version,
but
it's
a
quite
common
issue.
B
The
graph
in
this
case
is
not
really
interested,
but,
for
example,
if
we
check
what
are
the
co
occurring
alerts
symptoms
for
the
same
issue,
usually,
if
it's
a
d
is
slow,
we
would
have
problems
with
accessing
cube
api.
So,
whenever
a
cd
this
alert
is
hitting,
we
would
also
see
cube
api
error
budget
burn.
B
Usually
it
starts
happening
like
half
an
hour
or
an
hour
after
this
one
starts,
and
we
can
also
see
that
at
least
one
half
of
the
clusters,
where
scd
is
being
too
slow
to
respond.
We
also
have
that
an
alert
that
control
plane
is
quite
overloaded,
so
all
of
them
are
related
by
looking
at
what
are
the
other
issues
happening
on
the
same
clusters?
We
can
more
or
less
see
if
it's
a
infrastructure
issue,
if
it's
specific
deployment
procedure
that
the
customers
would
use
and
so
on
so
can.
E
B
We're
trying
to
so
with
this
investigation
we
always
try
to
understand
if
it's
the
customer
infrastructure
ratio
or
a
product
one
so
by
looking
at
the
concurring
things
at
the
graph.
So
when
there
is
an
actual
bug
in
the
product,
we
would
see
it's
kind
of
flat
and
then
it's
like,
we
see
a
quite
pronounced
spike,
so
we
detect
it
early
and
we
address
it.
So
basically,
this
is
the
last
line
of
defense.
I
would
say
so
we
have
whenever
we
release
a
new
version.
We
do.
B
We
have
unit
tests,
we
have
integration
tests,
we
have
quality
assurance,
folks,
doing
the
testing
and
then
after
it's
released,
we
are
monitoring
how
the
clusters
that
are
being
installed
or
upgraded
are
behaving.
If
you,
if
we
see
something
out
of
the
ordinary,
we
can
spot
it
before.
Hopefully,
the
customers
detected
themselves.
So
the
problem
is.
B
Yeah,
so
we
catch,
so
we
don't
catch
a
lot
with
this
detection,
but
it's
a
good
thing.
That
means
the
product
is
of
a
really
good
quality,
because
you
can't
catch
everything
during
quality
assurance,
because
we
run
on
a
lot
of
different
hypervisors.
We
run
bare
metal,
we
run
in
the
cloud.
We
have
different
installers,
different
types
of
deployments
and
so
on.
So
we
cannot
test
all
of
the
possible
combinations.
We
had
a
really
fun
bug
a
couple
of
months
ago
found
through
this.
B
It
was
if
a
cluster
started
in
like
4.1,
then
it
was
upgraded
through
each
minor
version
and
if
it
had
some
specific
feature
enabled
in
one
of
the
cluster
operators,
then
it
would
have
hit
a
very
specific
bug.
So
we
cannot
find
those
issues
through
qa,
because
the
way
to
reproduce
is
a
bit
obscure,
but
we
would
see
that
through
this
mechanism,.
A
That's
amazing
and
then
what
you
were
showing
here
also
is
a
really
interesting
point.
Where
you
can
use
the
insights
you
you
get
to
see
a
symptom
and
then
track
follow-on,
symptoms
or
track
other
correlated
behaviors
and
eventually
build
a
pattern
that,
like
what
you
just
said,
build
a
pattern
or
build
an
exception
case
or
try
to
figure
out
why
only
these
clusters
were
affected
and
that's
that
data,
where
you
know
you
a
customer,
doesn't
have
to
have
one
or
20
clusters
to
try
try
to
retry
and
produce
an
error.
A
We
can
say:
you've
had
this
problem,
12
12
other
people
that
seem
to
have
a
similar
configuration
based
on
our
insights
or
this
one
flag.
Have
this
problem
and
actually
diagnose
and
resolve
that
issue
at
a
product
level
for
you,
without
the
customer
having
to
bang
their
head
against
the
wall
for
a
while.
That's
amazing.
D
And
I'm
going
to
comment
real,
quick
on
the
tool
of
choice
here,
jupiter,
because
this
this
is
what
I
like
about.
What
katja
is
showing,
because
if
you
actually
are
suspecting
certain
thing
on
your
cluster
based
on
these
symptoms
with
jupiter,
you
can
you
can
drill
deeper
right.
You
can
look
at
specific
things.
Cacho
is
showing
the
cloud
performance
here,
but
if
we're
looking
at,
let's
say
networking
issue,
we
want
to
look
at
the
distribution
of
network
plugins
as
an
example
right
our
distribution
of
specific
configuration.
D
So
this
is
where
again
like,
by
drilling
deeper
into
these
components
and
into
these
symptoms
using
a
tool
like
jupiter,
you
can
eventually
find
the
root
cause
and
the
single
configuration
related
to
4.1
installation
back
in
the
days
right.
This
is
where
you
have
really
have
to
go
into
the
data
and
use
a
tool
like
this
to
find
it
out.
A
A
B
Is
a
highly
processed
data,
so
we
have
the
archives,
then
they
get
processed.
So
the
archives
are
uploaded
to
kafka.
Then
they're
processed.
We
extract
some
information,
then
we
create
daily
aggregations.
Then
we
filter
out
some
clusters
that,
for
example,
we
for
this
analysis
we
filter
out
clusters
that
didn't
finish
the
installation
successfully.
So,
in
the
end,
it's
like
the
more
you
clean
the
data,
the
better
date.
The
data
is
so
basically
we
get
less
false
positives.
So
with
this
I
would
say
it's
the
final
data
that
goes
into
these
dashboards.
C
Yeah
bear
in
mind
that
insights
operator
archive
on
its
own
isn't
isn't
a
large
data
data
package
right.
It's
roughly
200
kilobytes
that
you're
sending
every
every
two
hours,
unlike
the
mass
getter
archive,
which
contains
all
the
pieces
of
data
and
everything.
That's
that's
required
by
our
support
staff
and
debugging
quite
complex
issues.
D
Yeah-
and
I
have
to
comment
here
on
one
thing-
data
privacy
and
data
security
are
really
important
to
us,
and
this
is
the
reason
why
we
never
store
the
raw
archives.
We
aggregate
information
as
much
as
possible
and
don't
allow
individuals
to
look
at
your
specific
cluster
and
all
the
history
of
the
data.
B
And
the
notebook
I
was
showing
this
was
basically
for
debugging
those
notifications
in
slack,
but
we
also
have
dashboards
in
our
tool
of
choice,
superset,
which
is
open
source
where
we
can
look
at
the
same
symptom,
so
see
some
more
graphs
and
here
so
that
detection
is
based
on
openshift
version.
So
we
want
to
catch
new
bugs
in
newer
openshift
versions,
but
on
this
dashboard,
what
I
also
find
useful
is
we
have
the
hit
rate
timeline.
B
So
sometimes
we
see
that
certain
problems
start
hitting
on
a
specific
date
and
then
couple
of
days
it
goes
down.
We
had
that
when,
for
I
don't
remember
what
was
the
name
of
the
alert,
but,
for
example,
there
was
an
issue
with
ocm,
so
none
of
the
clusters
were
able
to
upload
things
to
ocm,
so
we
saw
a
specific
alert
spiking
up
like
this.
After
we
detected
the
issue
it
was
addressed,
so
it
was
going
down
again
so
usually
for
the
aggregation
by
version.
B
We
would
see
it
kind
of
flat
here
why
I
find
this
graph
really
interesting.
So
you
see
this
is
for
the
same
node
clock,
not
synchronizing.
So,
basically,
the
clusters,
the
nodes
on
those
clusters,
don't
have
crony
configured
properly,
so
they
need
to
configure
ntp.
So
what
we
see
here
is
this
is
the
last
30
days.
So
we
see
that
4.8
4.9,
4.10
behaving
so
quite
flat.
That
means
there
is
no
time
relation,
but
we
see
four
eleven
just
appearing
out
of
nowhere,
spiking
and
then
going
down.
B
So
what
that
means
is
this
was,
I
think,
august
10th.
Maybe
someone
can
correct
me.
That
was
the
ga4
for
11.,
basically
a
lot
of
clusters.
A
lot
of
customers
started
installing
it
checking
it
out,
and
at
that
point
that
were
mostly
dev
and
staging
clusters.
They
didn't
care
about
configuring
ntp
on
workers,
so
we
see
higher
than
usual
occurrence
of
nodes,
not
having
ntp
server
configured,
and
then
it
goes
down
once
we
get
more
production
like
and
more
cared
for
clusters.
B
Yeah
for
support,
we
have
more
dashboards,
so
we
have
so.
This
is
basically
a
graph.
I
picked
specifically
kind
of
sick
cluster
with
a
lot
of
issues,
so
these
are
all
of
the
alerts
happening
on
this
cluster
in
the
last
30
days
and
you
can
see
the
timeline
so
we
see
here,
I
hope
it's.
I
can
try
to
make
it
bigger.
B
B
I
think
there
were
also
issues
with
think
image
registry,
but
I'm
not
sure
where
it
is,
so
you
can
see
that
cluster
operator
conditions
you
see
that
image
registry
is
not
available.
It's
still
installing
it's
degraded,
so
you
can
actually
see
how
the
alerts
you
have
are
correlated
with
the
cluster
operator
conditions
and
the
support
rules
as
well.
So
this
is
basic
information
that
support
folks
can
use
to
have
a
like
overview
of
the
cluster.
D
D
C
A
A
D
Either
that
or
what
we've
seen
couple
times
happening
is
customer
opens.
The
support
case
doesn't
give
us
enough
information,
but
because
we
have
the
cluster
id,
we
have
all
the
internal
reports
and
the
second
comment
we're
telling
the
customer
go
fix
this
because
that's
your
problem,
that's
the
root
cause.
C
Yeah,
that
is
right,
especially
especially
if
it's
configuration
issues
our
our
tooling,
that
we
provide
to
support
basically
points
the
direct
line
in
the
email
file
that
needs
to
be
fixed,
you
know,
and
that
that
we
basically
need
to
copy
and
paste
to
our
customer
and
say:
hey,
that's
it.
This
is.
This
is
what
you
need
to
fix.
That's
amazing.
A
A
So
the
fact
that
we
can
say
here's
the
here's,
what
you
need
to
run.
I
still
need
to
open
a
support
ticket
open
a
support
ticket
and
the
support
engineer
is
lot.
You
know
able
to
see,
hear
what
our
insights
say:
here's
the
information
that
we're
able
to
tell
you
very
quickly
how
to
solve
your
problem
and
that's
that's,
that's
actual
hours
saved
of
a
customer's
engineer.
I
think
that's!
That's
just
amazing.
The
customer
gets
a
solution
and
twice
the
time
or
some
some
metric
folks.
This
has
been
amazing.
A
A
The
exciting
the
so
we'll
we'll
go
ahead
and
tease
that
thanks
everyone
for
coming
to
the
show
the
show
archive
will
be
up,
as
always,
we'll
be
back
next
week
with
the
show
on
search.
So
it's
going
to
be
the
search.
That's
in
acm,
it's
open
source,
going
towards
the
open
cluster
management
project,
a
really
cool
tool
around
search
and
then
we'll
be
back,
hopefully
in
another
week
or
two,
with
some
more
insights
discussions
about
updates
and
then
hopefully
have
radic
back
on
and
team
a
little
bit
later.