►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Sponsored Session: Salesforce - Using Sloop for Monitoring Highly Available Services
Speakers: Sana Jawad, Hemanth Siddulugari
Kubernetes cluster’s state is ephemeral in nature and the workloads can run on any nodes or pods. There is a set of great tools available for visualizing “current” state of the cluster. But often times, live site incidents are mitigated and root cause analysis is left for later. This becomes particularly challenging for incidents that happen due to various K8s events since these events are only available for one hour on the cluster. After this, the only way to debug what happened is by correlating various logs and timelines from control plane, hence making it harder to root cause (increases MTRR). Sloop provides a one stop shop solution with a single pane of glass by showing historic view of cluster. In this demo, we will be sharing the top real incidents for which we found the root cause in a matter of minutes using Sloop.
A
A
A
We
hope
by
the
end
of
this
talk,
you
will
learn
some
tips
and
techniques
to
in
how
to
investigate
different
kubernetes
events
in
using
slo.
So,
let's
get
started
here
is
the
high
level
overview
of
kubernetes
cluster.
It
consists
of
multiple
worker
nodes
which
run
containerized
applications
in
parts.
These
nodes
and
pods
are
managed
by
cube
api
server,
which
is
a
part
of
control
plane.
A
Kubernetes
has
great
set
of
tools
for
debugging
issues
currently
happening
in
the
cluster.
For
example,
we
have
prometheus
cube,
dashboard
or
cube
cutting
commands,
but
often
times
when
a
live
site
incident
occurs.
The
main
focus
is
on
the
mitigation
and
root
cause.
Analysis
is
left
for
later.
This
becomes
particularly
challenging
for
incidents
that
happen
because
of
some
kubernetes
events
in
the
cluster.
A
So
there
was
a
need
of
a
tool
which
could
provide
kubernetes
resource
life
cycle
with
time
stamp.
Also
like
show
all
the
related
kubernetes
resources
for
an
event
or
a
deployment
at
one
place
and
also
have
a
very
easy
interface
in
terms
of
bar
charts
and
diagrams,
which
are
very
easy
to
understand
and
get
information.
A
A
Also,
you
can
filter
out
the
resources
for
a
particular
namespace
as
well
as
there
is
an
option
to
basically
filter
by
kubernetes
kind
and
there
you
can
get
the
sorted
result
as
well.
Now,
all
the
all
the
resources
are,
for
example,
like
a
particular
kind
are
depicted
by
one
color.
Here,
the
red
ones
show
deployment
objects,
the
green
ones
show
corresponding
parts.
So
you
can
see
that
there
was
a
deployment
happening
at
this
point
in
time
and
you
can
see
when
this
part
was
created
in
a
very
timely
manner.
A
The
green
pills
here
show
that
there
was
a
kubernetes
event
that
happened
at
this
point
in
time
in
the
cluster,
and
if
you
click
on
it,
you
can
get
the
details.
The
yellow
one
indicate
that
there
was
something
going
wrong
at
that
time,
because
there
was
like
a
warning
or
an
error
in
the
kubernetes
events
that
needs
to
investigation.
A
A
Once
this
information
is
received,
it
is
processed
and
then
stored
in
badger
db.
Badger
db
is
also
an
open
source,
key
value
data
store,
and
basically
you
don't
need
an
external
database
or
anything
here
now
I
will
be
handing
it
over
to
hement
who
would
be
sharing
some
of
our
favorite
live
site
incidents
that
happened
in
salesforce
and
how
we
use
sloop
to
investigate
those
and
get
the
required
information
in
matter
of
minutes.
So
thank
you.
B
The
first
of
these
incidents
is
where
we
have
the
service
go
down,
because
the
the
leader
parts
of
the
service
kept
getting
bounced
right
and
digging
further.
We
found
that
it
was
happening
because
the
ephemeral
storage,
the
empty
dirt
being
mounted
by
the
the
pod,
is
getting
filled
up
by
the
container,
which
is
using
it
inside
the
pot
right
off
the
bat.
B
When
we
started
looking
at
this,
it
was
hard
to
figure
out
why
it
was
happening,
because
the
pod
logs
or
the
container
logs
wouldn't
show
anything
because
this
was
happening
at
the
communities
level,
and
you
know
the
the
different
control
plane
logs,
which
show
the
events
were
helpful.
But
you
know
it
takes
a
lot
of
time
right.
B
Let
us
see
how
this
was
easy
to
figure
out
by
using
the
sloop
ui.
So
I'm
going
to
switch
to
this
loop
ui
view
here.
The
stateful
set
we
are
interested
in
is
the
the
eviction
right.
It
has
four
pods
and
we
see
that
at
this
moment
something
happened
to
eviction
zero,
pod
and
clicking
on
this
and
looking
at
the
events
surrounding
that,
we
can
see
it
got
evicted
and
why?
Because
the
fml
storage
was
getting
full.
B
So
this
points
directly
to
you
know
something
happening
when
the
the
cron
job,
which
was
running
at
the
same
time,
was
filling
up
the
fml
storage
and
we
can
quickly
see
which
mount
point
it
is.
You
know
what
is
filling
it
up
and
change
the
design
so
that
it
doesn't
happen.
So,
as
you
can
see,
the
the
ui
simplifies
looking
up
correlating
the
events
so
easy
right.
B
Let's
look
at
another
incident,
so
in
this
second
incident
we
had
unexpected
restarts
of
pods,
which
were
running
on
nodes
running
in
a
totally
different
zone
than
where
we
were
running
maintenance
during
off-peak
hours
and
digging
further.
What
we
saw
was
that
a
cloud
provider
was
trying
to
balance
the
nodes
when
it
had
inadvertently
brought
up
extra
nodes
in
the
in
the
zone,
where
this
happened
to
compensate
for
the
total
desired
nodes
that
are
supposed
to
be
there.
B
B
The
auto
balance
stance
right
here.
You
can
see
that
everything
was
running
fine
for
this
stateful
set.
We
have
four
pods
and
they
are
auto
dance.
Auto
balance,
dance,
zero,
one
two
and
three
and
at
this
moment,
is
when
the
maintenance
happened,
and
we
know
that
you
know
these
parts
would
go
into
pending
state
to
find
another
node,
but
they
stayed
down
for
a
long
time.
B
They
didn't
find
a
node
to
run
on
and
also
we
noticed
that
the
arabian
stands
one
and
three
had
restarts
they
weren't
even
supposed
to
have
any
restarts
right
to
see
this.
We
move
to
the
part
the
node
view
where
we
can
see
that
this
node,
which
is
supposed
to
be
in
zone
c,
got
terminated
at
the
instance
where
we
saw
that
restart
of
the
the
unexpected
restart
of
the
part
and
same
is
true
with
the
other
node
right.
So
even
this
one.
B
If
you
look
at
the
details,
it
is
in
2c
where
the
termination
happened
and
what
we
noticed
is
that
you
know
the
new
nodes
which
came
up
as
a
result
of
these
two
going
away
were
actually
in
the
unexpected
zone.
So
the
zone
where
they
were
supposed
to
come
up
was
into
a
okay,
so
the
same
is
true
for
the
other
node.
As
you
can
see,
this
one
was
in
2a
right,
whereas
the
ones
that
came
as
replacement
right
here.
If
you
look
at
the
details,
it
was
into
c.
B
B
B
As
far
as
they
are
concerned,
there
are
four
nodes:
they
need
to
bring
it
down
to
two
and
two
were
terminated
which
disrupted
the
service
see
having
this
ui
simplifies,
correlating
these
things
so
easily
and
let's
switch
back
to
the
third
incident.
This
is
where
this
is
less
of
an
incident,
more
of
a
scenario
that
you
can
debug
using
the
spinnaker
ui,
sorry
using
the
sloop
ui,
where,
for
whatever
reason,
if
the
part
is
stuck
in
pending
state,
it
could
be
due
to
failed
web
hooks
failed
mount
points.
B
It
could
be
due
to
remaining
nodes,
not
meeting
the
resource
requirements,
etc.
If
that
were
to
happen
at
offbeat
cars,
where
you
are
not
watching,
and
you
have
to
debug
that
later
on,
then
it
helps
really
to
have
that
history.
In
the
ui,
so
one
such
example
can
be
shown
in
this
loop
ui,
where,
if
I
go
back
to
the
pod
view-
and
there
was
an
incident
where
this
eviction-
one
is
stuck
right
in
pending
state
for
a
long
time
and
digging
through
the
logs
we
can
see.
This
was
due
to
a
failed.
B
Mount
point
see
see
this
message
where
it
says
multi-attach
error,
of
the
mount
point,
and
you
know
so.
These
messages
are
at
your
service
when
you
pull
up
the
ui
to
see
right
away.
You
know
why
that
part
is
in
pending
state
at
that
moment,
instead
of
digging
through
a
bunch
of
logs
trying
to
correlate
exactly
what
happened
at
that
time.
So
in
this
case
you
see
the
mount
point
was
failing
to
attach
and
usually
when
this
happens,
the
part
would
get
stuck
for.