►
From YouTube: Fool-Proof Kubernetes Dashboards for Sleep-Deprived Oncalls - David Kaltschmidt, Grafana Labs
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Fool-Proof Kubernetes Dashboards for Sleep-Deprived Oncalls - David Kaltschmidt, Grafana Labs
Software running on Kubernetes can fail in various, but surprisingly well-defined ways. In this intermediate-level talk David Kaltschmidt shows how structuring dashboards in a particular way can be a helpful guide when you get paged in the middle of the night. Reducing cognitive load makes oncall more effective. When dashboards are organized hierarchically on both the service and the resource level, troubleshooting becomes an exercise of divide and conquer. The oncall person can quickly eliminate whole areas of problems and zone in on the real issue. At that point a single service or instance should have been identified, for which more detailed debugging can take place.
https://sched.co/MPbv
A
A
A
But
I
haven't
finished,
introducing
yet
I'm
I
it's
only
gonna
take
about
30
minutes
to
the
introduction,
so
you're
good
with
five
here
yeah
now,
I've
known
David
for
years
now,
I
work
with
it
worked
with
him
for
a
long
time
and
he's
kind
of
one
of
the
best
UX
guys
I
know
so
so
I
hope
you
really
enjoyed
the
talk.
Thank
you
for
staying
so
late.
This
I
know
this
is
the
last
one
and
David's
promised
he's
gonna.
You
know
cut
it
short,
so
we
can
all
go
and
get
some
beer
so
ran.
B
Alright,
hey
everyone:
if
this
title
of
the
talk
was
very
click,
bTW
I
think
I
succeeded.
There's
a
lot
of
people
here.
I
can
also
disappoint
you
right
away.
You
won't,
you
won't
be
getting
ready,
made
dashboards
to
solve
all
your
kubernetes
problems,
but
I
hope
I
can
inspire
you
to
follow
a
couple
of
practices
that
we
identified,
I
useful
and
so
just
real
quick
about
about
myself.
B
I'm
David
working
at
Cortana,
no
I
used
to
work
with
Tom
already
at
week,
Forks
and
causal
and
now
I'm
a
bit
more
interested
in
UX
and
also
got
really
excited
by
things
like
this.
So
has
anyone
taken
the
metro
fun
on
like
on
on
a
way
to
the
conference
centre,
because
initially
I
thought
well?
Taxi
is
good,
but
it
was
really
really
a
lot
of
traffic
here.
B
B
B
Already
to
me,
it
was
mind-boggling
about
all
learning
about
all
the
concepts,
all
the
different
dimensions
and
the
troubleshooting
process
is
just
so
so
difficult
because
you
can
be.
You
basically
have
to
do
the
troubleshooting
across
a
couple
of
these
dimensions
that
exist
along
these
kubernetes
concepts
and
troubleshooting
to
me
is
as
much
about
finding
the
issue
as
it
is
about
eliminating
where
the
issue
isn't
right
and
all
these
dimensions
the
existence
of
those
just
add
to
that
complexity.
B
So
the
tools
we
use
shouldn't
really
get
in
a
way
of
this.
Sometimes
they
do
especially
when
go
fana
makes
it
really
really
easy
to
modify
dashboards
and
all
of
a
sudden.
You
have
a
thousand
and
then
being
in
that
one
call
position
when
you
have
a
bad
time.
It's
not
really
helping
all
right,
because
what
you
want
to
do
is
you
want
to
have
tooling
that
reduces
cognitive
load
and
doesn't
doesn't
add
to
it
and
I'm
sure.
Everyone
here
has
been
a
situation
where
they
just
modified
a
tiny
thing
on
a
dashboard.
B
We've
already
heard
this
morning
in
a
keynote
or
three
is
a
magic
number
and
yeah
I
just
meant
a
bit
as
well
by
default,
you
probably
just
don't
really
have
a
strategy
for
boarding
and
you
just
sort
of
changing
some
things
here.
Some
things
there.
If
you,
if
you
are
a
bit
more
serious,
then
you
try
to
manage
your
boring
a
bit
more
and
you
try
to
use
methodical
dashboards.
B
I'll
tell
you
but
later
what
this,
what
those
are
and
in
what
I
think
is
very
high,
which
or
a
very
mature
level
is
when
you
try
to
optimize
that
use,
and
you
try
to
actively
combat
that
sprawl
and,
most
importantly,
you
try
to
ensure
that
those
dashboards
are
consistent
by
design.
So,
let's
dive
into
those
just
to
reiterate
real
quick
about
sort
of
bad
practices
and
for
us
it
was
always
duplicating
a
dashboard.
B
So
this
is
one
of
the
also
one
of
the
verse
you
exit
decisions
we
ever
made
was
adding
this
copy
tag
toggle
there,
because
who
here
uses
tags
for
dashboards?
Okay,
yeah,
so
you
probably
had
really
good
intentions
doing
those.
But
then
someone
else
later
came
along
and
cloned
this
dashboard
and
maybe
just
left
this
thing
on
but
modified,
something
that
was
somehow
also
represented,
semantically
in
one
of
your
tags
and
at
that
point,
there's
diverged
and
if
you
later
use
those
texts
to
find
your
things.
B
B
Another
problem,
if
you
don't
have
version
control,
so
basically
what
happens
when
you
do
modification
you
hit
save
if
you
have
the
standard
longer
fauna
instance,
and
you
don't
back
up
your
your
data
or
you
don't
have
your
dashboard,
JSON
inverted
control
and
your
graph
honor
goes
away.
Then
you're
gonna
have
a
bad
time.
B
So
this
is
another
characteristic
of
low
maturity
and
similarly
a
behavior.
That's
that's.
Also.
A
symptom
is
browsing
for
dashboards.
If
you
find
yourself
browsing
a
lot
like
going
through
these
folders
that
I'm
the
introduced
I
mean
they're
really
helpful,
but
sometimes
you
have
to
go
back
and
forth
and
there's
to
find
the
right
thing
and
that's
that's
that's
the
sort
of
behavior
that
for
a
very
mature
observability
platform,
kind
of
want
to
get
away
from
so
we
can
do
better.
Let
me
let
me
tell
a
bit
more
about
what
media
maturity
can
look
like.
B
B
B
So
what
you
can
do
with
Cortana,
for
example,
you
you
can
register
those
as
template
variables
for
that
dashboard,
and
then
you
get
this
nice
drop-down
and
then
you
can
have
a
look
through
all
of
these
instances
and
if
you're
really
clever,
you
can
also
do
this
for
various
data
sources
and
as
a
sort
of
higher
level
template
variable.
And
then
you
can
basically
access
a
lot
of
different
clusters.
B
So,
let's
get
to
methodical
dashboards.
What
does
that
mean?
So
there's
a
couple
of
methods
out
there
that
really
help
you
sort
of
make
sense
of
what
could
go
wrong
in
kubernetes
along
those
areas
dimensions
they
talked
about
earlier
and
one
of
the
popular
ones
are
the
used
method
and
also
the
rep
methods,
also
two
loci
here
again.
B
So
if
you
click
that
right
method
linking
I
see
another
sort
way,
but
what's
more
important,
I
think
is
that
you
don't
really
you
don't
necessarily
have
to
use
any
of
these,
but
it's
good
to
have
a
method.
So
it's
good
to
have
a
method
for
the
situation.
When
one
of
your
developer
teams
comes
to
you
and
says
we
want
to,
we
want
to
add
a
new
app.
C
B
So
here's
an
example
now
for
a
user
method
dashboard
these
are
actually
part
of
the
kuben.
Is
monitoring
mixin
there's
a
link
there,
so
the
slides
are
also
in
the
session
attached,
and
this
is
already
a
pretty
good
view
of
what
a
node
represents
and
also
what
sort
of
troubles
and
node
can
have.
Similarly,
there
is
a
whole
set
of
dashboards
in
this
repo,
so
I
just
took
a
picture
of
what
other,
what
other
sort
of
dashboards
are
there
and,
for
example,
there's
a
new
one
about
persistent
volumes
and
their
little
novelists
there.
B
So
the?
What
we
do
is
that
for
every
graph
that
we
do,
for
example,
this
is
this
cluster
view
broken
down
by
namespaces.
The
breakdown
here
in
these
queries
is
always
the
next
little
down
right,
so
you
can
move
into
the
next
level
by
using
by
using
one
of
these
drill
down
links
in
the
table,
and
this
is
also
part
of
the
kubernetes
monitoring
mixing.
So
you
can
see
how
that's
being
done
and
a
similar
hierarchy,
hierarchical
approach.
B
So
this
is
good,
but
there's
obviously
something
wrong,
because
the
lower
ones
are
red
and
if
we
go
the
other
way.
So
looking
at
the
app
there's
some
red
there,
but
we
know
that
the
app
relies
on
data
from
the
database
and
by
virtue
of
the
database
also
being
red.
We
get
a
bigger
if
we
get
a
hunch
that
the
error
may
be
in
the
database.
So
in
just
this
vertical
structure.
This
vertical
hierarchy
inherently
leads
me
towards
a
hunch
on
where,
where
the
system
is
not
working
on.
B
So
one
thing
to
remember-
or
one
thing
to
keep
in
mind
about
these-
is
that
sometimes
it's
worth
splitting
up
a
service
or
an
app
into
two
different
dashboards,
mainly
because
the
magnitude
can
differ.
So,
for
example,
we
were
on
cortex
at
microfauna
and
cortex
is
a
Prometheus
based
service
and
a
lot
of
data
gets
being
sent
all
the
time.
So
in
let's
say
it's
orders
of
magnitude
more
like
a
thousand
X.
B
B
So,
and,
and
one
more
thing,
one
more
tip
I
want
to
give-
is
to
make
the
charts
themselves
really
expressive
so
that
even
the
previous
ones
in
the
service
hierarchies
debuts
in
color
too,
to
give
you
a
quick
and
then
a
quick
hint
on.
What's
going
on
so
the
top
one
and
top
diagram,
you
can
see
the
only
two
hundreds
are
green
and
the
five
hundredths
or
red
someone
decided
to
color
the
4/4
blue
I.
B
B
So,
just
just
to
go
back
on
template
variables
again.
I
do
realize
that
template
variables
make
it
sort
of
hard
to
to
navigate
through
the
dashboards.
But
the
main
idea
is
also
that
that
this
is
okay
and
that
you
actually
shouldn't
just
navigate
through
them,
especially
if
you
have
three
level
hierarchies
and
that
what
we
actually
want
to
encourage
is
that
you
don't
really
browse
through
the
dashboards
anymore,
but
that
you
use
alerts
to
that
that
have
links
in
them
if
they
give
that
link
directly.
B
The
the
where
should
we
store
the
dashboard
code
themselves
and
there
is
a
couple
of
initiatives
again
sycophantic
going
on
and
the
most
important
rule
revolves
around
improving
the
provisioning
workflow
and
we
want
to
integrate
well
with
github
there
and
as
a
designer
out
there.
So
I
think
if
you're
really
interested
in
this,
it's
it's
worth
clicking
the
link
in
the
slides
and
also
just
go
into
the
issue,
13
8
to
3
and
comment
on
it.
B
Ok,
so
before
we
get
to
the
high
maturity,
I
just
want
to
show
you
another
situation
in
of
the
of
the
Barcelona
Metro
so
who
here
has
seen
these
did
anyone
have?
Did
anyone
also
have
a
little
trouble
there,
because,
because
everyone,
a
bus
or
a
lot
of
people
were
still
asking
to
be
left
handed,
but
I'm
right-handed,
like
I,
think
the
majority
of
people,
but
maybe
I'm
wrong
there
and
so
having
to
take
the
ticket
out
having
it
in
my
right
hands
and
then
putting
it
into
this
left.
B
Turnstile,
Swiper,
I,
don't
know
that
was
a
weird
contact,
though
not
all
subway
entrances
have
it
on
the
left.
Some
of
them
have
it
on
the
right:
ok,
yeah,
sir
I
guess
a
good
another
good
segue
here
now
is
these
people
that
put
it
in
place.
They
knew
fully
well
that
a
lot
of
people
all
right-handed.
They
decided
to
do
this
anyway,
which
you
also
find
sometimes
in
a
DevOps
organization
where
people
know
good
practices,
but
they
still
deviate.
B
So
then
the
question
becomes:
how
can
we
achieve
consistency
by
design
and
before
we
get
into
this?
Let's
quickly,
G,
let's
quickly
talk
about
sprawl
again,
yeah.
Just
look
at
how
happy
this
guy
is
trimming
the
hedges
just
because
think
about
about
all
these
all
these
dashboards
that
you
create
all
these
one-offs,
they're
gonna,
be
in
the
way
that
they're
they're
gonna
be
a
sprawling
hatch
that
they're
definite
leads
to
be
trimmed
and
in
a
very
mature
dashboarding
organization.
B
You
will
have
someone
that
goes
around
and
and
removes
dashboards
that
are
no
longer
being
used,
or
you
have
a
very
good
review
process
that
that
says
only
dashboards
that
are
proof
can
go
into
the
master,
the
master
Cortana,
for
example,
and
then
another
cool
feature
that
we're
thinking
about
internally.
Ingre
fauna
is
to
have
meta
analytics
to
basically
track
the
use
of
dashboards
and
of
queries.
To
give
you
also,
the
idea
on
which
of
these
could
be
able
to
move
potentially.
B
So
consistency
by
design
there's
a
lot
going
on
on
this
slide,
but
I
want
the
main
takeaway
is
that
there
are
libraries
out
there.
Scripting
libraries
that
give
you
high
order
functions
to
generate
to
generate
certain
types
of
dashboards
and
the
important
thing
there
is
that
those
functions
can
encode,
for
example,
a
query
panel
and
they
they
will
have
options
attached
to
them.
B
And
if
you,
if
you
didn't
only
use
this
little
function,
call
giving
it
the
data
source
and
the
career,
you
can
ensure
that
all
the
rows
and
all
the
dashboards
that
have
been
created,
they
will
all
share
the
same
style
and
there's
no
longer
a
fight
of.
Should
we
use
line
fields
or
should
we
use
not
line
fields?
And
then,
just
by
using
these
patterns,
you
guarantee
across
the
organization
that
that
your
more
panels
are
essentially
similar
enough,
so
that
people
can
find
their
way
really
quickly.
B
B
B
Yeah,
a
lot
of
those
resources
exist
and
I
think
you
should
make
use
of
this,
even
though,
even
if
you
don't
so
just
making
these
mix-ins
are
the
ones
I've
been
showing
you
they've
written
in
JSON
it,
but
even
though
they're
written
in
JSON
it
they
can
still
give
you
or
you
can
still
extract
the
queries
in
there
and
use
them
in
your
own
dashboard
in
Germany.
So
it's
really
so
it's
a
really.
It's
a
really
good
resource
to
look
at
how
people
people
decided
to
monitor
Prometheus.
B
There
are
other
mix-ins,
for
example,
for
console
or
other
services.
So
it's
worth
definitely
to
watch
this.
This
talk
Oh,
incidentally,
also
might
oh,
okay,
yeah
and
in
the
future.
We
also
really
like
a
workflow
where
we
can
in
the
browser.
I
have
also
haven't
an
editor
to
modify
the
JSON
live,
so
you
can
then
have
that
be
part
of
the
dashboards
code
journey
to
from
the
browser
to
your
PR
right
away
and
then
ideally
also
use
these
higher-level
function.
B
Languages
to
do
to
basically
have
smaller
exchange
sets
and
have
a
compiler
somewhere
that
renders
the
JSON
so
that
you
can
have
the
live
preview
again,
but
that's
a
bit
it's
a
bit
future
stuff
cool.
So
it
is
a
quick
summary
again
it's
good
to
have.
We
have
a
strategy
for
dashboarding,
it's
good
to
start
with
the
goal
of
managing
the
use
of
methodical
dashboards,
and
then
the
next
step
can
be
consistency
by
design.
B
I
think
this
is
kind
of
a
summary
slide
of
what
what
I
had
so
far
and
then
I
basically
want
to
leave
you
with
two
with
two
kind
of
main
takeaways.
One
of
them
is
the
dashboarding
method
that
you
adopt,
like
should
definitely
not
get
in
a
way
of
people
trying
to
use
the
dashboards
and
the
second
one
is
don't
be
the
persona
metric
of
dashboards.
A
Thank
you
very
much.
David
I
was
the
one
who
chose
blue
for
400s.
Okay.
What
color
should
they
be?
Is
the
first
question
yeah
purple
purple?
Okay.
Well,
luckily,
as
we
use
JSON
it,
you
can
change
it
in
one
place.
That's
true!
Anyway,
right
then
who's
got
a
question
for
David.
We've
got
a
few
minutes
on
someone's
got
a
question
you
go
next.
B
C
Hey
thank
first
of
all
thanks
for
the
talk
in
the
great
tips.
I
think
everyone
agrees
that
this
is
a
should
be
way
to
go
I'm
from
one
of
these
organizations
where
we
have
literally
literally
1800
dashboards,
we
bachelor
fana
to
actually
deal
with
it.
So
every
upstream
version
we
have
to
match
the
queries
yeah
with
it.
So
this
or
is
from
a
a
shared
graph
Anna.
So
every
team
is
free
to
do
their
own
stuff.
C
B
Yeah,
so
you
sort
of
answered
a
question
yourself
already,
because
your
training
is
the
internal
training.
Is
the
first
step
there
to
make
those
developers
aware
that
that
whatever
they
add
to
their
graph
Anna
instance,
can
also
get
in
a
way
for
other
people
getting
to
the
results
that
they
want
all
right.
So
I
guess
maybe
sharing
the
video
of
this
talk.
I
guess
my
first
step,
but
then
also
I
can
I
can
also
I
guess,
set
your
bit
at
ease
that
1800s.
It's
not
the
biggest
number
that
we've
heard
so
good
next.
B
A
Wanted
to
call
out
Frederick
here
and
Jack
here
and
is
Matthias
in
the
audience
yeah
Matthias
over
here.
These
guys
were
I.
Don't
just
do
the
mix
ins
on
my
cert
on
my
own,
like
these
guys
have
helped
me.
They
really
run
it
now,
I'm
too
busy,
but
I
wanted
to
call
them
out
for
the
kubernetes
mix.
Things
has
been
super
helpful
and
you've,
probably
already
using
it,
and
you
don't
know
it
right.
If
you're
using
the
Prometheus
operator,
that's
the
default
set
of
dashboards
that
come
with
it.
So
thanks
very
much
Ellison.
Thank
you.