►
From YouTube: CrashLoopBackoff, Pending, FailedMount and Friends: Debugging Common Kubernetes Cluster
Description
CrashLoopBackoff, Pending, FailedMount and Friends: Debugging Common Kubernetes Cluster and Application Issues [B] - Joe Thompson, Oteemo
Nothing is more frustrating than deploying a shiny new application on Kubernetes and having it fail immediately (usually five minutes before the big demo). Is it a problem with the pod network? Pods Pending or in CrashLoopBackoff, Services not serving, images not pulling? Maybe you're just plain out of resources. If you're new to Kubernetes, figuring it out from scratch can take hours you don't have. We'll show you how to dig in, identify the problem, resolve it, and learn what to watch for so you aren't taken by surprise next time.
About Joe Thompson
I'm a senior consultant in the container practice at Oteemo, a consultancy specializing in innovative IT automation, and also a co-organizer of the NOVA Kubernetes Meetup. Prior to Oteemo I worked at CoreOS and Red Hat, providing practical solutions and training in and for Kubernetes and other cloud environments. My experience in IT operations and engineering stretches from present-day cloud and virtualization back to before you could Google error messages to find out what the problem was, and my blood type is caffeine-positive.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Oh,
we
are
hiring.
This
is
my
team
lead
Sam
right
in
the
front
of
the
room,
my
blood
type
is
caffeine,
positive
and
there's
my
contact
info
down
there.
Should
you
care
to
contact
me
if
only
to
say
god
that
presentation
sucked
it
was
a
waste
of
my
time
first
thoughts,
so
Brad
giesemann,
who
some
of
you
may
have
saved
his
presentation.
I
know
Sam
has,
if
you
read
his
slides
one
of
the
things
that
he
goes
over.
A
Is
you
have
to
kind
of
get
in
the
right
frame
of
mind
for
some
things,
so
I
wanted
to
give
you
some
first
thoughts
when
you
kind
of
dive
in
to
debug
kubernetes
first
thing
is
it's
not
rocket
science
yeah,
it's
a
distributed
system
that
doesn't
mean
that
it's
actually
hard
to
get
the
info
that
you
need
to
get
out
of
it.
Nothing
that
we're
doing
is
really
new,
we're
just
poking
at
the
system
and
seeing
what
falls
out
some
of
the
tools
are
new.
Some
of
the
pieces.
You
prober
new,
that's
about
it.
A
This
is
how
it
feels
when
your
application
deployment
fails.
Lights
are
lighting
up,
red
people
are
getting
woken
up
by
phone
calls,
sirens
are
going
off
and
it's
tempting
to
just
you
know,
kind
of
throw
your
hands
up
and
go
take
a
deep
breath.
Don't
panic?
Do
we
have
any
IT
Crowd
fans
here?
Yep,
you
know
about
the
little
book
of
calm
then
find
the
little
book
of
calm.
A
You
know
take
whatever
other
advice
from
classic
works
of
fiction
and
film
you
care
to
take,
and
then
let's
fix
it.
The
first
thing
we
have
to
do
is
gather
info
and
that's
what
this
is
about.
Then
you're
gonna
form
a
plan,
you're
gonna
test
and
execute
your
plan.
So
how
do
you
gather
the
info?
Well,
let's
talk
about
that.
The
first
thing
you
want
to
do
if
you've
done
stuff
with
cube
control.
A
We've
now
been
told
that
that's
the
official
name,
if
you've
done
stuff
with
cube
control,
you've
probably
done
a
good
control,
get
cube.
Control
describe
those
are
nothing
new.
Sometimes
that's
enough!
You
you
do
a
cube
control.
Describe
you
see
an
image
pull
air,
you
go.
Oh
I
forgot
to
put
the
image
pull
secret
and,
of
course
it's
not
pulling
the
image
you
go
off
and
you
fix
it.
If
not,
let's
talk
about
logs
I
know,
my
humor
is
terrible.
A
The
first
thing
that
a
lot
of
people
don't
know
about
is
coups,
ETL,
get
events,
an
event
is
actually
a
resource
type
in
kubernetes,
and
so
when
you
do
cube,
CTL
get
events,
it
will
actually
list
them
out
just
like
it
would
list
another
resource
and
give
you
kind
of
a
summarized
view
of
what
it's
seen
and
how
often-
and
when
was
the
last
time
it
saw
it
now.
This
looks
like
terrible
formatting.
That's
because
when
you
see
this,
it
usually
is
terrible.
Formatting
like
that.
This
is
only
compressed
a
little
bit
in
line
length.
A
So
you
know,
if
you
have
the
ability
to
widen
your
terminal
way
out,
that's
what
you
want
to
do
and
in
this
case
I've
got
something:
that's
failing
to
bind
to
a
persistent
volume
because
there
are
no
volumes
available.
So
if
you
do
a
get
events-
and
you
see
that-
maybe
that
tells
you
oh
I
forgot
to
deploy
whatever
configuration
I
need
to
make
that
binding
work,
Coop's
ETL
logs
no
get
on
logs
I
still
do
it
all
the
time,
but
there's
no
get
there.
A
It's
just
coops
ETL
logs
and
then
a
pod
name,
and
so
you'll
see
things
like
this.
Now
this
is
fairly
innocuous.
This
is
just
a
pod.
Starting
up.
This
is
a
we've.
Scoped
pod
you're
gonna
see
a
lot
of
we've
scoped
logs
as
we
go
through
this,
because
that's
what
I
used
for
my
samples
and
if
I
had
time
to
do
demos,
that's
what
my
demos
will
be
using
if
the
pod
has
multiple
containers
you
have
to
specify
which
container
you
want.
This
one
only
has
one
container
container
logs.
A
So
most
of
the
time,
your
container
logs
are
your
pod
logs,
especially
if
your
pod
only
has
one
container
in
it.
If
something
has
gone
really
wrong
on
your
cluster
and
you
can't
get
the
logs
from
the
pod
with
cube
CTL,
you
may
have
to
SSH
into
your
container
host
and
get
the
logs.
That
way
and
how
you
do
that
it's
gonna
vary
by
your
container
run
time.
A
You're
gonna
do
one
way
with
docker
another
way
with
Rockit
another
way,
if
you're,
using
something
else,
I
honestly
haven't
had
the
time
to
look
in
depth
in
the
house.
Eri
does
it
but
I
know
there's
like
CRI
control
that
you
can
use
to
get
logs
and
things
like
that.
So
just
keep
that
in
mind.
That's
you
may
have
to
go
back
to
the
good
old
fashioned
SSH.
A
You
can
also
run
what
I
think
of
as
a
debugging
container
and
I
think
somebody
mentioned
that
in
one
of
the
keynotes
today,
when
they
were
talking
about
things,
you
can
actually
SSH
to
a
host
figure
out
where
the
process
that
señor
troublesome
container
is-
and
you
can
attach
another
process
to
that
containers
namespace.
If
you're
running
docker,
you
can
actually
attach
a
whole
container
to
that
containers,
network,
namespace
and
pit
namespace
and
all
that,
and
so
what
that
allows
you
to
do
is
you
can
run.
A
Let's
say
you
you're
on
a
rail
host,
but
you
feel
more
comfortable
debugging
things
in
a
debian
environment.
You
can
SSH
to
the
host
run
a
debian
container,
attach
it
to
the
namespace.
You
need
to
attach
it
to
and
then
use
the
tools
you're
familiar
with,
or
in
some
cases
there
may
be
a
specific
debug
container
that
you
have
for
your
particular
application.
A
It
has
diagnostic
tools
that
you
can
use
to
interrogate
what's
going
on
either
way,
and
it's
also
a
little
bit
of
a
Hail
Mary
right
if
your
application
doesn't
provide
good
logs
or
it's
logging,
but
it
doesn't
know
what
it's
having
trouble
with.
At
least
you
can
fire
up
a
debug
container
and
look
at
the
environment,
and
maybe
that
will
give
you
an
idea
what's
going
wrong.
Also,
if
your
host
is
having
issues,
do
you
really
want
to
be
installing
packages
on
that
host
right
now
container
is
a
good
way
to
just
get
something
running.
A
A
Networking
issues
I,
don't
really
have
an
example
of
this
one,
because
it's
so
vast
I
don't
want
to
give
you
an
idea
that,
like
oh
there's,
eight
bullet
points
and
if
I
go
through
them,
I
don't
have
a
networking
issue.
No,
but
the
advantage
is
networking.
Issues
are
the
same
as
they've
always
been.
So
if
you're
on
the
host
you're
debugging,
the
network
you're
still
debugging
the
network
in
the
same
old.
What
you've
always
done,
TCP
dump
or
you
know,
whatever
your
packet
capture
method
of
choice-
is:
do
you
see
the
interfaces
you
should
see?
A
A
Your
network
flows
is
something
that
you
may
occasionally
need
to
do
and
the
annoying
thing
about
it
is
if
it's,
your
network
flows
that
are
the
problem,
that's
tends
to
be
when
your
applications
just
don't
know
what's
going
on,
because
all
they
know
is
they
can't
communicate
with
each
other
pod
startup
issues
are
your
pods
getting
scheduled?
Do
you
see
them
in
the
list
when
you
do
get
pods?
If
they're
getting
scheduled?
Are
they
starting
up?
Do
they
just
stay
in
pending
forever?
We'll
look
at
some
examples
of
what
that
might
be
later.
A
A
In
this
case,
it's
the
we've
scoped
agent
is
trying
to
contact
the
we've
scoped
front-end
and
failing
because
I
put
the
wrong
host
name
in
there
deliberately
for
demonstration
purposes.
But
this
is
what
it
looks
like
note
that
this
is
coming
from
the
application.
Again
we're
going
to
come
back
to
that
theme
of
you
do
depend
a
lot
on
your
application
logs.
So
are
you
having
DNS
issues
beyond
that?
Maybe
it's
not
something
you
did
wrong.
Maybe
it's
that
your
cluster
DNS
isn't
starting
up
or
running
properly.
A
B
A
Get
endpoints
on
the
service
and
see
if
there
are
any
endpoints
access
control.
This
is
one.
That's
been
biting
a
lot
of
people
lately,
especially
in
the
helm
community,
because
when
helm
charts
started
to
do
our
back
things,
they
initially
were
advising
people
to
default
to
are
back
false,
meaning
don't
create
the
are
back
resources,
but
then
everything
kind
of
flipped
and
for
about
the
last
couple
of
months
all
the
cluster
installers
have
been
defaulting
to
turning
our
back
on.
A
So
now,
all
these
old
help,
charts
that
worked
fine
when
they
were
first
created
and
uploaded
to
github,
are
now
failing
because
they're
not
creating
our
back
resources
and
that's
what
they
need
to
have.
This
is
what
an
are
back.
Failure
looks
like
again.
This
comes
from
the
application.
You
don't
really
get
to
see
this
in
the
API
server
logs
or
anything
like
that.
I
mean
I.
A
Suppose
if
you
sat
down
and
actually
did
a
TCP
dump
on
that
IP
for
the
API
API
server,
you
would
be
able
to
see
these
requests
coming
in,
but
you
don't
really
get
a
good
easy
view
of
our
back
issues
and
kubernetes.
So
again,
the
application
has
to
be
smart
enough
to
tell
you
what's
going
on,
and
here
we
see
it's
failing
to
list
things,
because
it
doesn't
have
permissions
to
list
anything.
It
can't
even
read
anything
out
of
the
API
server.
A
The
other
thing
that
people
are
running
into
now
since
it's
gone,
stable
is
network
policy
preventing
traffic.
Just
like
people
used
to
bite
themselves
with
iptables
rules.
I
just
saw
somebody
do
it
the
other
day
in
a
production
environment,
network
policy.
Sometimes
if
your
policies
aren't
written
exactly
the
way
they
should
be,
you
can
cut
off
traffic
that
shouldn't
be
cut
off
the
other
one
I
see,
although
not
so
much
anymore,
because
most
cluster
installers
take
care
of
it.
A
For
you
to
a
large
extent,
is
TLS
issues
either
using
the
wrong
TLS
credentials
or
not
having
TLS
credentials
for
something
that
requires
them
or
certificates
have
expired
or
whatever
the
case
may
be.
How
do
you
fix
it?
Well,
it
used
to
be
you
had
to
go
into
the
hosts
and
restart
daemons
and
change
config
files
and
all
that
you
still
change
a
config
file.
But
kubernetes
does
a
lot
of
cleanup
for
you.
So
you
change
your
config
file.
You
fix
your
container
tag
or
your
network
policy
rule
or
whatever
you
do.
A
A
cube,
CTL
apply.
Kubernetes
will
do
a
large
part
of
the
cleanup
for
you,
so
don't
make
work
for
yourself.
Don't
do
something
like
go
in
and
delete
a
bunch
of
pods
in
a
failing
deployment
and
then
fix
the
deployment,
because
if
you
delete
those
failing
pods
they're
just
going
to
be
respond
as
new
failing
pods,
let
kubernetes
fix
it
for
you
preventive
measures.
This
is
where
I
get
to
rant
for
a
little
bit
application
design.
How
many
developers
we
got
in
here
yep
lots
of
it.
A
How
many
of
you
developers
are
deploying
applications
on
kubernetes?
Okay,
not
as
many
but
still
quite
a
few.
Do
your
applications
log
diagnostic
info,
and
do
they
do
so
correctly
and
I
call
out
here
just
because
I'm
always
amused
by
the
fact
that
it
is
still
in
the
Linux
source
code
lp0
on
fire?
Now,
if
you
get
that
error-
and
you
can
get
that
error,
it
doesn't
actually
mean
anymore
that
your
printer
is
on
fire.
It
means
that
Euler
combination
of
air
conditions
has
happened,
but
that
error
message
is
still
in
the
Linux
source
code.
A
It
still
comes
up
for
that
combination
of
air
conditions
that
it's
no
longer
the
message
itself
is
no
longer
relevant.
They
should
be
giving
people
a
much
more
informative
error
message
as
to
what's
going
wrong.
Don't
write,
lp0
on
fire
error
messages
and
the
reason
I
call
that
one
out
is
because
that's
one
that
actually
did
approximately
mean
your
printer
was
on
fire
back
when
it
was
written.
But
it's
been
superseded
by
events,
so
always
make
sure
that
you
go
back
and
review
the
failure,
circumstances
of
your
application
and
how
it
behaves.
A
Are
you
logging
enough?
So
maybe
you're
logging
correct
things,
but
you
aren't
logging
enough
of
them
or
in
enough
detail.
I
love
applications
that,
let
me
specify
the
level
of
verbosity
you
do
a
Kubb
control,
be
99,
you
get
everything,
you
probably
don't
want
everything,
but
it'll
give
it
to
you
anyway,
and
even
if
you're
sure
you're
writing
enough
detailed
logs
go
ahead
and
make
it
more
detail.
A
Do
you
have
diagnostic
tools
and
do
you
document
them?
I
talked
earlier
about
using
a
debug
container.
What
good
is
a
debug
container?
If
you
don't
have
the
tools
to
use
in
it
to
debug
anything?
So
if
your
application
needs
specific
debug
stuff
I
mean
if
your
application
can
be
debugged
by
a
regular
old
TCP
dump
session.
A
Fine
document
that,
because
even
if
you
have
the
tools
and
they're
not
documented,
they
don't
do
anybody
any
good
and
then
the
last
thing
and
I
know
I'm
kind
of
preaching
to
the
choir
on
this
and
I
will
be
again
in
the
next
slide.
Is
how
safe
is
your
data
and
state
in
case
of
your
app
failure,
we're
working
in
cloud
environments?
Now
this
is
the
thing
we
have
to
worry
about.
You
can't
just
go,
oh
well.
If
the
process
gets
killed,
then
everything
is
resident
on
the
disk.
A
What
disk
you
might
not
be
even
be
even
saw
on
the
same
node
when
you
come
back
up?
Can
you
roll
back
from
migrations?
Have
you
tested
rolling
back
from
migrations?
If
you
haven't
tested
your
recovery
plan,
you
don't
have
one
pre-deployment
I
know,
there's
been
lots
of
talk,
and
especially
in
the
keynotes
about
automation
tools.
My
favorite
one
is
helm.
I
participate
in
helmand
community.
A
My
demos
are
actually
generated
by
helm,
templating,
a
helmet
art
that
I've
been
working
on
and
have
a
you're
in
for
so
you'll,
see
like
the
helm,
labels
they're
not
necessarily
relevant
to
the
demo
itself,
but
they're
there.
It
was
just
an
easy,
quick
way
to
get
a
demo
going
and
one
of
the
things
that
the
help
community
preaches
and
for
good
reason
is
factored
out
redundancy,
because
every
time
you
repeat
yourself,
you
give
yourself
a
chance
to
repeat
yourself
wrong
and
your
environment
should
not
only
make
both
of
those
possible.
A
It
should
be
the
easiest
way
to
operate
in
your
environment.
You
should
be
telling
your
developers,
you
know,
or
your
code
developers,
don't
do
all
that
work
to
deploy
manifest
manually.
We
have
a
helmet
art
in
a
home.
Chart
repo,
just
helm,
install
it's
easy,
and
if
it's
easy-
and
if
it
works
correctly,
then
that's
what
people
will
do
so
helm
draft.
The
other
brigade
is
a
scripting
language
that
helps
automate
a
lot
of
things.
A
Consider
the
cluster
autoscaler.
It
can
help
you
avoid
outer
resources,
problems
and
the
last
thing
is
test.
Environments
are
not
optional,
I,
don't
care
how
small
your
budget
is.
You
need
a
test
environment.
If
you
don't
have
the
budget
for
a
test
environment
you
don't
have
the
budget
to
run
and
I
know.
Somebody
has
a
reason
why
they
can't
afford
a
test
environment
you're
wrong.
You
need
a
test
environment,
post
deployment,
application,
validation
tests.
This
is
another
thing
the
helm
community
has
started
doing.
A
A
I,
don't
have
to
tell
anybody
in
here
about
monitoring
your
applications
right,
but
I
think
some
people
and
I
don't
think
anybody
in
this
room,
but
some
people's
management
of
people
in
this
room
may
not
have
gotten
the
message
that
you
don't
need
to
alert
on
every
little
thing,
and
especially
in
a
kubernetes
environment.
A
node
going
down
at
3:00
a.m.
in
a
properly
designed
environment
is
not
a
reason
to
page
anybody
and
wake
them
up
out
of
bed.
A
It
is
a
reason
to
log
something
and
maybe
throw
an
email
out
there,
but
unless
you
actually
lose
the
end-user
access
to
the
application,
don't
wake
people
up.
That's
I
saw
that
I
saw
a
Prometheus
presentation
when
I
worked
at
core
OS,
and
that
was
the
guy's
main
idea
is
what
matters
is
what
the
end-user
can
or
can't
do,
and
at
the
end
user
can
do
everything
they
should
be
able
to
do.
You
shouldn't
be
waking.
A
People
up
adopt
the
chaos
monkey
every
now
and
then
go
into
your
data
center
and
just
yank
a
plug
out
whether
you
do
that
in
a
literal
sense
or
a
virtual
sense.
Somebody
has
a
chaos
monkey
agent
that
you
can
deploy
under
kubernetes
cluster
it'll,
randomly
kill,
pods
and
things
like
that,
and
the
reason
I
say
that
is
you
show
me
a
server
with
high
uptime
and
I'll.
Show
you
a
server
that
has
unstirred
changes
just
waiting
to
bite
somebody
as
soon
as
it
reboots.
A
Where
you
can
get
more
help,
kubernetes
docks
the
API
Docs
I
go
back
to
all
the
time.
I
was
all
over
this
API
Docs,
while
I
was
researching
for
this
talk,
there's
also
other
good
info
on
the
main
tasks.
Page
there's
a
sidebar
there
and
it
doesn't
seem
to
work
to
link
directly
to
specific
things
under
the
sidebar.
But
if
you
go
to
the
main
page
and
you
go
under
monitor
login
debug,
there's,
actually
troubleshooting
clusters
and
troubleshooting
applications
and
he'll
recap
a
lot
of
this
info
kubernetes
slack
channels,
kubernetes
Navas,
kubernetes
users,
depending
on.
B
A
A
Look
within
yourself
if
you've
been
working
in
IT
for
20
30
40
years
and
you're
going
well
now,
I'm
in
a
cloud
environment.
All
that
knowledge
I
have
is
just
it's
not
relevant
anymore.
No,
it
still
is.
None
of
this
is
new.
Hen
Goldberg
said
that
yesterday,
right
here
on
stage,
none
of
what
we're
doing
is
new
we're
just
putting
it
in
a
different
environment.
This
is
my
favorite
networking
document,
RFC
1925.
It
was
submitted
as
a
joke,
but
it's
one
of
those
only
serious
jokes.
It
was
an
April
1st
RFC
from
1996.
A
If
you
go
read
it,
it
will
give
you
advice
that
you
will
be
able
to
apply
to
your
kubernetes
clouds
today,
I
guarantee
it.
The
other
thing
that
I
personally
find
useful
is
take
the
time
to
actually
sit
down
and
fully
describe
the
problem
you're
trying
to
solve,
because
in
the
act
of
this
driving
it
you
often
go.
A
Oh
I
know
how
to
fix
that
they
call
it
rubber
duck
debugging
or
teddybear
debugging
or
whatever
some
people
go
and
describe
it
to
a
co-worker
who
doesn't
know
anything
about
it,
so
it
forces
them
to
describe
it
at
a
high
level
of
detail
either
way.
Just
take
the
time.
Take
a
breath,
sit
down
and
think
about
it.
A
B
A
Scratched
the
surface
like
I,
said
I'm,
just
gonna
run
right
over
this
and
I.
Don't
expect
anybody
to
walk
out
of
here
and
retain
everything.
I
just
said:
that's
why
I
posted
the
slides
online
and
why
there's
a
link
to
the
slides,
aren't
there
tools
for
these
kind
of
things
am
I
actually
gonna
have
to
go
in
there
and
do
coop
control
logs
and
cube
control,
get
events
there
are
tools,
but
what
happens
if
deploying
the
tools
goes
wrong?
What
happens
if
the
tools
get
deployed
and
they
fail?
What?
A
A
Tell
us
all
about
this
awesome
tool
that
you
found
and
how
it
helps
you
recover
from
a
you
know:
production
sub-zero
outage
and
how
much
you
love
that
tool,
because
that's
what
everybody's
been
talking
about
community
the
last
three
days,
that's
kind
of
what
the
community
is
all
about,
go
out
and
find
the
things
that
work
and
come
back
and
tell
everybody
else
about
it,
don't
fall
for
impostor
syndrome.
This
is
just
more
believe
in
yourself.
A
You
know
more
than
you
think
you
know,
and
once
you
start
digging
down
into
it,
you're
gonna
realize
oh
I
know
what
we
need
to
do
here.
If
you
feel
like
you're
drinking
from
the
fire
hose,
it
just
means
you
actually
know.
What's
going
on
and
Ilya
I'm
gonna
butcher,
his
last
name
check,
check
Regan
on
stage
yesterday
said
trust
yourself,
that's
what
this
is
all
about.
A
A
Yes,
there
is
a
way
to
do
that.
I,
don't
remember
what
that
is
off
the
top
of
my
head,
but
I
know
that
you
can
get
the
logs
from
a
previous
execution
of
if
you're
doing
it
through
coop
control.
It's
the
previous
execution
of
that
pod,
but
if
it's
a
single
container
pod
it'll
be
the
logs
for
that
container.
A
Oh
I
should
have
put
that
in
there
and
I
and
and
I,
don't
remember
it
off
the
top
of
my
head.
But
if
you
go
out
there
and
you
look
up
chaos,
monkey
kubernetes
agent
in
Google,
I
I,
think
you
will
find
it.
I
will
make
a
point,
though,
when
I
post,
the
updated
version
of
these
slides
I,
will
link
that
to
the
one
that
I
was
thinking
of
so.
A
If
it's
the
cluster
itself,
then,
depending
on
how
that
cluster
was
built,
you
may
be
going
in,
you
will
almost
always
be
going
into
a
node
like
connecting
directly
to
a
node
and
then
either
doing
a
journal,
CTL
or
looking
at
syslog.
If
it's
a
syslog
system
or
looking
at
the
docker
logs
directly
of
the
containers
that
run
the
control
plane,
components
if
that's
how
its
set
up
there
are
different
ways,
and
so
there
isn't
really
one
answer
to
that.
But
yeah.
A
Generally,
if
it's
something
to
do
with
the
cluster,
you
will
see
really
severe
problems
like
you.
Can't
coop
control
anything
right
and
it
will
be
stuff.
That's
not
even
like
bad
credentials
that
you
go.
Oh
I
have
the
wrong
context,
file
or
something
it'll
just
be
like
timed
out
or
connection
refused
or
whatever
the
case
may
be.
You
will
see
if
you
do
a
coup.
Control
get
nodes,
you
may
see
nodes
not
ready,
that's
often
because
their
networking
isn't
correct.
A
The
most
common
modes
that
I
see
there
and
the
most
common
install
methods
are
running
them
directly
out
of
a
container
on
the
host
or
actually
running
them
as
self
hosted
pods
and
the
system
that
I
use
for
spinning
up
little
demo
clusters
called
boo,
cube
and
also
cube.
Atm
tend
to
run
them
as
actual
pods
with
manifests.
A
A
A
These
files,
by
the
way,
are
also
linked
off
of
the
slides,
which
I
will
put
up
the
thing
for
the
there's,
a
QR
code
and
a
slide
link
that
I'll
put
up
at
the
very
end
of
this.
So
the
demos
I
can
do
I
can
do
an
app
communication
error,
a
crash
loopback
off
here.
A
discovery.
Failure
failed
mount
error,
that's
a
fun
one
because
it
just
never
comes
up
image.
Polar
pods
stuck
in
pending
or
an
hour
back
issue
show
hands
for
app
communication.
Error
show
hands
for
crash
loopback
off.
A
Oh
I
think
that
one
might
win
discovery,
failure
failed
mount
image,
pull
air
and
penning
pod.
Oh
and
our
back
issue.
Okay,
I
will
do
crash
Lea
back
off
and
then,
if
I
have
time,
I
will
do.
Let
me
see
pending
again
and
let
me
see
our
back
again.
Okay,
I
think
that's
a
rough
high,
but
I'm
gonna
do
our
back
if
I
have
time.
A
B
A
Not
get
logs
I
have
to
remind
myself
of
that
now.
This
is
it.
This
looks
cryptic,
but
what
it
actually
is,
if
you
look
at
the
manifest
this
pod
just
runs
a
busy
box
container
that
runs
a
sleep
command,
and
so,
if
I
look
at
that
sleep
container
broken
deployment
down
here,
I
gave
it
an
argument.
Sleep
R,
well,
R
is
not
a
number
of
seconds
to
sleep,
so
it
quite
reasonably
terminates
with
an
error.
Kubernetes
says
well
that
pod
terminated
with
an
error.
A
A
And
now
it's
configured
my
deployment
and
if
I
do
what
I
did
before
we
can
see
the
old
one
is
terminating
already.
The
new
one
is
coming
up
and
has
already
gone
into
a
running
state,
so
we
fixed
it
kubernetes
cleaned
up
after
us.
We
didn't
have
to
do
anything
but
fix
it
and
reapply.
That's
the
great
thing
about
declarative
systems
management
all
right,
one,
more
I'm,
gonna
do
the
are
back
one.
Let
me
clean
up
after
this
ABC
help
delete.
A
A
Everything's,
that's
the
old
one,
that's
still
terminating
from
before,
but
everything
looks
like
it's
running,
so
everything
should
be
okay
right,
but
if
I
actually
go
into
my
weave
scope,
front
end,
which
I
don't
have
time
to
set
up
the
port
forward
for
right
now,
I
will
find
out.
There
is
no
data
being
reported
to
it,
which
means
the
agents
can't
talk
to
it
or
the
agents.
Don't
have
any
data
to
give
it.
So
what's
the
problem?
A
Well,
we
know
it's
a
problem
with
the
agent
just
because
we
know
how
we've
scope
works,
so
I
do
a
cube
CTO.
Yet
log,
no
I
do
a
coop
CTL
logs,
not
get
logs
uh-uh
see
here
we
go.
This
is
all
the
stuff
that
I
showed
you
before
failed
to
list
failed
to
list
cannot
list
services
at
the
cluster
scope.
What's
the
fix?
The
fix
in
this
case
is
I'm,
not
updating
any
of
the
resources
that
I
created
before
I'm,
adding
resources,
a
cluster
role
in
a
cluster
role,
binding
and
then.
A
Away
and
now,
if
I
do
a
cube,
see
tale
logs
on
the
agent
well,
it
goes
into
a
back
off.
It's
backed
off
to
a
minute
and
20
seconds
now,
so
what
I
would
probably
actually
do
in
reality
is
just
go
ahead
and
delete
that
demon
set
and
reapply
it.
In
fact,
I
have
probably
just
about
enough
time
to
do
that.
A
A
Publish
loop
starting
its
collecting
info
and
it's
already
started
to
publish
it
to
the
front
end
which
again,
if
I
pork
forward
it
to
the
front
end
we'd,
actually
be
able
to
see
this
nice
little
diagram
of
the
cluster
I.
Don't
think
I
have
time
for
that.
Just
because
I'm
getting
a
signal
time
is
up
so
with
that.
I
will
very
quickly
give
my
grateful
appreciation
to
my
ot,
my
management,
Sam
and
Raja,
and
the
other
people
at
O
Teemo
for
supporting
me
getting
here.