►
Description
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
How to Debug the Pod Which is Hard to Debug - Eohyung Lee, Kakao Enterprise
When using Kubernetes, users face a variety of problems. The most diverse of these problems occurs in pods. So users need to know how to debug pods.
Of course, the start of debugging a pod is taking a look at it's status and logs and events. But, in most case, this is not enough. So the users want to get the shell from a pod and test it by reproducing the same situation as the problem. But there are pods that are impossible to get the shell such as scratch image.
So this session will introduce various techniques for debugging pods with minimal or no modification workloads.
In particular, the following cases will be explained.
* debugging docker, containerd pod
* debugging pod using host informations, tools
* debugging pod in crashloopback status
* debugging pod based scratch image
* debugging readonly disk pods
A
A
A
A
Since
before.
I'm
still
making
cloud
services
with
Tis.
So,
before
I,
tell
you
about
the
red
bean
I'm
going
to
talk
about
today,
I
need
to
talk
about
it.
If.
We
do
that,,
then
we'll
look
at
what
problems
a
lot
occur
while
using
Netis.
In
the
case
of
things,,
a
lot
of
problems.
Occur.
In,
my
case,
I,
am
now
working
as
a
cloud
engineer,
and
now
the
local
government
is
managing
a
little
bit
of
the
local
government
in
number
9,,
which
we
couldn't
do,
and
managing
the
creative
applications,
and
now
on
top
of
it.
A
A
A
A
A
What,
padding
or
weighting
or
language,
shaker
garter
scheduling
itself
seems
to
be
a
problem
now
or
something
in
the
contact
manager.
N0,
red
bean
status
is
not
updated
anymore,
and
there
are
states
like
that
and
now
how
to
deal
with
many
cases,
The
larger
this
is,,
the
more
it
is
taken
away
or
falls
into
an
error
state,,
so
that
the
cache
body
continues
to
re-clock,
and
it
dies
and
does
not
love,
anymore,
and
so
on.
Since.
A
A
A
A
A
A
However,,
like
the
ones
I
just
explained
in
the
section,
most
of
the
pods,
have
no
ku
body,.
So
there
is
no
red
bean,
or
just
ku
city
mod,
scribe
park
or
It's
like
looking
at
the
neck
or
something
like
this..
Most
of
us
are
familiar
with
it.
4
As
you
know,
now,
anyway,.
We
only
have
logs
or
status
checks,,
so
it's
very
difficult
to
solve
the
problems.
Now.
How.
Should
we
solve
the
problem.
Now?
Spring
is
coming
now.
In,
fact,,
Lindsay,,
February,
and
Abraham.
So,
in
reality,.
A
The
problems
that
arise
during
service
are
very
common,
and
there
are
many
cases
where
the
three
problems
mentioned
above
are
mixed
in
a
complex,
way.
I
think
there
are
many
cases
where
we
encounter
a
problem
without
knowing
it
is
a
problem.
Now.,
In
fact,.
Now
that
we
know
it,,
we
know
that
it
was
one
of
these
three
problems,
or
we
now
know
that
it
was
two
problems
out
of
these.
You'll
find
out
about
those
things.
After
solving
the
problem.
Before
that,
in
fact,,
when
you
get
into
the
car,,
you
can't
know
this.
A
So,
to
solve
this
problem,.
We
start
by
checking
the
logs,,
events,
and
status
that
we
mentioned
earlier,
and,
of
course,
the
application
itself.
The
problems
that
occur
in
will
now
be
confirmed
with
more
detailed
logs,
such
as
logs,
etc.
And.
Even
in
these
cases,,
there
are
a
lot
of
problems
that
could
not
be
solved,
and
we
redeployed
them
directly
and
tested
them.
So,
problems
that
occur
in
production
are
usually
reproduced
in
a
test
environment
such
as
a
stage,,
and
we
try
to
test
the
problem
in
it.
A
There
are
many
cases
where
it
is
difficult
to
test
the
problem.
So
I
want
to
do
these
things,,
but
I
really
can't
afford
another
case.
I
want
to
finance
this
somehow
on
stage,,
but
the
problems
occur
very
intermittently
After
that,.
All
of
these
problems
occur
again
and
again,
irregularly,
and
as
a
result,,
we
test
by
directly
accessing
the
cluster
that
we
actually
service.
Now,.
There
are
extreme
cases.
In,
the
near
future,.
We
are
actually
trying
to
make
these
solutions
into
functions,,
such
as
a
lot
of
tools
in
cover,
netis,,
etc.
A
A
A
A
A
Unfortunately,,
it
is
currently
still
in
development,
and
the
function
has
not
been
released,,
but
in
the
future,
through
commands
such
as
debug
to
this
Ku
sheet,,
we
will
Now,,
if
you
add
a
container
inside
an
additional
red
bean,,
put
all
the
debugging
information
in
it,
and
enter
the
temporary
container
to
be
debugged,.
You
can
now
debug,
through
the
container
of
the
food.
After,
creating
a
container
and
entering
the
temporary
container
with
a
command
such
as
acq,
frame,,
debugging
inside
it,,
and
then
automatically
deleting
these
things.
A
A
A
Now,,
if
we
try
to
define
a
part
in
a
difficult
state
for
debugging
1,,
first
of
all,,
when
we
first
think
about
the
difficult
part
for
a
moment,.
When
we
first
think
about
these
two
kinds
of
problems,
I
think
it
was
difficult
to
debug,
now.,
It's
ridiculous,,
as
I
said
earlier,.
The
container
itself
now
continues
to
restart,
and
the
bigger
we
are,,
the
more
Ubex
status
or
the
status
continues
in
error,
status.
So,.
Now?
What
command
do
we
use?
Even?
A
A
A
A
A
A
Cases,
I'll
die
again
because
I
haven't
created
a
file
like
healthy
in
Team
Fit
by
checking
the
live
now
like
dew.
That's.
Why?
The
second
comment
itself,,
which
was
moved
like
the
first
one,
now
fails,?
The
comment
itself
is
actually
Asa:
Rise
Second,.
If
the
live
ns
itself
is
continuously
restarting
from
failure,.
If
you
want
to
do
something
with
the
live
ns
pro
group
for
a
while,,
you
can
alleviate
this
condition
and
buy
time
to
debug
and
debug.
In
the
meantime,
Of.
A
Course,,
but
if
you
do
this,,
you
can
modify
the
container
itself
and,
in
fact,
the
2
of
this
lime,,
and
we
can't
guarantee
2
of
these
things
in
the
middle
of
that,.
So
in
fact,
as
much
as
possible,
duplicate
this
red
bean,
so
that
it
can
be
reproduced,
I've
tried
it
and
it
would
be
nice
to
test
it
in
a
different
environment,.
But
when
those
things
are
impossible,
I
use
a
way
to
keep
this
container
itself
alive
through
these
methods
and
allow
us
to
debug
with
other
commands
in
it.
In.
A
This
way,,
if
you
solve
the
problem
and
forcibly
keep
it
alive
in
this
way,,
the
problems
that
are
difficult
to
solve
are
now
included,,
such
as
the
tool
de
variable
shell
to
debug,
using
sketch
images.
It
's,
a
problem
of
debugging,
an
unopened
container.
It's,
a
very
common
problem,
of
course,,
but.
A
We
have
to
use
sketch
images
because
of
various
issues
such
as
security
issues
and
security,
issues.
However,.
If
this
is
actually
the
case,,
it
is
difficult
to
provide
d,.
So
if
there
is
a
problem
of
the
basis,,
these
things
will
now
be
put
back
in
the
debugging
tool,
and
a
new
image
will
be
tested
again,
and
after
finding
the
problem,
change
it
back
to
the
original
image.
A
These.
Things
now
complain
that
you
restart.
So,,
the
methods
I'm
going
to
tell
you
about
here,
are
actually
a
bit
different
from
the
methods
above,.
But
now
it's
like
direct
access
to
the
container
host,,
so
we
can
use
the
information
in
it
and
the
functions
in
it
as
much
as
possible.
Now
I'm,
going
to
tell
you
about
how
to
debug
a
container
by
using
these
methods.
If.
You
use
these
methods,
now,
as
an
advantage,.
We
can
look
at
all
the
information
without
restarting
the
content
itself
anyway,
or
we
can
debug
it
for
the
longest
time.
A
A
A
A
Can
look
into
all
the
information
such
as
Daedeok
usage
or
what
kind
of
session
is
established.
In
addition
to
this,.
There
is
a
lot
of
various
information
in
forming
this
container,.
So
if
you
take
a
closer
look
now
through
links
like
this
below,,
we
know
what
Even,
if
you
do
not
enter
the
container,,
you
can
look
into
these
container
states
with
only
such
poster
plus
information.
A
A
A
A
Have
visited
the
dead
space
for
the
container,
we
are
targeting,
and
have
already
created
a
space
on
the
network.
Daemon.
In,
the
state,,
the
parent
terminal
on
the
host,
is
like
us,
here,
like
ss,,
and
now
we
have
debugged
through
such
a
command
to
see
the
socket
state
now,
as
an
example.
If.
You
use
these
things,.
We
can
now
enter
the
curtain
directly
or
find
out
a
lot
of
information
without
having
to
do
that.
But
in
this
case,.
Now
we
have
a
little
bit
of
debugging.
Since,
the.
A
A
A
A
A
A
If
you
get
10
won
by
actually
copying
our
bag,
lady
child,
to
the
root
directory
under
such
a
bouncing
tool,
in
this
case,,
we
plant
binaries
like
busy
box,
and
you
can
use
it.
Now.
I
gave
you
a
copy
of
the
root
directory
that
is
useful
for
persecuting
the
binary,
and
through
it,.
It
is
now
a
command
for
the
curve
that
is
used
by
retrieving
something
in
the
container.
A
In
the
case
of
a
midi
file
system,,
it
is
typical
for
my
core
dms.
Core
dns
puts
something
like
a
lidon
root
file
system
in
the
context
of
the
Caeon
Inje
suit,
to
create
any
arm
in
it
or
to
block
such
things
now,.
So
the
actual
moto
kernel,
cp
Even.
If
you
use
a
command,,
you
can't
put
the
binary
in
the
cafe
anymore,
and.
A
A
A
Think
that
we
can
find
ways
to
solve
problems
even
when
we
have
to
solve
problems
in
something
really
extreme,.
So
it
is
helpful
to
prepare
in
advance.
I
had
a
session
because
I
thought
it
wouldn't
work.
4
Actually,.
The
time
is
ending
at
5:05.
I
'll
ask
if
there
is
anyone
who
would
like
to
confirm
it
for
about
2
minutes.
A
A
Result,
in
the
case
of
9
blacks,,
the
accu
flat
now
has
the
name
that
generates
the
normal
certificate
when
communicating
with
the
processing
of
the
second
half
of
the
chise
api,
and
now
determines
this
node
for
cover
netis.
So,
hey,.
We
made
a
mistake
by
duplicating
the
name
of
the
node,.
So
when
we
created
it,,
we
did
not
have
a
certificate
before,,
so
it
took
9
minutes.
I.
A
Couldn't
distinguish
it,
and
I
thought
it
was
the
same
node
because
I
upgraded
the
same
Kuble
processor
and
requested
to
create
an
ip
twice,,
but
we
used
something
when
we
solved
problems
like
this
with
two
ip's.
Most
of
the
problems
can't
be
prevented,
and
we
are
representative
of
parties
such
as
core
dns,,
which
has
a
very
high
security
level.
Anyway.
We
use
all
of
these
methods
when
debugging
such
things.