►
From YouTube: Secure and Debuggable: Debugging Slim, Scratch and Distroless Kubernet... Saiyam Pathak & Kyle Quest
Description
Don't miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from 18 - 21 April, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Secure and Debuggable: Debugging Slim, Scratch and Distroless Kubernetes Containers - Saiyam Pathak, Civo Limited & Kyle Quest, Slim AI
You reduced the attack surface of your containers with docker-slim or manually optimised scratch or distroless images, and you are super excited about it, but now you have a problem. How do you debug your app? You can't simply "kubectl exec" to your containers because there's no shell. You can't even "kubectl cp" to copy and run your debug tools! Kubernetes 1.25 is out, and the Ephemeral Containers capability is finally GA, which means it'll be available everywhere for you to use to debug your applications. In this talk, You'll learn how to use Ephemeral Containers as debugging tool sidecars to make your application container images as small as possible. You'll learn how "kubectl debug" uses Ephemeral Containers and its gotchas (what you get and don't get with "kubectl debug" by default). You'll also learn how it's possible to use Ephemeral Containers without "kubectl debug". Speakers will also discuss the existing debugging tool container images and how you can use them.
A
Cool
so
hello,
everyone
and
welcome
to
cube
day
Japan
and
we'll
start
the
session.
Now
secure
and
debuggable
debugging
screen
slim
scratch.
Distillers
kubernetes
containers
a
lot
of
terminologies
over
here.
We'll
try
to
you
know,
simplify
all
these
and
see
how
you
can
debug
it
so
expected
background.
As
you
know,
Kyle
mentioned
some
of
the
container
fundamentals
is
needed.
You
should
know
what
kubernetes
is.
You
know
the
basic
knowledge,
namespaces
fundamentals,
architecture,
nodes
and
all
these
things
and
then
we'll
cover
the
rest
of
the
stuff.
A
B
B
It
was
one
of
those
things
on
the
wish
list
and
you
couldn't
do
it
well,
you
could
do
it
the
hard
way,
but
in
terms
of
mass
adoption
it
wasn't
possible
because
to
use
smaller
containers,
you
still
need
that
debuggability,
and
this
is
what
we're
going
to
talk
about
and
apparently
you
know,
there's
a
lot
of
interest
in
it.
So
let's
keep
going.
A
So,
let's
get
started
with
the
general
debugging
techniques
for
for
kubernetes,
which
everybody
like
we
have
been
doing
it,
just
a
quick
preview
of
that.
So
whenever
there
is
any
failures,
we
go
through
the
events,
the
logs,
the
describe
host
and
node
level
debugging.
So
we
SSH
into
that
use.
Privileged
containers
use
some
embedded,
debugging
tooling,
also
the
cube
CTL
exec,
which
is
pretty
common.
A
If
you
want
to
get
inside
the
container,
you
use
Cube
serial
exec
coincide
and
do
a
bunch
of
stuff
to
see
what
went
wrong
and
then
some
sidecar
stuff,
which
is
there
so
yeah.
This
is
some
of
the
stuff
like.
If
you
do
Cube
CTL
describe,
you
can
see
the
you
know
the
events
like
if
a
pod
is
spending
insufficient
resources.
A
So
all
these
you'll
get
and
if
you
have
a
crash
to
back
off,
you
can
do
Cube,
City
logs,
and
sometimes
you
have
to
do.
You
can
also
attach
that
hyphen
iPhone
previous,
so
that
you
can
get
the
logs
from
the
previous
container
to
see
what
failed
it's
useful
and
then
the
exact
one,
the
cube
CTL,
exec,
interactive
pod
and
the
bash
or
sh.
Whatever
is
there
inside
that
particular
container
that
you
can
attach
to
and
run
the
commands
with
the
debug
utilities?
A
A
A
A
So
environment
variable
is
undefined,
so
so
these
are
standard
and
very
simple
to
use
the
the
first
level
of
the
basics
that
you
have
to
go
through
whenever
you
are
debugging
a
container
so
either
describe
the
logs,
the
exec,
the
events
and
all
these
things.
So
this
gives
you
a
bit
of
gist
of
how
you
know
you
do
debug
it
now
over
the
time.
What
has
happened
is
we
have
moved
to
like
we
have
been
creating
the
containers,
but
now
we
have
moved.
A
We
want
to
move
to
Slimmer
images,
to
you
know
the
images
which
contains
less
so
that
you
know
we
have
less
secure
vulnerability,
less
vulnerabilities
in
that
and
the
image
size
is
less
so
with
that
there
are
some
Concepts
scratch
distrollers
and
slim.
What
are
these
so
scratch?
Image
is
basically
an
empty
image.
It
has
nothing
in
it.
It
is
used
to
run
the
binaries
which
don't
have
anything
linked
to
that
I
mean
yeah.
There
are
some
gotchas
that
will
be
covering
later
on.
Mostly
you.
A
So
not
everything
that
you
can
run
with
that,
so
it
is
a
compiled
binary,
which
is
there
that
have
no
dependencies
and
if
it
has
the
dependencies
that,
probably
you
have
to
know
all
the
linked
dependencies-
and
you
have
to
you-
know
manually,
get
that
because
it
doesn't
even
have
the
like.
You
can
see
the
ca
search,
Etc
password,
nothing
is
there,
so
you
have
to
make
all
the
things
up
in
the
multi-stage
builds
and
then
you
can
run
it
yeah.
B
A
That's
where
it
gets
complicated,
that's
where
it
gets
complicated
coming
to
digitalis,
so
distronus
is
not
completely
empty.
It
has
some
of
the
base
directories
which
helps
you
in
creating
the
containers
and
running
them.
It
has
like
Etc,
password,
CA
search.
It
has
the
app
plus
the
runtime
dependencies
both
of
these
have
no
shell
and
no
package
manager.
So
that's
where
it
becomes
tricky
right.
These
are
good
to
have,
but
these
are
only
good
to
have.
A
If
you
have
everything
sorted
like
if
you
know
your
container
won't
fail,
but
if
your
container
fails,
then,
probably
since
it
has
no
shell
package
manager,
it
becomes
difficult
to
debug
these
set
of
containers
and
that's
what
we'll
be
discussing
and
showing
you
the
demos
for
so
the
gorgeous
extra
dependencies
are
super
complicated.
Whenever
you
have
extra
dependencies
in
scratch
image,
you
have
to
know
them
beforehand,
if
you
even
have
to
run
it,
no
debugging,
tooling,
no
shell,
and
not
always
a
static
binary.
A
A
B
To
how
you
can
get
to
a
minimal
container
image
for
your
application,
some
of
them
require
a
lot
more
work
than
others.
Distro
list
is
probably
one
of
the
most
known
ways
to
get
there.
There
are
several
flavors
of
distralis,
starting
with
the
most
basic
static
that
gives
you
mostly
a
directory
layout
and
a
couple
of
things
on
top
of
that,
including
the
The
Zone
info.
B
That's
pretty
much
the
biggest
chunk
of
that
image,
and
then
we
have
other
images
like
the
next
level
is
the
base
image
which
includes
the
static
version
plus
a
few
extra
libraries
basic
OS
libraries,
and
then
you
have
application
specific
distro-less
versions,
they're
much
bigger.
They
include
the
base
plus
the
additional
system,
libraries
and
the
application
runtime
now
and
I'll
talk
about
the
gotchas
later
on.
B
So,
but
this
Docker
file,
one
of
the
interesting
things
about
it,
is,
is
that
what
you
end
up
with
you
might
have
a
multi-stage
build
and
then,
in
the
final
stage,
you
copy
stuff
to
your
Deployable
image.
Now,
in
most
cases,
you
end
up
with
those
kind
of
copies
where
you
copy
a
whole
directory
with
things,
because
otherwise
you
you're
not
really
sure
what
you're
doing
so.
So
that's
that's.
B
So
this
is
an
example
for
those
compiled
languages,
because
sometimes
you
know
with
go
the
first
thing
that
comes
to
mind
the
static
binaries,
a
single
binary.
You
can
easily
create
a
scratch
based
application
image
and
that's
pretty
much
it
in
reality
for
real
applications,
real
world
applications,
you
actually
have
extra
dependency
certificates,
user
information
and
in
some
cases
it's
not
even
compiled
statically.
So
this,
let's
see
okay
yeah,
let's,
let's
keep
going
so
in
this
case
I.
B
This
is
a
sample
app
for
Docker
slim,
one
of
the
third
party
apps,
and
it's
not
compiled
statically.
So
we
end
up
pooling
stuff
like
lip.
B
thread,
shared
objects,
Etc,
so
there's
a
lot
of
stuff
that
gets
pulled
in
and
that's
when
it
gets
tricky
when
you're
trying
to
do
it
the
hard
way
build
those
minimal
container
images
by
hand.
So
let's
keep
going
so
with
the
with
the
slim
image.
The
nice
thing
about
it
is
that
you
have
a
regular
Docker
file.
This
is
probably
the
most
basic
node
application.
B
Docker
file-
it's
not
multi-stage.
It's
simple
derives
from
your
node
base
image.
You
copy
the
the
package
manifest
you
install
the
dependencies
you
copy
the
app
and
then
you
just
run
the
app
that's
all
now.
That's
all
you
would
have
for
a
regular
app
and
then
with
the
slim
images
it
takes
that
and
then
it
produces
a
much
smaller
image
kind
of
like
an
out
of
scratch.
It
does
it
for
you.
So
let's
keep
going.
A
So
now
we
have
discussed
what
slim
image
is.
What
distortless
is
what
scratch
image
is,
and
we
have
discussed
some
of
the
toolings,
but
I
mentioned
one
particular
point
that
it
it's
hard
to
debug
the
digitalis
and
slim
and
scratch
images
due
to
the
lack
of
shell
and
debugging
tools
within
that.
So
we
cannot
use
the
standard
stuff
which
is
Cube
CT,
Alexa
CP,
embedded
debugging
tools,
because
it's
not
there.
So
then
it
becomes
difficult
and
Cube
serial
CP.
Just
it
requires
a
tar
to
be
there
in
the
container
else.
A
It
will
fail
and
it
copies
files
from
the
directory.
Exactly
is
a
shell
to
be
there
and
debugging
utilities
like
Edge,
stop
curl
Etc,
the
netstat,
whatever
you
need.
These
are
not
there.
These
are
the
standard
ones
and
if
you
try
to
do
that
over
any
of
the
images
of
distress
or
scratch,
you'll
end
up
in
getting
the
error
that
it
is
not
able
to
process
that
because
the
shell
is
not
there,
so
it
won't
be
able
to
run.
A
A
So
ephemeral
containers
are
the
containers
to
help
debug
the
pods,
where
there
is
no
way
to
debug
directly
using
exec.
So
what
happens
is
it
is
when
you
write
a
pod
it
will.
You,
like
you,
will
be
using
cubectl
debug
to
create
those
ephemeral
containers
and
it
attaches
the
in
the
live
pod
State
itself.
You
will
have
a
pod
in
the
spec
section,
pod
spec
section.
There
will
be
a
container
that
will
be
added
and
it
will
also
auto
update
the
container
statuses
with
ephemeral
container
status.
A
So
that's
a
very
good
thing
and
it
attaches
to
The
Container
to
the
same
pod
share
the
name
spaces.
So
you
have
that
same
process.
You
will
be
able
to
see
the
process
of
a
distro
list,
or
you
know
that
particular
container
itself
and
you
can
have
the
file
view
as
well.
That
will
show
you
some
of
the
things
to
be
taken
care.
It
doesn't
have
any
Readiness,
liveness
probes.
The
resources
are
not
allowed
on
that
and
you
can
create
it
using
the
ephemeral
containers.
Api
Kyle
will
show
the
demo
for
that.
First
question:
for.
B
A
Again,
adding
a
regular
container
won't
help
because
you,
you
have
to
add
that
and
you
the
mounting
becomes
the
issue:
the
process
ID,
the
sharing
of
the
namespace.
All
these
things
won't
happen,
so
that
is
where
the
ephemeral
containers
will
help
in
basically
not
touching
the
existing
pod,
not
restarting
the
existing
pod,
so
that
stays
there.
Your
application
stays
there
and
if
anything
goes
wrong,
you
still
will
be
able
to
debug
that.
A
So
this
is
the
simple
example
of
like
Cube
CTL
debug
and
let's
say
there
is
a
nginx
pod
running
before
that.
So
you
are
debugging
that
and
you
are
using
the
image
busy
box
and
you're
getting
the
shell.
So
what
it
does
it?
It
is
adding
a
new
container
with
BusyBox
image
to
that
nginx
pod.
So
if
you
see
the
like,
if
you
get
the
cube,
CTL
get
for
siphon
or
yaml
you'll
be
able
to
see
two
sections
which
are
important
over
here,
which
is
ephemeral
containers.
A
A
B
A
A
A
and
one
thing
with
you
can
see
the
difference
between
the
the
file,
so
it
it
doesn't
look
same.
So
your
file
system
won't
be
exactly
the
same
like
when
you
do
a
cube,
CTL
exec,
you
are
into
the
port.
You
will
be
able
to
see
the
the
exact
file
system
over
there,
but
when
you
are
in
the
fmri
containers,
you
won't
be
able
to
see
the
exact
file
system
directly.
B
A
So
you
can
also
use
ephemeral
containers
without
the
cube
CTL
debug
commands,
so
you
can
use
the
apis
internal
apis
and
you
can.
That
is
very
interesting
because
there's
a
demo
that
we
have
to
set
the
security
context
like
if
you
have
to
set
the
security
context
or
if
we
need
any
additional
mounting,
which
you
cannot
do
by
default,
using
the
cube
serial
debug.
So
you
have
to
use
Cube
CT
the
apis.
A
For
that
the
call
examples,
some
of
the
gotchas,
you
can't
remove
the
FM
containers,
not
all
Container
properties
are
available,
which
we
discussed
process
name.
Space
sharing
and
the
security
context
and
mounting
volumes
are
not
there
out
of
the
box
when
you
use
Cube,
CDL
debug,
so
ephemeral
containers
like
Kyle
mentioned
it
was
a
fantasy
to
use
before
The,
Slim,
scratch
and
distortless,
and
there
have
been
a
lot
of
work
effort
and
getting
this.
A
This
set
of
you
know
ephemeral,
Tech
containers,
support
natively
into
kubernetes,
and
that
happened
very
recently,
so
it
went
in
GA
1.25
and
you
can
see
like
all
the
local
clusters,
Rancher
desktop
or
Docker
desktop
should
work
and
even
the
cloud
providers.
You
know
Co
GK
eks,
they
all
have
1.23
onwards,
so
you
should
be
able
to
use
the
same
demos
that
we
are
doing
with
Cube
serial
debug
with
those
clusters
as
well.
B
Okay,
so
let's
say
you
want
to
use
Cube
control
debug
to
debug
your
minimal
container
image
application.
So
what
do
you
do?
You
have
a
few
options.
You
can
create
your
own
debugging
image,
which
is
kind
of
this
option.
You
can
use
a
couple
of
existing
debugging
images
to
the
most
popular
one
is
not
shoot.
It's
a
it's
a
well-known
system
and
network
level.
Debugging
image
lots
of
great
tools.
There
then
there's
also
a
set
of
debugging
images.
By
light
run,
they
called
cool
kit.
B
B
B
B
B
B
B
B
A
B
B
B
Okay,
okay,
all
right
again
we
have
the
same
file
system
problem.
So
what
you
want?
You
want
to
see
the
file
system
over
the
target
application
image,
but
you
want
the
tools
from
your
debugging
image.
So
for
that
you
need
to
bring
those
debugging
tools
into
your
target,
namespace
well,
figuratively
speaking,
I'm
gonna
copy
and
paste,
because
I'm
really
bad
with
typing.
B
B
A
A
Now
you
have
the
tooling
which
exists,
so
you
have
seen
some
of
the
cool
demos
using
the
tools
like
nx3
net
shoot,
cools,
cool
kids
that
you
can
use
to
debug
your
slame
distro,
less
or
scratch
containers
as
well,
so
you
can
actually
now
use
them
for
in
production,
so
FML
containers
make
it
possible
to
debug
scratch.
Slim
digitalis
images.
A
Thank
you,
and
this
is
dim.
This
is
the
demo
repository
Cube
day,
Japan
demo,
and
you
can
try
FML
containers
on
CEO
and
create
a
minimal
container
images
using
Docker
slim
and
then
try
to
do
that
and
shout
out
to
akiro
for
the
nerd
CTL
and
if
you
have
any
yeah,
it's
an
awesome
tool
and
if
you
have
any
queries,
we
we
don't
have
time
but
we'll
be
hanging
around
here,
so
feel
free
to
ask
any
any
questions
as
well.
Thank
you.
So
much
for
joining
in.