►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Debug Your Clusters with eBPF-Powered Tools - Mauricio Vásquez Bernal, Microsoft
The intrinsic distributed architecture of applications running in Kubernetes makes them difficult to debug. When there is a problem, it is not clear where to start looking at or even which tools to use. Many of the traditional troubleshooting tools are designed to be used at the host and process level, i.e., they have to run on the same host where the application is running, and they usually only allow to filter by things like process PID or UID. In this presentation, we will introduce Inspektor Gadget. We’ll explain why this project was created and its goals. We’ll show the architecture and then we’ll make some demonstrations of how we can use it to debug issues with networking, storage and configuration of Kubernetes resources and applications.
A
Okay,
so
in
the
agenda
of
today,
I
want
to
show
you
to
describe
a
little
bit.
What
is
the
problem
about
debugging
a
kubernetes
cluster
I
will
explain,
which
are
some
of
the
traditional
stools
that
we
use
to
do
that
task.
I
will
give
a
demo
of
that
and
then
I
will
introduce
this
Inspector
Gadget
project
I
will
show
how
it
works,
and
then
I
will
present
also
a
demo
of
this
project.
So
yeah,
let's
get
started.
A
But
what
happens
with
kubernetes
is
that
you
know
kubernetes
is
distributed
operating
system,
so
our
services
are
usually
running
on
different
nodes,
so
you
don't
know
where
they
are
running
or
better.
Even
if
you
don't
know,
if
you
know
this
is
not
easy
to
trace
the
microservice,
because
that's
not
running
locally
on
your
machine,
so
you
have
to
connect
to
a
horse.
You
have
to
connect
to
other
one
and
also
in
this
micro
Services
area.
You
have
all
those
small
services
that
are
communicating
to
each
other.
A
So
when
there
is
something
that
is
not
working
fine,
there
are
so
many
things
that
you
have
to
understand
that
you
have
to
look
at
to
see
what's
wrong.
Maybe
there
is
a
networking
problem.
Maybe
there
is
a
storage
problem
with
your
giving
holes,
so
there
are
a
ton
of
things
that
could
be
going
wrong
in
a
given
cluster.
A
There
are
some
traditional
tools:
I
call
them
traditional,
because
those
are
tools
that
are
designed
to
trace
what
is
going
on
on
a
single
node.
So
when
you
have
a
process,
a
monolithic
application,
you
can
use
those
tools
to
understand
what
is
going
on,
but
those
tools
are
designed
to
work
in
a
single
node,
so
the
art
they
are
tools
that
are
not
designed
to
work
in
kubernetes
and
one
of
the
limitations
of
those
tools
is
that
they
are
not
container
aware.
A
What
it
means
is
that
if
you
want
to
filter
what
is
going
on,
if
you
want
to
understand
what
is
going
on
on
a
given
container,
you
cannot
do
that
by
using
those
tools.
Maybe
you
are
able
to
filter
by
a
given
PID
by
a
given
process,
but
there
is
no
an
easy
way
to
understand
what
a
given
container
is
doing
by
using
those
tools.
Also,
they
are
not
integrated
with
kubernetes
at
all.
A
Oh
yeah.
Okay,
so
before
going
into
the
tools,
I
want
to
show
you
the
application
that
I
want
to
deploy
here.
This
is
a
very
simple
HTTP
server
that
is
composed
of
three
different
replicas
three
different
parts
and
in
this
case
I
have
configured
somethings
related
to
security,
so
I
have
six
CTL
there.
Configured
I
also
have
a
second
profile
and
I
also
have
limited
the
list
of
capabilities
that
this
application
can
use.
Also
I
have
limited
the
resources
that
my
application
can
use
and
yeah.
A
A
So
what
a
value?
What
if
we
want
to
understand
what
is
going
on
on
that
applications?
If
we
want
to
say
the
activity
that
this
application
is
performing
on
the
cluster,
so
let
me
open
a
different
terminal.
I
will
using
that
to
send
some
requests
to
the
cluster
and
in
the
terminal
on
the
top
I'm
going
to
run
some
of
those
traditional
sold.
That
I
was
talking
before
in
this
case.
A
This
is
a
very
simple
cluster
by
many
Cube
I
have
two
different
nodes
there
and
in
order
to
trace
what
is
going
on,
I
have
two
ssh
in
one
of
the
nodes
and
I
have
to
install
those
traditional
stool.
There
I'm
not
going
to
talk
about
how
I
did
install
those
tools
there.
I
think
this
is
not
important
for
do
this
presentation,
but
in
a
real
case
you
will
have
to
find
a
way
to
install
the
tools
in
the
different
nodes.
A
So
I
log
into
the
mini
qn02
node
I
have
different
tools
there,
pre-install
and
yeah.
Let
me
execute
some
of
them.
For
instance,
let
me
start
with
TCP
Tracer.
This
is
going
to
show
you
the
different
TCP
connections
that
are
being
processed
by
this
node
and
if
you
see
in
the
below
window,
I'm
sending
some
requests
to
that,
but
not
all
the
requests
are
being
print
in
the
window
there.
The
reason
is
that
there
are
three
different
replicas
two
nodes,
so
only
some
of
the
requests
are
going
on
to
that
note.
A
So
what
happens
here
is
that
we
have
a
partial
view
of
what
is
happening
on
the
cluster.
We
are
only
having
the
bands
on
a
single
node
and
we
don't
have
like
a
global
view
of
all
the
activity
that
is
going
on
on
the
cluster
and
yeah.
Also
in
the
bands
that
we
have
there,
there
is
no
information
about
kubernetes.
We
have
the
PID,
we
have
the
name
of
the
process
and
yeah.
In
this
specific
case,
we
have
the
source
address
so
by
having
the
source.
A
Otherwise,
we
will
be
able
to
see
what
is
the
path
that
was
processing
that
request,
but
you
know
this
is
something
that
is
not
easy
to
do
and
if
you
are
trying
to
debug
a
problem.
This
is
something
that
you
would
not
like
to
do
manually.
In
all
the
case,
let
me
show
you
a
different
tool.
This
is
open
Snoop.
This
is
to
show
the
different
files
are
being
opened
on
a
given
system.
As
you
can
see,
there
is
a
simple
node,
only
two
parts
running
there
and
there
is
a
lot
of
activity
going
there.
A
So
if
this
was
a
real
one
with
100
parts
that
will
be
even
more
activity,
there
I
sent
some
requests
to
that
and
what
I'm
trying
to
look
there
is
to
see
if
I
able
to
understand
the
files
that
my
application
is
trying
to
access.
But,
as
you
can
see
there,
there
is
a
lot
of
information,
a
lot
of
events
that
I
really
don't
care
about,
because
those
are
not
related
to
my
application
but
yeah.
There
is
no
uneasy
way
to
understand
what
is
going
on
there.
A
In
this
case,
there
is
a
possibility.
I
could
try
to
check
what
is
the
name
of
the
process
of
my
application
to
get
the
PID
and
then
I
could
try
to
filter
only
by
the
bands
that
are
generated
by
that,
given
PID
so
yeah,
while
preparing
this
demo
I
realized
that
I
have
to
use
the
PID
of
this
NGS
worker
process.
So
let's
pass
that
to
open
this
node,
so
this
is
only
going
to
print
the
open
files,
but
that
given
process
again
I
sent
some
requests,
but
you
know
not
all
the
times.
A
A
So
let
me
introduce
the
Inspector
Gadget
project,
so
the
idea
of
this
project
is
that
we
want
to
fill
the
Gap,
that
is
in
those
traditional
tools
or
whether
those
traditional
tools
are
great.
There
is
a
lot
of
four
on
implementing
those
tools,
but
those
tools
are
not
a
good
fit
for
kubernetes,
so
Inspector
Gadget
is
a
project
that
is
trying
to
close
that
Gap.
The
idea
here
is
to
make
those
tools
easier
to
use
in
kubernetes.
A
This
is
heavily
inspired
by
a
similar
project
that
is
called
Cube
CTL
trace,
and
we
reduce
many
of
the
tools
from
the
BCC
project.
Of
course,
we
use
ebpf
as
a
Building
Technology.
We
inject
EBP
apps
programs
into
the
kernel
to
understand
what
is
going
on
and
yeah.
We
are
coordinates
aware.
What
it
means
is
that
we
are
able
to
filter
the
events
for
a
given
kubernetes,
let's
say
namespace
pod
whatever,
and
we
also
provide
coordinates
information
for
domains
that
we
capture
on
the
cluster
and
additionally,
to
that
we
provide
a
cube,
CTL
plugin.
A
What
it
means
is
that
a
user
that
is
used
to
use
a
cube
CTL
doesn't
have
to
learn
how
to
use
a
completely
new
tool.
So
we
provide
an
interface
that
is
as
aligned
as
possible
to
cube
CTL
so,
for
instance,
how
to
filter
by
a
given
name
space,
how
to
configure
the
cube
config
of
the
cluster
that
you
want
to
handle
and
so
on,
thus
don't
directly
or
better
in
the
same
way,
in
the
same
way
as
it
is
done
by
Cube
CTL.
So
what
are
the
features
of
a
Inspector
Gadget?
A
One
characteristic
that
I
say
is
very
important
about
Inspector
Gadget
is
that
this
is
very
simple.
To
install
I
mean
when
you
are
trying
to
fetch
something
in
your
cluster,
you
don't
want
to
worry
about
installing
something
that
is
very
difficult
to
set
up.
So
the
idea
here
is
to
have
something
that
is
very,
very
easy
to
do
that.
You
only
have
to
run
like
one
command
and
that's
it,
because
the
idea
is
that
we
are
providing
a
tool
to
fix
the
problem.
A
So
you
don't
want
to
get
into
trouble
installing
that
tool,
and
this
is
I
will
say,
Mark
like
a
detail,
but
this
is
an
important
one.
So
there
are
some
cases
when
you
have
a
pot
and
when
the
port
tries
to
run
there
is
a
crash
just
at
the
beginning,
and
you
don't
know
what
is
going
on.
Maybe
you
don't
even
have
the
logs
of
that
pod,
because
this
is
just
crashing
at
the
beginning.
So
we
have
a
bunch
of
logic.
A
I
will
say
complex
mechanism
in
Inspector
Gadget
to
be
able
to
capture
the
first
event
of
a
container.
So
we
have
a
mechanism
that
when
a
container
is
starting
with
stop
the
creation
of
the
container
for
a
bit,
then
we
set
up
all
the
infrastructure
that
we
need.
Then
we
let
the
creation
of
the
container
go
and
what
it
allows
us
is
that
we
are
able
to
trace
the
very
first
events
that
the
container
is
doing.
A
So
we
have
the
complete
information
about
what
the
container
is
doing
from
the
very
beginning,
and
this
is
something
that
is
like
an
ongoing
work,
so
we
are
trying
to
implement
some
apis
for
different
users.
So
so
far
we
have
some
Golem
packages
that
can
be
used
by
third-party
applications.
So
if
you
are
implementing
a
Golan
application-
and
you
want
to
understand
what
are
the
processes
that
are
created
in
a
horse-
the
TCP
connections
that
are
created,
you
can't
use
reuse
our
Golem
packages
and
then
you
don't
have
to
implement
the
ebpf
logic
by
yourself.
A
This
is
the
Inspector
Gadget
architecture.
We
deploy
Inspector
Gadget
as
a
demon
set
on
the
cluster,
so
there
is
a
An
Inspector,
Gadget
pod,
perish
note.
Then
there
we
use
we
inject
those
ebpf
programs
into
the
kernel,
as,
as
you
know,
the
kernel
is
shared
between
all
the
pods
running.
On
the
same
also,
by
injecting
those
programs
we
are
able
to
capture,
we
are
able
to
trace
information
from
those
different
parts,
in
this
case
the
user
interrupts.
A
We
have
different
categories,
different
kind
of
gadgets.
We
have
a
advisors,
so
those
are
gadgets
that
help
you
to
recommend
configuration
for
the
cluster.
For
instance,
we
have
a
set
chronovisor.
This
is
a
gadget
that
captures
the
differences
and
calls
that
are
executed
by
a
given
application,
and
then
this
gadget
tell
you:
what
is
the
setcom
profile
that
you
should
be
using
for
that
specific
application?
A
We
have
audit,
so
those
are
gadgets
to
check
if
a
security
profile
is
blocking
some
activity
from
your
application.
Again,
if
you
have
a
second
profile,
this
tool
will
show
you
when
that
second
profile
is
blocking
zone
system,
calls
from
your
application
profile.
Those
are
for
checking
the
performance
of
the
systems.
One
example
is
for
checking
the
latency
of
the
block.
I
o
disk
on
the
on
the
cluster
Snapchat
filters
are
forgetting
the
the
status
of
a
given
subsistence,
for
instance,
for
printing
the
different
processes
are
running
on
the
cluster
top.
A
Those
are
I
will
say.
Well,
you
know
my
favorite,
so
those
gadgets
allow
you
to
check
where
the
resource
that
is
the
most
used
on
the
cluster.
We
have,
for
instance,
file
top.
So
this
will
show
you
what
are
the
files
that
are
the
most
used
on
the
cluster.
So
if
you,
if
you
have
any
problems
like
high,
I
o
utilization
by
using
this
gadget,
you
will
be
able
to
understand
what
is
the
file
that
is
creating
the
high
IO
activity
and
who
is
doing
that.
So
what
is
the
Pod?
A
What
is
the
container
that
is
trying
to
read
or
write
that
file?
And
finally,
we
have
tracers,
so
these
are
like
the
most
traditional
ones
if
you
are
familiar
with
BCC,
those
are
like
the
snoopers
from
BCC
that
allow
you
to
understand
different
events
on
the
cluster.
So
when
a
new
process
is
created,
a
file
is
open,
a
TCP
connection
is
created
and
so
on.
Okay,
this
is
time
for
a
demo
of
Inspector
Gadget.
A
A
Okay,
so
the
first
thing
is
how
to
install
Inspector
Gadget,
where
we
can
go
to
the
Inspector
Gadget
repository
there.
We
have
the
different
releases
and
for
this
release
we
have
the
cube
CTL
Gadget,
compiled
for
different
operating
systems
and
architectures
in
this
case
I'm
running
a
Linus,
and
this
is
for
machine,
so
I'm
going
to
take
that
binary
and
yeah
I
only
have
to
download
the
binary
actually
yeah,
there's
no
direct
to
the
binary.
That
is
a
compressed
file,
so
I
have
to
uncompress
that
file.
A
There
is
the
binary
there
the
license
and
to
make
this
binary
available
as
a
cub
CTL
Gadget
plugin
I
have
to
copy
that
into
a
path
that
is
available
to
cube
CTL
and
then
I
can
use
this
just
as
cube
CTL,
Gadget,
so
yeah.
So
so
far
we
have
installed
a
decline
part
of
Inspector
Gadget,
but
we
still
have
to
deploy
that
into
the
cluster.
So
if
we
check
the
different
parts
that
are
running
on
this
gadget
name
space,
we
can
see
that
there
is
no
any
port
there.
A
A
So
what
I'm
doing
here
is
I'm
creating
a
different
window
to
run
some
commands
so
yeah,
let's
start
by
using
one
of
the
categories
that
I
showed
you
before.
This
is
just
to
print
the
different
processes
that
are
on
my
cluster,
in
this
case
I'm,
printing,
all
the
processes
in
the
in
the
in
the
default
name
space.
If
I,
if
I,
want
to
print
the
processes
in
a
different
name,
space
I
can
use.
I
can
use
this
dash
n
option.
So
those
are
the
processes
in
the
cube
system
name
space.
A
A
A
Let
me
show
you
some
of
the
tracers,
so
this
is
to
show
when
a
file
is
open
on
the
cluster.
So
if
I
send
some
request
to
my
application,
you
can
see
that
in
the
top
we
get
the
file
that
is
being
accessed
there,
so
that
should
be
in
this
dot
HTML.
So,
unfortunately,
there
is
no
enough
space
on
this
screen
and
on
the
left
part
of
the
terminal
we
set
all
the
information
related
to
kubernetes.
A
We
also
get
all
the
information
on
the
specific
you
know
like
PID
command
and
for
this
specific
tool,
file,
descriptor
and
so
on.
Let
me
try
to
use
a
different
tool.
This
is
for
tracing
the
TCP
connections,
but
yeah
in
this
case
that
tool
prints.
A
lot
of
information
on
the
screen
is
so
small
here
so
I'm
going
to
configure
that
to
only
print
some
of
the
information
that
I'm
really
interested
in
so
yeah
in
this
case,
I
only
want
to
print
the
the
know,
the
port,
the
type
of
event.
A
This
is
offset
or
close
a
process
and
information
related
to
the
IP
addresses
and
ports.
Actually,
yeah
I
made
a
typo
there.
It
was
the
part
yeah,
that's
right,
so
if
I
try
to
send
requests
to
my
application
again,
we
can
see
how
this
is
printing.
The
different
requests-
and
what
is
interesting
here
is
that
I
get
all
the
requests.
A
So
this
is
what
is
happening
on
all
the
Clusters
I
mean
not
only
that
not
only
the
requests
are
on
sent
to
a
specific
backend,
but
I
get
like
the
global
view
of
what
is
happening
in
my
application.
Okay,
but
this
is
just
to
understand
what
what
is
what
is
going
on
with
my
application?
What
about,
if
I,
have
an
application
that
is
broken?
What
about
if
there
is
something
that
is
not
working
right
with
my
application?
A
So
what
I'm
going
to
do
here
is
that
I
will
break
my
application
for
sure
I
I
will
I
won't
tell
you
how
I
break
the
I
broke
that
but
yeah,
let's
see
if
we
are
able
to
fix
that
by
using
Inspector
Gadget,
so
I
broke
my
application
I'm
trying
to
send
some
requests
again,
but
yeah
the
application
is
not
responding.
We
are
getting
a
connection
refuse
and
in
this
case
we
can
see
that
the
different
parts
are
in
this
crash.
Look
back
off
stage.
A
So
if
we
try
to
see
the
logs,
let
me
check
the
locks
of
the
application
I'm
going
just
to
use
the
label
to
avoid
typing
the
name
of
the
Pod.
Here
we
can
see
there
in
this
line
that
yeah
there
is
something
going
on
with
the
buying
call.
For
some
reason,
we
are
getting
a
permission,
deny
error
so
yeah.
What
could
it
be?
If
you
remember
I?
Was
configuring
some
things
related
to
security,
so
yeah?
A
If
you
look
the
last
line
there,
we
can
see
the
the
application
trying
to
use
this
net
buying
service
and
it
was
denied
by
the
kernel.
So
what
it
means
is
that
the
kernel
is
now
allowing
the
application
to
use
this
capability
right.
So
what
is
happening
here
in
this
case
is
that
we
don't
have
that
capability
on
the
list.
We
were
dropping
all
the
capabilities,
so
yeah
I
have
to
add
that
capability
there
actually
yeah.
A
Actually,
I
I
have
to
close
and
start
from
the
beginning,
because
I
wasn't
able
to
update
that
there.
Here
we
go
so
I
am
going
to
add
the
capability
there
and
yeah
I
have
to
reapply
that
and
yeah.
Let's
try
to
see
if
this
scene
is
working
again,
we
are
getting
a
connection
review.
So
maybe
there
is
another
problem.
If
we
try
to
get
information
about
the
pots,
pots
are
running,
so
it
seems
to
be
a
different
problem
in
this
case.
If
we
get
the
logs,
there
is
no
any
useful
information
there.
A
Everything
appears
we
working
fine,
so
yeah.
Let's
check
the
specification
of
the
service
that
we
are
using
to
access
this
service.
Actually,
what
I
want
to
do
here
is
to
print
the
yaml
specification
to
get
the
full
information
about
the
service
so
yeah.
What
I'm
looking
here
is
for
the
Target
Port
to
see
what
is
the
port
that
the
service
should
be
exposed
in
the
path
so
yeah.
The
Target
Port
is
Triple
Eight
in
this
case.
So
how
can
we
know
if
this
is
the
part
that
our
application
is
using?
A
Well,
we
have
a
different
Gadget.
That
is
called
Trace
bind.
This
is
going
to
show
us
an
event
when
application
opens
a
port
on
a
given
part
again.
What
I'm
going
to
do
is
to
scale
down
scale
up,
and
we
consider
what
is
the
part
that
our
application
is
using.
So
what
was
happening
in
this
case
is
that
I
have
configured
the
service
in
a
wrong
way.
I
was
using
the
wrong
port
on
the
service.
So
please
let
me
update
that.
A
Yeah
I
only
have
to
redeploy
that
and
yeah.
Let's
try
again,
okay,
this
sign
a
different
error,
but
yeah.
Let
me
try
some
more
times
yeah,
so
sometimes
it's
working,
sometimes
it's
not
working.
This
is
one
of
those
difficult
problems
to
to
debug,
so
maybe
yeah.
We
can
use
this
TCP
Gadget
again
to
see
what
is
happening
to
those
connections.
So
I
sent
a
request.
A
He
went
to
mini
Cube,
node
and
I
can
see
that
everything
is
working
fine
for
that
one,
but
yeah
this
one
was
sent
to
the
mini
qn02
node,
but
in
this
case
it
wasn't
working
so
I'm
just
trying
to
send
some
more
requests
to
see
if
I
able
to
get
more
information
so
yeah,
if
you
consider
it
since
the
like
all
the
requests
that
are
going
to
that
mini
Cube
and
zero
to
note-
are
not
working
fine.
So
what
could
be
going
on
there?
A
Maybe
there
is
a
networking
problem
with
that
node,
but
actually
the
the
request
is
arriving
there.
Well,
in
this
case,
let
me
check
the
second
profile.
If
you
remember
correctly,
the
second
profiles
are
installed
on
the
host.
So
maybe
there
is
something
wrong
with
the
second
profile
on
that
specific
node.
So
let
me
use
this
audit
setcon
Gadget
to
understand
if
the
second
profile
is
blocking
some
activity
from
my
application
again
in
this
case,
I
have
so
many
information.
Let
me
only
print
the
the
things
that
are
interested
in.
A
Okay,
let
me
send
a
request
to
my
application
and
yeah.
As
you
can
see
there,
we
have
that
there
is
this:
send
file
system
called
that
was
performed
by
those
parts,
and
the
code
means
the
action
that
the
kernel
took
in
this
case
was
to
kill
the
process,
actually
the
threat
of
my
application.
So
what
is
happening
in
this
case
is
that
probably
we
are
missing
that
system
call
in
our
second
profile.
So
let
me
SSH
into
the
node.
Let
me
update
the
security
profile,
the
second
profile,
sorry,
so
I'm
going
to
check.
A
A
Of
course,
this
is
a
demo
in
a
real
world
scenario,
you
will
be
a
lot
more
complicated
because
there
are
so
more
things
to
look
at,
but
this
is
just
to
show
you
some
of
the
things
that
you
will
be
able
to
debug
by
using
this
Inspector
Gadget
tool.
So
what
is
the
future
ahead
of
this
soil?
Well,
we
are
trying
to
make
this
a
more
a
community
project,
so
we
are
trying
to
get
more
involved
with
the
community
there.
So
we
are
preparing
for
some
meeting.
A
This
is
some
boats
to
the
cncf
that
could
happen
today
this
week
or
next
week.
We
are
almost
ready
to
do
that
and
from
the
technical
point
of
view,
we
are
implementing
these
API.
So
if
different
third-party
applications
wants
to
consume
the
data
that
we're
capturing
with
the
cluster,
we
are
going
to
provide
apis
for
that
and
yeah
just
to
mention
two
different
talks
that
are
related
to
this
project.
So
today,
in
this
same
track
in
the
afternoon,
there
is
going
to
be
a
talk
for
my
colleague
Jose.
A
This
is
to
show
a
similar
tool
that
we
developed,
but
this
is
a
tool
that
doesn't
require
the
API
server
to
work
and
on
Wednesday
I
will
be
explaining
how
to
generate
security
profiles
by
using
ebpf
based
tools.
One
of
those
tools
is
Inspector,
Gadget
and
yeah.
That's
it.
Thank
you
very
much
and
happy
to
take
any
question
that
I
can
have.
Thank
you
so
much
Mauricio.