►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Who Needs an API Server to Debug a Kubernetes Cluster? - Jose Blanquicet, Microsoft
How would you debug your Kubernetes cluster if the API server goes down but you still have access to the nodes? You are probably thinking of using commands like ss/netstat, tcpdump or any of the BCC tools directly on the node, right? Well, the problem with all those options is that they do not have knowledge of Kubernetes, so you still have to map the PIDs to containers and then to pods. It could sound easy, but it is pretty tricky. So, if you want to focus on the actual issue and skip all this low-level stuff, this talk is for you!
In this talk, Jose will demonstrate how you can trace the container's events like the creation of new processes, access to files, network and disk activity, if you still have access to the node. To do that, he will use Local-Gadget, an eBPF-powered open-source project that provides a global view of all the containers running in a host and gives the possibility of running essential eBPF tools, or "gadgets", to debug your standalone container or your Kubernetes application without using the API Server. In addition, Jose will show how to take advantage of the framework already created by Local-Gadget from 3rd-party applications.
A
Hello,
everyone
and
welcome
to
this
talk
about
who
needs
an
API
server
to
debug
cover
its
cluster
I'm
host,
about
who
hello
everyone
and
welcome
to
this
talk
about
who
needs
an
API
server
to
the
back
coverage.
Cluster
I'm
Jose
I'm,
a
senior
software
engineer
at
Microsoft.
The
main
question
for
this
talk
is:
how
will
you
debug,
when
it's
clustered
if
the
API
server
goes
down
well,
I
will
try
to
answer
that
question
by
running
a
couple
of
demos,
the
first
one
will
be
to
debug
an
API
server
is
using
BCC
and
standard
Linux
tools.
A
I
will
show
what
are
the
main
difficulties
of
using
those
tools
in
a
containerized
environment.
Then
it
will
help
me
to
introduce
the
local
Gadget
project
and
run
a
game
the
same
demo,
but
this
time
using
local
Gadget
to
finalize
I
would
like
to
share
what
is
the
row
map
and
our
plan
for
the
local
Gadget
project?
A
It
is
running
container
DS
runtime,
and
we
have
a
couple
of
Bots
running
on
it.
Everything's
running
okay
and
let
me
SSH
into
the
node
okay
I
would
like
to
start
this
demo
by
showing
how
we
can
use
BCC
gadgets
PCC
tools
to
to
debug
containers.
So
you
know
that
if
we
execute
XXL
like
this,
it
will
capture
the
event
from
the
horse
itself
and
also
from
all
the
containers
that
are
running
in
this
host
and
if
we
want
to
filter
this
output,
of
course,
we
can
grab
it.
A
But
it's
not
that
efficient,
of
course,
because
we
are
collecting
a
lot
of
information
on
the
Kernel,
sending
it
to
user
space
to
then
grab
only
a
couple
of
them.
It's
not
the
most
efficient
way,
so
the
business
tools
already
provide
us
some
examples
on
ways
to
to
filter
the
event.
It
is
possible
to
to
run
to
filter
the
name
for
the
name
of
the
command
or
the
arguments
of
the
command.
A
But
if,
for
example,
we
are
running
a
port
like
this
and
we
are
executing
a
different
existing
call
in
a
in
our
Port,
we
are
not
able
to
use
those
kind
of
filters
to
get
all
the
events
happening
in
our
container.
What
we
can
use
is
this
option
and
we
have
been
collaborating
with
the
BCC
Upstream
to
introduce
this
option.
A
The
idea
is
to
have
an
ebpf
map
that
contains
the
list
of
the
mounting
space,
and
that
represents
the
containers
that
we
want
to
get
the
events,
for
instance,
that
when
we
deploy
the
trades,
the
exact
nope
program,
the
ebpf
program
will
every
for
every
event
will
check.
If
the
mounting
space
of
that
event
is
in
the
list,
if
it
is,
then
it
will
be
forwarded
to
the
user
space
and
show
to
the
user.
A
Otherwise
the
event
will
be
discarded,
that's
most
much
more
efficient
because
we
are
doing
all
the
filtering
in
the
kernel
space
and
just
sending
the
to
the
user
space,
the
information
that
is
actually
requested
and
let's
try
to
do
it
ourselves,
yeah.
First
of
all,
we
need
consider
that
the
the
evpf
map
needs
to
be
painted,
so
the
only
option
that
we
have
to
provide
is
the
is
the
the
pathway
where
the
map
will
be
pinned.
So,
let's
define
it
like
file.
A
It
is
perfect
like
this,
but
this
time
we
pass
the
minus
M
option.
It
will
use
that
map
to
consider
to
discard
or
keep
an
event.
As
you
can
see,
we
are
not
capturing
an
event.
It
is
because
it
is
because
the
evpf
mark
right
now
is
empty,
this
seat
yeah
it
is
empty
and
what
we
need
to
do
is
to
put
in
there
the
mounting
space
of
our
container,
so
I
usually
use
PS3.
To
do
that,
then
I
look
for
my
container
this,
our
container
in
the
copy,
the
feed.
A
Let's
keep
the
pit
okay
and
now
we
need
with
using
the
beat
we
are
able
to
extract
all
the
namespaces
for
that
bit.
This
is
the
information.
The
Mal
name.
Space
is
the
one
that
we
want
to
put
in
the
Epps
map,
but
we
need
to
do
it
in
a
in
hexadecimal,
so
I
will
run
this
command
to
transforming
to
hexadecimal
and
with
the
correct
endedness,
and
now,
let's
update
the
map.
A
A
A
No,
let's
see
again,
it
is
running
no,
it
is
not
running
anymore,
so
if
it
is
being
restarted
Maybe
we
are
seeing
some
logs
about
resources.
Let's
try
the
out
of
memory
tool
and
let's
see
if
this
is
the
reason
why
the
the
the
content
is
being
restarted,
for
instance,
for
this
kind
of
situation,
our
filter,
the
filter
I've
shown
before-
is
not
possible
to
be
used,
because
if
the
port
is
the
container
that
is
being
restarted,
it
will
change
every
time
the
mounting
a
space.
A
So
by
the
time
we
added
the
money
space
in
the
map,
the
container
will
be
already
fill
and
restarted
again.
So
that's
the
problem.
The
another
problem
that
we
have
here
is
that
we
are
not
able
to
filter
by
the
name
of
the
container,
but
we
are
directly
using
the
mounting
speeds,
which
could
change
about
every
time
that
they
container
is
restarted.
For
instance,
we
here
we
captured
event.
It
is.
It
is
being
killed
because
of
Auto
memory.
So
if
we
check
VI,
we
check
what
is
the
configuration.
A
A
Well,
what
we
have
seen
we
need
to
we,
we
saw
that
we
need
to
manually
retrieve
the
contained
information,
the
pit
we
use
PS3
to
get
the
beat.
Then
we
go
through
the
broke
directly
to
get
the
mounting
space
inodes
ID,
and
for
this
for
the
filtering
it
was,
we
were
able
to
filter
by
container
but
using
the
mounting
space.
So
it
means
that
every
time
the
container
is
resulted,
we
need
to
update
the
map
and
it
is
not
possible
for
the
cases
like
when
the
pole.
A
The
container
is
a
restarting
and
restarting
again,
because
the
modeling
space
change
and
changing,
and
so
it
is
difficult
to
do
it
manually
and
for
instance,
it
is
if
we
want
to
do
something
like
don't
get
the
sockets
or
a
container.
You
know
that
each
container
is
running
on
a
different
networking
space
unless
it's
using
specific
the
host
net,
for
example,
and
so
if
we
want
to
get
the
socket,
we
need
to
switch
move
into
the
network
name.
Space
of
that
socket
network
is
based
on
that
container
and
then
run
our
tools.
A
So
this
is,
there
is
still
a
lot
of
things
that
we
have
to
do
manually.
We
are
able,
but
if
you
need
a
good
background
on
Linux
namespaces,
to
do
this
so
for
this,
let
me
introduce
you
local
Gadget,
and
the
idea
here
is
that
allows
you.
The
local
Gadget
allows
you
to
trace
local
containers
using
ebpf,
it's
a
single
binary,
statically
linked.
So
it's
easy
to
install
the
information.
Another
problem
we
saw
before
is
that
we
don't
have
the
information
on
kubernetes.
We
are
at
the
level
of
pit.
A
It
is
I'm
not
saying
that
the
BCC
and
the
standard
tools
are
not
working
correctly,
I'm,
just
saying
that
they
we
are
using,
then
in
a
different
context
for
which
they
were
created,
they
were
stalled.
We
are
trying
to
adapt
them
to
add
different
environments
So
within
with
local
gadgets.
What
we
are
trying
to
do
is
to
a
thing
in
a
tool
that
is
aimed
to
be
to
work
in
a
local
in
a
containerized
environment.
A
Another
feature
thing
that
a
local
Gadget
provide
us
is
that
it
is
possible
to
trace
kubernetes
and
all
kubernetes
containers
and
available
tools.
Some
of
them
are
comes
from
BCC
and
some
other
were
developed
by
our
team
for
some
cases
that
use
cases
that
we
found,
and
let
me
describe
how
local
gadgets
is
working.
We
have
three
main
tasks.
A
The
main
one,
of
course,
is
collect
the
insights.
That
is
the
trades
that
are
in
charge
of
all
these
tasks.
We
need
to
enrich
the
data
with
the
kubernetes
metadata
and
then
we
would
like
to
avoid
making
all
those
steps
for
filtering
manual,
filtering
and,
of
course,
being
able
to
do
it.
Also,
in
the
case,
the
container
is
restarted
and
understanding.
Okay.
Regarding
the
tracers,
this
is
a
very
simple
example
to
run
a
trace,
a
we
are
using.
A
We
are
using
the
the
ebpf
code
from
BCC
or
the
BCC
tools
we
are
using
then,
but
we
have
brought
our
own
control
playing
colon.
It
is
because
we
will
like
to
make
it
easy
to
be
integrated
in
Cloud
net
environment,
which
is
where
the
most
of
the
applications
are
written
and
go.
So
we
we
are
right
now
defining
this
API
for
our
tracers,
so
you
have
to
Define,
for
example,
for
the
exact
Tracer.
You
have
to
define
a
config
where,
mainly
you
have
an
eppf
map.
A
You
can
already
start
thinking
that
it
will
be
the
map
we
are
filtered
for
filtering
that
we
were
creating
before
manually.
We
have
an
enricer,
which
is
an
interface
that
can
be.
We
will
use
to
call
every
time
we
have
an
event.
We
want
to
reach
it
with
the
kubernetes
metadata
and,
of
course,
the
event
callback.
A
We
have
the
command
and
the
pit,
but
of
course
we
have
all
this
information
and
we
just
just
printing
those
two
regarding
the
container
collection,
the
main
tool
task
for
container
collection
is
to
enrich
events
and
to
not
and
to
not
on
also
to
notify
us
about
the
creation
and
deletion
of
the
containers
and
everything
starts
PR.
We
have
the
runc
notify
model
that
will
notify
us
every
time.
A
container
is
created,
Created
and
deleted.
A
It's
very
important
I
think
to
specify
that
this
is
a
synchronous
call
that
is
allowing
us
to
get
this
information
when
the
container
is
being
created,
but
it's
not
yet
running
so
this
point.
This
is
very
important
because
we
will
be
able
to
prepare
everything
to
start
tracing
the
events
right
before
the
containers
are
running,
so
we
will
not
lose
any
event
of
this
doing
the
startup
of
the
container,
so
every
kind
of
controller
is
created.
A
We
get
the
ID,
then
the
pit
and
we
use
a
set
of
enricers
that
we
have
to
get
more
information
about
this
container.
The
idea
is
to
keep
a
list
of
containers
with
all
this
information.
We
are
taking
the
kubernetes
metadata
from
the
container
runtime
and
we
are,
of
course
getting
all
the
the
Linus
name
spaces
that
we
will
use
to
then
map
with
the
with
events.
A
For
example,
at
the
moment,
the
Tracer
have
a
new
event
that,
and
it
wants
to
want
to
enrich
that
even
with
the
kubernetes
metadata,
it
will
call
them
Reach
API
and
we
we
pass.
You
know
also
the
mounting
space
and,
with
the
mounting
space,
we
will
be
able
to
match
what
is
the
container
related
to
that
event,
and
then
we
can
enrich
the
event
with
the
kubernetes
metadata.
This
module,
as
I
said,
will
also
provide
a
notifications
about
the
container
added
and
removed
using
the
container
structure
that
we
have
created
here.
A
So
also
this
model
as
the
Tracer
can
be
used
in
an
independent
way.
If
you
just
need
to
have
this
kind
of
notification
with
all
this
information,
then
research
and
optional,
you
can
add
all
of
them.
As
a
couple
of
days.
We
have
some
others,
for
instance,
our
last
module
the
trace
collection.
It
is
using
the
notification
about
the
container
life
cycle
because,
when
user
asks,
for
example
like
this
to
run
the
exec
to
trace
exciting
calls
and
pass
in
this
kind
of
filter,
they
want
to
filter
by.
A
Could
they
container
name,
that's
called
mycond,
so
we
will
ask
the
trace
collection
to
add
a
tracer
for
such
a
filter.
If,
by
the
time
we
ask
this,
the
content
is
not
yet
created.
Then
the
ebpf
map
will
a
bbpf
map
will
be
created
empty,
but
but
when
we'll
receive
the
notification
about
a
new
container
has
been
created.
We
check
if
the
mounting
space
of
that
container
is
a
match,
matches
the
filter
that
we
will
ask.
If
it
is,
then
we
are
without
the
mounting
space
in
the
eppf
map.
A
This
is
exactly
what
we
were
doing
manually
before,
but
we
are
adding
the
automation
that
receiving
the
notification
about
the
container
when
it
is
created
and
updated
immediately
the
map
in
this
way
we
are
not
given
that
these
events
are
synchronous
and
they
added
notification
will
arrive
before
they
containers
start
running.
We
will
be
able
to
prepare
our
evpf
map
before
we
receive
the
first
event,
so
when
we
will
receive
the
first
event,
we
will
have
everything
in
place
to
not
discard
the
the
those
very
first
events
and
provide
them
to
the
user.
A
Okay.
This
is
an
internal.
Those
were
the
internal
modules.
If
you
want
to
know
more
about
them,
we
have
written
a
couple
of
blogs
and
we
have
more
examples
for
each
of
them.
Where
that
you
can
run
easily
and
I.
Just
want
to
mention
what
are
the
use
cases?
We
have
thought
for
local
Gadget,
the
first
one,
of
course,
is
debugging
using
without
the
API
server,
where
we
have
seen
that
this
is
difficult
and
some
things
are
not
yet
possible,
at
least
using
the
PCC
tools.
A
That
was
the
first
thing
that
I
thought
when,
in
a
situation
like
this,
where
the
API
server
is
down,
another
use
case
we
have
is
is
that
if
you
are
implementing
a
tool
that
needs
to
get
inside
from
the
node,
we
can
provide
you
two
plus
two
options.
One
will
be
to
include
the
local
Gadget
binary
in
your
container
image
and
then
run
it
or
you
can
use
the
our
modules
and
to
just
get
maybe
just
a
notification
about
the
containers
or
you
can
also.
A
You
can
also
just
run
the
tracers
okay
remember
we
are
collaborating
with
the
BCC
repo
to
keep
those
eppf
code
updated
and
with
the
new
features,
and
we
have
all
only
the
rewriting
the
the
the
control
plane
to
be
easily
introduced
integrated
with
your
code
here
and
the
last
use
case
will
be
of
sharing
and
debugging
containers
also
outside
kubernetes
environment.
So
we
are
also
able
to
trace
all
those
containers
that
were
not
created
via
kubernetes.
A
Here
this
time
we,
let
me
show
you
local
Gadget.
We
have
those
options.
The
most
the
first
one
I
would
like
to
show
is
the
list
of
containers.
We
are
able
to
show
the
list
of
containers
and
also
with
the
kubernetes
metadata
we
can.
You
can
see
that
we
have
support
for
lock,
containerdy
Docker,
and
it
also
tried
to
communicate
with
cryo,
but
it
is
not
available
in
this
system,
so
it
fails,
but
you
can
run
it
only
for
the
runtimes
that
you
need
and
let's
try
to
do
the
same.
A
We
want
to
filter
by
the
pot
we
expect
before.
So
sorry,
the
port
we
created
before
and
this
time
we
don't
need
to
manually,
get
the
mounting
space.
The
pit,
then
the
mounting
speed
we
can
directly
run
the
name
of
our
container
and
to
debug
it
so
the
same
here
the
same
like
this,
we
can
also
reach
this
information.
These
events
with
the
kubernetes
metadata-
and
you
have
here
the
namespace,
the
current
name,
space,
the
name
of
the
port
and
the
name
of
the
container.
A
And
now
let
me
again
Drive
the
same
issue
we
had
before
with
the
with
the
with
the
API
server
break
API
server
and
again
we
still
are
able
to
get
everything.
Even
today,
API
server
now
is
down.
We
still
are
able
to
continue
debugging
and
let's
try
to
capture
again
also
the
following
of
out
of
memory:
Trace
out
of
memory
kill,
and
this
time
thought
we
are
able
to
filter
like
this
Cube
API
server.
A
Yeah:
okay:
here
we
can
just
wait
to
get
the
same
event
on
as
I
mentioned.
This
event
can
also
be
intrigued
with
the
kubernetes
metadata.
Okay,
me
I
think
that
all
for
local
Gadget
I
just
would
like
to
add
that
you
can
also
snapshot
socket,
for
instance,
container
name,
a
mypod
to
get
this
information
manually.
Getting
this
information
with
SS
or
netsat,
you
will
have
to
enter
in
the
namespace
of
the
network
name
space
of
the
container
to
get
this
information
because
each
name
it
container
is
running
a
different
network
name
space.
A
A
A
Is
Page
Changing
every
time
that
the
that
the
the
port,
the
the
container,
fails
if
I
run
again
the
container,
we
can
see
that
we
are
capturing
also
the
Caps
that
are
used
for
currency
during
the
startup,
and
we
can
see
that
nice
here
is
the
capability
that
is
being
denied.
That
is
the
reason
why
it
is
failing.
A
Okay,
I
think,
that's
all,
and
let
me
just
finalize
here
just
some
notes
about
local
gadgets
about
the
demo.
We
were
able
to
debug
when
it
container,
even
if
the
API
server
was
down,
we
enriched
that
information
with
the
kubernetes
metadata.
There
was
no
manual
step
and
we
don't
lose
the
event
we
download
any
event
at
this
container
start
up.
A
What
is
the
feature
for
local
Gadget
in
our
roadmap?
We
have
now
that
we
have
all
the
information
of
the
containers
of
the
of
kubernetes.
We
would
like
to
be
able
to
also
filter
by
the
container
resource,
for
example,
filter
for
all
the
containers
in
the
kubernetes
namespace,
and
so
that,
for
example,
in
the
ebpf
lab
in
that
case,
will
be
a
list
of
mountaineer
spaces
or
for
the
pot
the
same
the
container
name
remember.
It
could
be
different.
A
The
kubernetes
container
name
from
the
one
given
by
the
runtime,
and
we
would
like
to
support
non-cubernetes
containers
created,
but
all
the
runtimes.
We
are
not
supporting
only
Docker
and
we
would
like
to
add
more
and
more
and
more
guided.
We
will
start
with
the
ones
already
available
in
Inspector
Gadget,
but
if
you
want,
if
you
have
any
use
case
that
we
want
to
share,
we
are
open
to
listen
and
we
would
like
to
get
people
involved
in
this
project
so
reach
us
slack
or
the
Repository.
Thank
you
very
much.